Campuses:
This shows you the differences between two versions of the page.
| Both sides previous revisionPrevious revisionNext revision | Previous revision | ||
| computing:department:unix:file_storage [2015/12/16 10:51] – [Project Data storage] allan | computing:department:unix:file_storage [2016/02/02 13:52] (current) – [sshfs] allan | ||
|---|---|---|---|
| Line 1: | Line 1: | ||
| - | <box 20% right right-clear red|More Information> | + | |
| - | {{indexmenu>: | + | |
| - | </ | + | |
| ====== Data storage on Unix ====== | ====== Data storage on Unix ====== | ||
| - | ===== Home directory storage and backups | + | ===== Home directories |
| Your home directory on the Unix cluster has a usage quota, to avoid too much space being taken by individual users. The standard quota is normally somewhere around 10GB. You can see your current usage and quota by visiting [[https:// | Your home directory on the Unix cluster has a usage quota, to avoid too much space being taken by individual users. The standard quota is normally somewhere around 10GB. You can see your current usage and quota by visiting [[https:// | ||
| Line 10: | Line 8: | ||
| If you run into your quota, and after reviewing your file usage find that you still need more space, you can reply to the warning email to request a change from us. Please understand, however, that the amount of space available is limited. Your home directory is not intended for large research data sets, for which separate project-specific storage should be used. | If you run into your quota, and after reviewing your file usage find that you still need more space, you can reply to the warning email to request a change from us. Please understand, however, that the amount of space available is limited. Your home directory is not intended for large research data sets, for which separate project-specific storage should be used. | ||
| - | < | + | **Home directories should not be used for data-intensive |
| The home directories are backed up nightly, and in addition filesystem " | The home directories are backed up nightly, and in addition filesystem " | ||
| - | ===== Project | + | ===== Shared project |
| - | If you need a shared area for your project where multiple people can develop or store code, we can create such an area and back it up for you. This is intended for small-scale usage; if you need to store large data sets then read under " | + | If you need a shared area for a group or project where multiple people can develop or store code, we can create such an area and back it up for you. These are made available under the ''/ |
| - | ===== Research | + | ===== Research |
| - | Other file systems are provided for research or project-specific data, under the **/data** hierarchy. This storage space is **purchased by the research group**. It can take the form of simple single drives in linux workstations, | + | Other file systems are provided for research or project-specific data, under the **/data** hierarchy. This storage space is **purchased by the research group**. It can take the form of simple single drives in linux workstations |
| These file systems are usually named either after the research group group (for a fileserver volume), or with the name of the workstation which hosts it, and contain further directories organized by user or by project. These areas should be used for large data sets and storage for local processes. Note that these areas are provided by the // | These file systems are usually named either after the research group group (for a fileserver volume), or with the name of the workstation which hosts it, and contain further directories organized by user or by project. These areas should be used for large data sets and storage for local processes. Note that these areas are provided by the // | ||
| + | |||
| + | ===== Backups ===== | ||
| <note warning> | <note warning> | ||
| - | **Research data areas are not backed up** | + | * Unix **home directories** are backed up daily. |
| - | * For our ZFS data storage, we create | + | * Older files may need to be recovered from snapshots or from tape. Complete tape backups are currently made monthly and retained for 6 months. |
| - | * If you have critical research data which requires backup, | + | |
| + | * **Research data areas are not backed up** | ||
| + | * Data areas on our ZFS storage | ||
| + | * Linux RAID storage does not have snapshots. | ||
| + | * If you have critical research data which requires backup, | ||
| + | * Local data drives in workstations are **never** backed up. | ||
| </ | </ | ||
| + | |||
| ===== Local " | ===== Local " | ||
| Line 43: | Line 49: | ||
| - | ===== Remote access to file systems | + | ===== Remote access to data ===== |
| + | |||
| + | The security model of Unix NFS is too weak to permit exporting it to any systems | ||
| + | |||
| + | ==== globus ==== | ||
| + | |||
| + | Please see [[: | ||
| + | |||
| + | ==== sshfs ==== | ||
| - | You can use //fuse// (the userspace filesystem driver) to mount any remote filesystem which you have ssh access to. You can use this to access files from other systems | + | You can use //fuse// (the userspace filesystem driver) to mount any remote filesystem which you have ssh access to. Although this is not a high-performance solution, you can use this to access files from non-Physics linux systems. For example: |
| mkdir ~/mnt #create a place to put it, can be called anything you want. | mkdir ~/mnt #create a place to put it, can be called anything you want. | ||