Home Directory and Data

Your Home Directory

Your home directory is for your day to day, non-computational work, papers, software development, administrative documents, etc., etc. Basic statistical work can run in your home directory fine, but under no circumstances should you run intensive batch jobs or condor out of your home directory. That puts a heavy load on the file server and will make life agonizing for most people in the department. We have separate research spaces for this sort of work (see below).

Backups

We do not use tape backup for any of our file storage. Instead, we use a system called ZFS as well as a replication infrastructure which keeps a copy of everything offsite. The most useful part of ZFS for the average person is the periodic snapshots — a sort of live backup — that you can access easily on your own. From OSX or Linux, you just need to go to the directory ~/.zfs/snapshots and pick a directory with the date you want. A copy of your home directory at that time will appear in the snapshot directory. Just copy the files you want.

Windows users: to mount your home directory and project directories see How to map a samba drive.

Your Research Data

Your home directory is not an appropriate place for collecting the output from intensive computational work — it slows down the server for everyone else. We have separate file servers for both computational work as well as for long-term storage of inactive data (archives, essentially).

Please contact us to set up a new computational space for your research.

We also recommend that labs with multiple people working on a single project get project directories. That way, work for particular project is in a central location. It avoids problems when people graduate and research has to be moved around to be saved.

Seeing your Project and Computational Directories

When you get a project directory, it will have a name like /z/Proj/your-project. Many people have the common Unix workflow where they will cd to successive steps in the directory path to get somewhere, using ls to make sure they're in the right place:


$ cd /z
$ ls
Comp/ DW/ Proj/ Users
$ cd Proj
$ ls
$

However, very often you will not see your project directory, as in the example above. This is due to the automounter, which only attaches shared drives to machines when someone has asked for them by name. This reduces workload for the file servers.

To see your project directory, you have to ask for it by its full name from the start, cd /z/Proj/your-project, and that will mount the drive for you. Once it is mounted, ls and shell tab completion will work as expected, but the automounter will unmount your project directory if no one has used it for 10-20 minutes or so. If you cd into the directory, that counts as use, and as long as you have a shell active in the directory, it will not be unmounted.