This document explains how to use our machines which have been set up to do access both your DoIT and your Biostat storage, to make it easier to move data over as part of the IIT Transition Project.
I will use $NETID to represent your personal NetID, and $BUID to represent your local Biostatistics unix account login. You will need to use both to get both systems available to you on these hosts.
Connecting
First, you need to connect to the host halo-1 using your NetID and password. From a terminal, you can use:
ssh -l $NETID@wisc.edu halo-1.biostat.wisc.edu
The first time you connect it may take a few seconds as your local account is set up on the machine.
While you are logged into this machine your account is your NetID. The only way you will get access to BCG storage is through the sshfs steps given below.
DoIT Storage
The different DoIT storage options, Research Drive, etc., are available under the /mnt directory.
- Research Drive:
/mnt/rd/$NETID - Restricted Research Drive:
/mnt/rrd/$NETID - Shared Drive:
/mnt/bmi/homedir/$NETID
Accessing Biostat Directories
First, you need to create a directory name where the Biostat directory can be attached. We have set up the local /ua space for that. Just do this:
mkdir /ua/$BUID
You could call this directory anything, but it is easier to start by using your biostat account name.
Next, you use the command sshfs to connect to your Biostat storage with your local, Biostat unix name and password. Pick one of the nebula compute servers to connect through:
sshfs $BUID@nebula-7:/ua/$BUID /ua/$BUID
(it will ask for your Biostat password)
Now you can go into the /ua/$BUID directory and access your files.
You can unmount your Biostat drive when you are done with it with:
fusermount -u /ua/$BUID
Copying Files
The most convenient tool for copying files between DoIT and BCG filesystems is rsync. If the copy is interrupted for some reason, rsync does a better job that just cp (copy) of not recopying files that have already been moved.
It is extremely important, though, to tell rsync to not copy over your snapshots, because that will eat up far too much space on the DoIT servers (which have their own backup system).
As an example of copying your BCG home directory to a DoIT shared drive (note the final trailing slashes – it is important to get those right):
rsync -av --exclude='*/.zfs/' /ua/$BUID/ /mnt/bmi/homedir/$NETID
The rsync command has many options, so please do contact us with specific questions.