Transferring files (Windows)
To do any meaningful work on Neon, you will probably have to transfer files back and forth (pushing data sets and programs you’ve written out to Neon, pulling results from Neon to back to your machine, etc.). These are the instructions for Windows; Linux instructions are below.
The easiest way to transfer files on a Windows machine is to simply mount your home directory on Neon so that it appears on your local machine as a regular directory. Click on Computer > Map network drive, then enter \\neon-home.hpc.uiowa.edu\myHawkID as the Folder. Done!
It is possible to set this up on a Linux/Mac machine, although you will have to have IT set it up for you as it requires administrative privileges on your machine.
Transferring files (Linux/Mac: scp)
The basic way to transfer files from a Linux/Mac terminal is scp:
scp -P 40 file.txt neon.hpc.uiowa.edu:
This will copy the file file.txt from your current working directory to your home directory on Neon. Note the -P 40 option to specify port 40; this is not necessary on-campus (but doesn’t hurt either). To copy a directory, you need to specify the -r option:
scp -r -P 40 mydir neon.hpc.uiowa.edu:
Transferring files (Linux/Mac: rsync)
Another useful option to know about for directories is rsync. The advantage of rsync is that it only copies files that have changed between the two directories. So for example, suppose you copy a directory containing large amounts of data to Neon, analyze the data using Neon, then want to copy the directory back. This will be very inefficient, since the data is unchanged – you really just want to copy the files that have changed. You could do this manually, but rsync automates the process:
rsync -avz -e "ssh -p 40" --delete mydir/ neon.hpc.uiowa.edu:mydir/
This will synchronize the mydir directory on Neon with the one on your local machine. Note that the above code is a little aggressive; the --delete option means that if a file on mydir on Neon does not exist on your local mydir, it will be deleted from Neon (in other words, the two directories will look exactly the same once the above command is finished). You can read the full rsync documentation to find out more about all the options.
To synchronize in the opposite direction (i.e., make your local mydir folder look exactly like the one on Neon), simply reverse the order of the last two arguments:
rsync -avz -e "ssh -p 40" --delete neon.hpc.uiowa.edu:/Users/pbreheny/$1/ $1/
Again, note that you must be very careful while doing this, as you could inadvertently delete files in your local directory.
Transferring files using git
Both of the Linux/Mac options given above are not as useful as they used to be, now that the Neon cluster requires dual-factor authentication. Another option that I have found useful is to set up a git repository for your code, then push/pull as needed. If you know how git works, then this is self-explanatory; if you don’t, well, that’s a separate tutorial.