Chapter 6 File Upload and Download

We are on the sixth step of the pathway.

Exchanging files with the cluster is very important. You can imagine scenarios where:

You want to download log files or output files
You want to upload a custom .sh script file that you wrote on your laptop
You want to upload other files

In this course, upload and download of files is performed using Cyberduck. Cyberduck is a tool that lets us connect to the cluster securely, browse files, and transfer files securely.

If you are working with sensitive data (such as data with PHI that requires HIPAA compliance, you need to be extra cautious about transferring your data to the cluster. Your home directory is not an appropriate storage option for such data. Make sure you consider any stipulations in your data use agreements.

6.1 Download Cyberduck

Download the latest version of Cyberduck here.

Downloads page for Cyberduck.

Note that the version of Cyberduck in the Software Center or Self Service might not be current, causing compatibility issues with some operating systems.

6.2 Create Connection

Launch Cyberduck and click on “Open Connection”.

From the dropdown menu, select “SFTP (SSH File Transfer Protocol)”
For Server, type “rhino.fhcrc.org”
Fill in your HutchNetID for Username and fill in your password

Click “Connect”

Screenshot of Cyberduck “Open Connection” configuration.

Click “Allow”. You can also check the box to indicate “Always”.

You can select allow when Cyberduck asks about an Unknown Fingerprint.

You should see your script file “01.sh” and the log file.

Files, including the script we previously ran, can be accessed via Cyberduck.

6.3 Download and Edit the Script

Right click on “01.sh” and select “Download”
You will see a “Transfers” prompt open, and the 01.sh file should now appear in your Downloads folder
Open the 01.sh in your Downloads folder

Screenshot of the script opened on the users local laptop.

Edit the message to include your name and save the file. Rename the file 01-name.sh.

Screenshot of the edited and renamed script file contents. The contents of the script now reads "Hello, Ava!".

6.4 Upload the New Script

From your Downloads folder, simply drag the file to Cyberduck.

Screenshot of local and Cyberduck files, depicting the dragging that transfers the file over.

You should now see the new script among your cluster files.

The new file is now listed in Cyberduck and is therefore on the cluster.

6.5 Run the New Script

Return to your Terminal. Submit a job with your new script by running the following. When you type ls you should see a new log file!

sbatch 01-name.sh

After running the newly edited script, there is a new log file present on the cluster.

The job numbers included in log file names generally increase in number. The greater the number, the more recently the job was run.

Use the cat command to inspect the log. Make sure you replace [your-number-here] to match your file. The message should show the new text that you added!

cat slurm-[your-number-here].out

The message "Hello, Ava!" has been printed to the Terminal using the cat command.