1  Navigating the Bash Filesystem

1.1 Learning Objectives

By the end of this session, you should be able to:

  • Navigate and copy data to the different filesystems available at Fred Hutch.
  • Explain the difference between absolute and relative file paths.
  • Set Permissions on and execute a bash script
  • Execute scripts written in Python and R on the command line
  • Find help on the system and on the web

1.2 Exercises

Open up the exercises here or in Google Classroom.

NoteReminder about Terminology

Defined words are double underlined. You can click and hold on them to see the definition. Try it below!

1.4 Setting Yourself Up for Success

So we have logged into rhino. Now what?

1.6 Going /home: ~/

There is one important file alias you should always remember: ~/ is shorthand for your own home directory.

Depending on the linux distribution, this can be a different location. On the FH filesystem, when I use ~/, it maps to:

/home/tladera2/

The home directory is also important because it is where your configuration files live, such as .bashrc (see Section 11.1).

Depending on how you work, you may want to store your scripts and workflows in /home/. Some people prefer to keep their scripts, data, and results in a single folder. This is not really practical for most genomics projects, unless you are saving processed data. For more info, see Section 12.10.

NoneYour current working directory: .

There is an alias for your current directory: . (the period sign).

This becomes useful when you want to output files to your current location. We’ll use this later.

1.7 Grabbing Stuff from GitHub

For the rest of the exercises for today, we’ll be grabbing the scripts from github using git clone.

git clone https://github.com/fhdsl/bash_for_bio

This will create a folder called bash_for_bio/ in our current directory. This directory has all of the course materials, including the scripts.

ImportantStay in bash_for_bio/

Throughout this course, I expect you to run code in the base bash_for_bio/ folder, not in scripts or in data. All of the code is tested with this in mind.

If you are having problems executing the code, please make sure you are in the base bash_for_bio folder, or adjust your file paths when running the script.

1.7.1 du: How much space?

One of the first things we can do is check for disk usage with the du command. If I run du by itself on the command line, it will give me the disk usage of all folders and files in our current directory, which is a lot of output.

There is an option called -d that lets us specify the depth. -d 1 will give us only the file sizes of the top level folders in our directory.

Make sure you are in the bash_for_bio/ directory. Then try the following command:

du -d 1 -h .

Here are the first few lines of my du output within the bash_for_bio folder:

240K    ./_extensions
192K    ./.quarto
616K    ./scripts
1.9M    ./data
8.6M    ./.git
6.7M    ./docs
10M ./images
30M .

If we want to specify du to scan only a single folder, we can give the folder name.

du -d 1 scripts

And I will get the following output:

144K    scripts/week1
56K scripts/__pycache__
128K    scripts/week3
232K    scripts/week2
616K    scripts
NoneTry it out

Try checking the disk usage using du for the bash_for_bio/ folder in your /home directory (mine is /home/tladera2/bash_for_bio/).

du -d 1 bash_for_bio/

Try out using du -d 2 on your home directory:

du -d 2 ~/

1.8 FH users: the main filesystems

When working on the Fred Hutch HPC, there are four main filesystems you should consider:

  • /home/ - The home filesystem. Your scripts can live here. Also where your configuration files (such as .bashrc) live. Can be accessed using ~/.
  • /fh/fast/ (also known as fast) - Research storage. Raw files and processed results should live here.
  • /hpc/temp/ (also known as temp) - The temporary filesystem. This filesystem is faster to access for gizmo nodes on the cluster, so files can be copied to for computation. The output files you generate should be moved back into an appropriate folder on /fh/fast/. Note that files on /hpc/temp/ will be deleted after 30 days.
  • /fh/regulated/ - A secure filesystem meant for NIH regulated data. If you are processing data that is regulated under the current NIH guidelines, you will process it here.

So, how do we utilize these filesystems? We will be running commands like this:

1ml BWA
2bwa mem -M -t 2 \
3  /fh/fast/reference_data/chr20 \
4  /fh/fast/laderas_t/raw_data/na12878_1.fq \
  /fh/fast/laderas_t/raw_data/na12878_2.fq > \
5  /hpc/temp/laderas_t/aligned_data/na12878_1.sam
1
Load bwa software
2
Start bwa mem (aligner)
3
path of genome index
4
path of paired end reads files
5
path of output

To understand the above, We first have to familiarize ourselves with absolute vs relative paths.

NoneWhen you need to span multiple lines: \

Sometimes it’s hard to read code that is a single line. You can break up a very long line of code using the \ (backslash) character.

For example, instead of:

bwa mem -M -t 2 /fh/fast/reference_data/chr20 /fh/fast/laderas_t/raw_data/na12878_1.fq /fh/fast/laderas_t/raw_data/na12878_2.fq > /hpc/temp/laderas_t/aligned_data/na12878_1.sam

We can rewrite it as:

ml BWA                                                
bwa mem -M -t 2 \                                     
1  /fh/fast/reference_data/chr20 \
2  /fh/fast/laderas_t/raw_data/na12878_1.fq \
3  /fh/fast/laderas_t/raw_data/na12878_2.fq > \
4  /hpc/temp/laderas_t/aligned_data/na12878_1.sam
1
Path of reference genome
2
1st paired end read FASTQ file
3
2nd paired end read FASTQ file
4
Output file location

We’ll use this throughout the book so the code is easier to read.

1.8.1 More about the FH Filesystems

https://sciwiki.fredhutch.org/scicomputing/store_posix/

1.9 Absolute versus relative paths

You may have muddled with file paths, and maybe have used absolute paths to specify the location of a file. When you are processing files, it is important to understand the difference.

Absolute paths contain all the information needed to find a file in a file system from the root / directory. For example, this would be an absolute path:

/fh/fast/laderast/immuno_project/raw_data/chr2.fa.gz

Absolute paths always start with /, because that is the root directory, where all the top folders and files live.

In terms of folder structure, this is what this looks like:

1/
2├── fh
│   └──fast
│       └──laderast
|            └──immuno_project
│                 └──raw_data
│                    └──chr2.fa.gz
1
Root directory
2
Folders in root directory

Relative paths break up an absolute path into two pieces of information: 1) your current directory and 2) the path relative to that directory. Relative paths are really helpful because things don’t break when you move your folder or files.

If my current working directory is the directory /fh/fast/laderas_t/immuno_project/, then the relative path to that same file would be:

raw_data/chr2.fa.gz

We can visualize the relative path like this, where our working directory is indicated by a star:

1/
2├── fh/fast/laderast/immuno_project/
3|                                   └──raw_data
│                                      └──chr2.fa.gz
1
The root directory
2
Our working directory
3
Our relative path

Note that this relative path does not start with a /, because our current directory isn’t the root directory. Relative paths are incredibly useful when scripting in a reproducible manner, such as using project-based workflows to process files in a single folder.

None<TAB> is for autocompletion of paths

Never underestimate the usefulness of the <TAB> key, which triggers autocompletion on the command line. It can help you complete paths to files and save you a lot of typing.

For example, say I have a path that I want to navigate to

/home/tladera2/my_long_path

I can type in the first part of the path and then hit <TAB>:

/home/tladera2/my_<TAB>

And if the prefix my_ is unique in my folder, it will autocomplete the path:

/home/tladera2/my_long_path

Note that we need to use enough of the folder name so that completing it is unambiguous. If there are multiple choices, then autocomplete will list all of them.

1.10 File Permissions

File permissions are that are attached to file objects. They are how the system prevents certain files from being modified or restricting access of these files to certain people or groups.

All files have the following level of access permissions:

Level Description
Owner-level The owner of the file
Group-level The group of the file
Everyone The rest of the world

For example, if I’m the owner of the file, I can restrict the type of access to only myself (owner-level), the group I’m in (Group-level), or make the file freely available to everyone on the system (Everyone).

Each level has the following type of access:

Type Description Abbreviation Example
Read Level can only read contents of file r A list of users in a text file
Write Level can write to the file w Appending an entry to the end of a log
Execute Level can run the file as an executable x samtools

You can see the permissions for a file using the ls -l <FILENAME>. For example:

ls -l -h .

will give me the following line:

-rw-rw---- 1 tladera2 g_tladera2 1.8K Oct  2 15:21 01_assignment.qmd
-rw-rw---- 1 tladera2 g_tladera2  21K Oct  2 15:21 01_basics.qmd
-rw-rw---- 1 tladera2 g_tladera2 2.3K Oct  2 15:21 02_assignment.qmd
-rw-rw---- 1 tladera2 g_tladera2  21K Oct  2 15:21 02_scripting.qmd
-rw-rw---- 1 tladera2 g_tladera2 1.2K Oct  2 15:21 03_assignment.qmd
-rw-rw---- 1 tladera2 g_tladera2  13K Oct  2 15:21 03_batch.qmd
-rw-rw---- 1 tladera2 g_tladera2    0 Oct  2 15:21 04_assignment.qmd

It’s this first part that we want to examine:

folder  everyone_else
|       |     
-rw-rw----
 |  |
 you group

The 2nd to 4th letters define your own access to the files. The

rw-

Means that you have read and write access. If we had the following:

rwx

That would mean we have read, write, and execute access.

Similarly, the 5th to 7th letters are the permissions for your group. In many cases at Fred Hutch, the group level maps to your lab or group.

The last 3 letters correspond to everyone else on the system. Think about whether you want to make your scripts accessible to everyone.

In this case, since the permissions are:

---

That means that everyone else cannot read, write, or execute your files. Your files are locked down.

The cardinal rule to remember is that:

If you want to run a file as an executable, you (or your group) needs to have executable level permission.

For example, if I want to run a script called run_samtools.sh in my directory like this:

./run_samtools.sh my_bam_file.bam

I will need to have execute privileges at the user, group, or others level for run_samtools.sh.

We can change the permissions of our files using the chmod command.

NoneHelpful unix permissions situations

I tend to just go by memory when setting file permissions. These are the common situations that I’ve encountered that I use regularly:

Situation Command
Only I can execute/read/write a file chmod 700 <filename>
Only I and my group can read a file chmod 110 <filename>
Only I and my group can read/write/execute a file chmod 770 <filename>
Grant my group read permissions chmod 710 <filename>
Make executable/read/write by all chmod 777 <filename>

For right now, let’s set scripts to be read/write/executable by us:

chmod -R 700 scripts/

The -R in this case is the recursive argument, which means apply chmod to everything within the scripts/ folder.

This will save us some headaches when we try to execute our scripts.

If you have different permission situations that you need to set, please refer to: https://www.redhat.com/en/blog/linux-file-permissions-explained

NoneEven if you don’t have execute permissions

With bash scripts, you can still run them if you have read permissions. You can still run bash scripts by using the bash command:

bash run_samtools.sh my_bam_file.bam

This is also the case for scripts that use she-bangs (Section 3.4) for R or Python or any other executable.

1.10.1 Try it out

What are the permissions for the GitHub repo (bash_for_bio) that you just downloaded?

1.11 Moving Things Around

A lot of the time, we need to move files between shared filesystems. One filesystem might be good at storage and be backed up on a regular basis, while another filesystem might be better for temporary work on the cluster.

You might be familiar with mv, which lets you move files around in Unix. One thing to keep in mind when you’re mving things to a new folder that there is a difference between:

mv log.txt my_folder   ## renames log.txt to my_folder

and

mv log.txt my_folder/  ## moves log.txt to be in my_folder

This is one thing that still trips me up all the time.

This is one situation where using a GUI such as Motuz can be very helpful. You don’t have to worry about accidentally renaming files.

Other tools for sync’ing between filesystems include rsync, which requires careful reading of documentation.

NoneThings I always forget: the difference between /home/mydir/ and home/mydir/

Some things that trip me up all the time. The difference between

/home/mydir/    #absolute path

and

home/mydir/     #relative path

The first one is an absolute path, and the second is a relative path. Your clue is the leading / at the beginning of a path. If you’re getting file not found messages, check to make sure the path is the right format.

1.11.1 Keep Everything in Folders

We need to talk about code and data organization. For the FH system, we have a /home/ directory, and if we have generated research data, a /fh/fast/ directory. If we want our scripts to live in /home/ and our data is in /hpc/temp/, we’ll need to refer to each of these file locations.

Ideally, we want to make the naming conventions of our code and our data as similar as possible.

NoteTry it Out

Copy the script tell_the_time.sh in the scripts/week1/ directory to the top directory of bash_for_bio.

Make the script executable.

1.12 What’s in the script

We can see what’s in the script by using cat:

cat tell_the_time.sh

And you’ll get the following:

#!/bin/bash
date

1.13 Running a Bash Script

Ok, now we have a bash script tell_the_time.sh in our current directory, how do we run it?

Because the script is not on our $PATH (Section 11.5.2), we’ll need to use ./ to execute it. ./ is an alias for the current folder, and it is an indicator to bash that the command we want to execute is in our current folder.

tladera2$ ./tell_the_time.sh

If we haven’t set the permissions (Section 1.10) correctly, we’ll get this message:

bash: ./scripts/tell_the_time.sh: Permission denied

But if we have execute access, we’ll get something like this:

Fri Jul 11 13:27:47 PDT 2025

Which is the current date and time.

1.14 Editing on a Linux Machine

On the rhino machines, we have the option to use the nano editor. nano is the most like a word processor or code editors.

  • Open a file in nano: nano <filename>
  • Save and quit: <CTRL> + x and then yes
  • Navigate in file: using the arrow keys will work
  • Find in file: <CTRL> + w
  • Copy from outside the terminal (dependent on terminal program)

Pasting will depend on your terminal program. On macs it is Command-V, and on PuTTY it is using Right Click on the mouse

1.14.1 Try it Out

Try making your own file called my_file.txt:

nano my_file.txt

Add some text to it.

Use CTRL-X to exit, and make sure to select “Yes” to save.

1.14.2 Editing on your own machine

One thing you can do to make your life easier is to edit scripts with an editor on your own machine that is connected to rhino through Samba. You will able to open up files you create in the editor of you choice on your system, and when you save the file, the changes will be make on rhino.

Instructions are here: https://sciwiki.fredhutch.org/scicomputing/store_posix/#how-to-access-fred-hutch-storage and follow the instructions for mapping your network drive (Mac) and (Windows).

You will want to use the connection strings specified for home.

1.15 Running an R or Python Script on the command line

1.15.1 Loading the fhR or fhPython modules

Before we can run our scripts in R or Python, we’ll need to load up either R or Python on the cluster. We can do this with the module load command:

1module load fhR
2module load fhPython
1
Load up fhR module - has R and most packages installed
2
Load up fhPython module - has Python and most packages installed.

We’ll talk more about software modules next week (Section 3.5).

1.15.2 R Users

You might not be aware that there are multiple ways to run R:

  1. as an interactive console, which is what we usually use in an IDE such as RStudio
  2. on the command line using the Rscript command.
Rscript my_r_script.R

To run this script, we’ll need to first load fhR:

module load fhR
Rscript my_r_script.R
module purge

1.15.3 Python Users

Python users are much more aware that you can run Python scripts on the command line:

python3 my_python_script.py

To execute this on gizmo, we’ll first need to load fhPython:

module load fhPython
python3 my_python_script.py
module purge

Within a shell script, you can also use a shebang (Section 3.4) to make your script executable by providing the location of python3:

#!/bin/python3
python3 my_python_script.py

1.16 Getting Help

You may have heard about man pages. You can usually get help by using the man command:

man wc

This is the first part of the output:

NAME
     wc – word, line, character, and byte count

SYNOPSIS
     wc [--libxo] [-Lclmw] [file ...]

DESCRIPTION
     The wc utility displays the number of lines, words, and bytes contained in each input file, or standard input (if no file is
     specified) to the standard output.  A line is defined as a string of characters delimited by a ⟨newline⟩ character.
     Characters beyond the final ⟨newline⟩ character will not be included in the line count.

     A word is defined as a string of characters delimited by white space characters.  White space characters are the set of
     characters for which the iswspace(3) function returns true.  If more than one input file is specified, a line of cumulative
     counts for all the files is displayed on a separate line after the output for the last file.

     The following options are available:

         --libxo
             Generate output via libxo(3) in a selection of different human and machine readable formats.  See xo_parse_args(3)
             for details on command line arguments.

     -L      Write the length of the line containing the most bytes (default) or characters (when -m is provided) to standard
             output.  When more than one file argument is specified, the longest input line of all files is reported as the value
             of the final “total”.

I personally find man pages very hard to read, especially when there are lots of options for a command.

Instead, I use tldr, which contain examples of the most commonly used options in a command. It is not installed on gizmo, but you can use the page at https://tldr.inbrowser.app/, which has all of the tldr help pages.

1.17 Recap

We learned the following this week:

  • Navigate and copy data to the different filesystems available at Fred Hutch.
  • Explain the difference between absolute and relative file paths.
  • Set Permissions on and execute a bash script
  • Find help on the system

1.18 Next Week

We’ll focus on adding arguments to our scripts, and more about software modules