8  Reading: JSON/Container Basics

8.1 What is JSON?

One requirement for running workflows is basic knowledge of JSON.

JSON is short for JavaScript Object Notation. It is a format used for storing information on the web and for interacting with (APIs).

8.1.1 How is JSON used?

JSON is used in multiple ways:

  • Submitting Jobs with complex parameters/inputs

So having basic knowledge of JSON can be really helpful. JSON is the common language of the internet.

8.1.2 Elements of a JSON file

Here are the main elements of a JSON file:

  • Key:Value Pair. Example: "name": "Ted Laderas". In this example, our key is “name” and our value is “Ted Laderas”
  • List [] - a collection of values. All values have to be the same data type. Example: ["mom", "dad"]
  • Object {} - A collection of key/value pairs, enclosed with curly brackets ({}).
NoteCheck Yourself

What does the names value contain in the following JSON? Is it a list, object or key:value pair?

{
  "names": ["Ted", "Lisa", "George"]
}

It is a list. We know this because the value contains a [].

{
  "names": ["Ted", "Lisa", "George"]
}

8.1.3 JSON Input Files

When you are working with WDL, it is easiest to manage files using JSON files. Here’s the example we’re going to use from the ww-fastq-to-cram workflow.

#| eval: false
#| filename: "json_data/example.json"
{
  "PairedFastqsToUnmappedCram.batch_info": [
    {
      "dataset_id": "TESTFASTQ1",
      "sample_name": "HG02635",
      "library_name": "SRR581005",
      "sequencing_center": "1000-Genomes",
      "filepaths": [{
        "flowcell_name": "20121211",
        "fastq_r1_locations": ["tests/data/SRR581005_1.ds.fastq.gz"],
        "fastq_r2_locations": ["tests/data/SRR581005_2.ds.fastq.gz"]
      }]
    },
    {
      "dataset_id": "TESTFASTQ2",
      "sample_name": "HG02642",
      "library_name": "SRR580946",
      "sequencing_center": "1000-Genomes",
      "filepaths": [{
        "flowcell_name": "20121211",
        "fastq_r1_locations": ["tests/data/SRR580946_1.ds.fastq.gz"],
        "fastq_r2_locations": ["tests/data/SRR580946_2.ds.fastq.gz"]
      }]
    }
  ]
}

This might seem overwhelming, but let’s look at the top-level structures first:

1{
2  "PairedFastqsToUnmappedCram.batch_info": [
   ...   
  ]
}
1
The top level of the file is a JSON object
2
The next level down (“PairedFastqsToUnmappedCram.batch_info”) is a list.

This workflow specifies the file inputs using the PairedFastqsToUnmappedCram.batch_info object, which is a list.

Each sample in the PairedFastqsToUnmappedCram.batch_info list is its own object:

  "PairedFastqsToUnmappedCram.batch_info": [
    {
      "dataset_id": "TESTFASTQ1",
      "sample_name": "HG02635",
      "library_name": "SRR581005",
      "sequencing_center": "1000-Genomes",
      "filepaths": [{
        "flowcell_name": "20121211",
        "fastq_r1_locations": ["tests/data/SRR581005_1.ds.fastq.gz"],
        "fastq_r2_locations": ["tests/data/SRR581005_2.ds.fastq.gz"]
      }]
    },
    ....

Because we are aligned paired-end data, notice there are two keys, fastq_r1_locations and fastq_r2_locations.

8.2 Containers

We already learned about software modules (Section 3.5) on the gizmo cluster. There is an alternative way to use software: pulling and running a software .

8.2.1 What is a Container?

A container is a self-contained unit of software. It contains everything needed to run the software on a variety of machines. If you have the container software installed on your machine, it doesn’t matter whether it is MacOS, Linux, or Windows - the container will behave consistently across different operating systems and architectures.

The container has the following contents:

  • Software - The software we want to run in a container. For bioinformatics work, this is usually something like an aligner like bwa, or utilities such as samtools
  • Software Dependencies - various software packages needed to run the software. For example, if we wanted to run tidyverse in a container, we need to have R installed in the container as well.
  • Filesystem - containers have their own isolated filesystem that can be connected to the “outside world” - everything outside of the container. We’ll learn more about customizing these with bind paths (Section 10.2.3).

In short, the container has everything needed to run the software. It is not a full operating system, but a smaller mini-version that cuts out a lot of cruft.

Containers are . They leverage the the file system of their host to manage files. These are called both Volumes (the Docker term) and Bind Paths (the apptainer term).

8.2.2 Docker vs. Apptainer

There are two basic ways to run Docker containers:

  1. Using the Docker software
  2. Using the Apptainer software (for HPC systems)

In general, Docker is used on systems where you have a high level of access to the system. This is because docker uses a special user group called docker that has essentially root level privileges. This is not something to be taken lightly.

This is not the case for HPC systems, which are shared and granting this level of access to many people is not practical. This is when we use (which used to be called Singularity), which requires a much lower level of user privileges to execute tasks. For more info, see Section 10.2 .

WarningBe Secure

Before we get started, security is always a concern when running containers. The docker group has elevated status on a system, so we need to be careful that when we’re running them, these containers aren’t introducing any system vulnerabilities. Note that on HPC systems, the main mechanism for running containers is apptainer, which is designed to be more secure.

These are mostly important when running containers that are web-servers or part of a web stack, but it is also important to think about when running jobs on HPC.

Here are some guidelines to think about when you are working with a container.

  • Use vendor-specific Docker Images when possible.
  • Use container scanners to spot potential vulnerabilities. DockerHub has a vulnerability scanner that scans your Docker images for potential vulnerabilities. For example, the WILDS Docker library employs a vulnerability scanner and the containers are regularly patched to prevent vulnerabilities.
  • Avoid kitchen-sink images. One issue is when an image is built on top of many other images. It makes it really difficult to plug vulnerabilities. When in doubt, use images from trusted people and organizations. At the very least, look at the Dockerfile to see that suspicious software isn’t being installed.

8.2.3 The WILDS Docker Library

The Data Science Lab has a set of Docker containers for common Bioinformatics tasks available in the WILDS Docker Library. These include:

  • bwa mem
  • samtools
  • gatk
  • bcftools
  • manta
  • cnvkit
  • deseq2

Among many others. Be sure to check it out before you start building your own containers.

8.2.4 Pulling a Docker Container

Let’s pull a docker container from the Docker registry. Note we have to specify docker:// when we pull the container, because Apptainer has its own internal format called SIF.

module load Apptainer/1.1.6
apptainer pull docker://ghcr.io/getwilds/scanpy:latest
apptainer run --bind /path/to/data:/data,/path/to/script:/script docker://getwilds/scanpy:latest python /script/example.py

8.2.5 Bind Paths

One thing to keep in mind is that containers have their own filesystem. They can only read and write to folders in the external filesystem that you give them access to with bind paths. The one exception is the current working directory.

For more info about bind paths see Section 10.2.3.

8.3 Glossary

Term Definition