Singularity is an open-source computer program that allows for applications to be deployed on Linux as containers: lightweight, stand-alone, and executable packages for an application, containing everything that the application needs to run. Similar to Docker, Singularity allows for the creation of containers from a simple configuration file called a definition, or ready-made from an online repository. Singularity allows for the deployment of software containers on CCAST, allowing users to quickly deploy software locally and also to utilize features such as GPU hardware acceleration.

While the full Singularity user documentation is available here, this document is a brief overview of Singularity and its use on CCAST.

Running prebuilt containers on CCAST

To use Singularity on CCAST, load the singularity module:

module load singularity

Then use the singularity command to interact with Singularity. For example, to see the available commands, use the singularity help command:

singularity help

To see the help for a specific command, use the singularity help <command> command:

singularity help pull

Singularity interacts with containers, which appear on the filesystem as .sif files. Users can pull an existing container from an online repository or build one themselves.

Downloading a Pre-built Singularity Container

Pulling (i.e. downloading) a Singularity container copies a ready-made container from an external, internet-based source. To pull a Singularity container, use the singularity pull command after loading the singularity module:

singularity pull <container>

Here, the can be either a container name or a container name with a tag. For example, to pull the container ubuntu:latest from the Official Docker Hub Source, use the command:

singularity pull docker://ubuntu:latest

This creates a local, ready-to-run containerized version of the Ubuntu distribution, named ubuntu_latest.sif by pulling it from Docker Hub.

Finding Containers

Many online repositories of containers exist, including:

• Docker Hub
• Nvidia GPU Cloud
• BioContainers
• Singularity Hub (archive only)
• Singularity Github Examples

Running a Singularity Container

To run a Singularity container, use the singularity exec command:

singularity exec <container> <command>

This runs the command <command> inside the container <container>. For example, using the previously-created container:

singularity exec ubuntu_latest.sif cat /etc/lsb-release

This runs cat /etc/lsb-release inside the container ubuntu_latest.sif, which prints the version of Ubuntu inside the container.

Running an Interactive Shell

To run an interactive shell inside a container, use the singularity shell command:

singularity shell <container>

This runs an interactive shell inside the container <container>.

Example: PBS scripting Singularity

Singularity containers can be run from within PBS scripts, and thus leverage the power of GPU acceleration for HPC jobs. The entire workflow of setting up and running can be performed using the CCAST scheduler. For example, the following PBS script runs a containerized version of the gromacs molecular dynamics package to perform a simulation of human alcohol dehydrogenase solvated in water, using 4 GPUs on CCAST:

#!/bin/bash
#PBS -q gpus
#PBS -N gromacs-test
#PBS -l select=1:mem=16GB:ngpus=1:ncpus=8:mpiprocs=4:ompthreads=2
#PBS -l walltime=00:30:00
#PBS -j oe
#PBS -W group_list=<YOUR GROUP HERE>

## Load the singularity module
module load singularity
module load wget

## Change to the scratch directory and create a working directory
cd $SCRATCH
mkdir gromacs-test
cd gromacs-test

## Copy the input files to the working directory
wget https://zenodo.org/record/3893789/files/GROMACS_heterogeneous_parallelization_benchmark_info_and_systems_JCP.tar.gz
tar -xzf ./GROMACS_heterogeneous_parallelization_benchmark_info_and_systems_JCP.tar.gz GROMACS_heterogeneous_parallelization_benchmark_info_and_systems_JCP/adh_dodec/ --strip-components=2

## pull the container
singularity pull docker://nvcr.io/hpc/gromacs:2022.3

## Run the container
SIF=$(pwd)/gromacs_2022.3.sif
APP="singularity run --nv -B ${PWD}:/host_pwd --pwd /host_pwd ${SIF}"
${APP} gmx grompp -f pme_verlet.mdp
${APP} gmx mdrun -v -nsteps 100000 -resetstep 90000 -noconfout -ntmpi 4 -ntomp 2 -nb gpu -bonded gpu -pme gpu -npme 1 -nstlist 400 -s topol.tpr

## Copy the output files back to the home directory
cd ..
cp -r gromacs-test $HOME

## Clean up
rm -rf gromacs-test
exit 0

Installing Singularity on your local machine

To create custom singularity containers for deployment onto CCAST, you will need to install Singularity on your local machine. While Singularity is Linux software, it can be installed on Mac OSX and Windows using a virtual machine. To install Singularity on your local machine, follow the instructions here. In brief, you can get Singularity working on your machine with the following:

Linux

Singularity is easiest installed via the command line with the following commands, depending on your distribution:

# Red Hat, Rocky Linux, CentOS, and Fedora
$ sudo yum install -y epel-release
$ sudo yum update
$ sudo yum install singularity-ce
# Ubuntu, Debian, Mint, and other Debian-based distributions
$ sudo apt install -y software-properties-common build-essential \
    uuid-dev libgpgme-dev squashfs-tools libseccomp-dev wget pkg-config git cryptsetup-bin golang
$ wget https://github.com/sylabs/singularity/releases/download/v3.11.3/singularity-ce_3.11.3-jammy_amd64.deb | sudo dpkg -i -
# Arch, Manjaro, Artix, and other Arch-based distributions
# using yay
$ yay -S singularity-ce
# using pkgbuild
$ git clone https://aur.archlinux.org/singularity-ce.git
$ cd singularity-ce && makepkg -si

Windows

While Singularity is not available for Windows, it can be installed in a Linux virtual machine. Two options are available:

• Vagrant and VirtualBox
• Windows Subsystem for Linux (WSL)

Vagrant and VirtualBox

Vagrant is a tool for managing virtual machines, and VirtualBox is a virtual machine manager. Vagrant and Virtualbox can be easily installed on Windows using the Windows Package Manager (winget) using powershell:

> winget install virtualbox
> winget install vagrant

Then, to install a Linux virtual machine, run the following commands:

> vagrant.exe init ubuntu/latest
> vagrant.exe up

The above would install the latest version of Ubuntu. To access the virtual machine, run the following command:

> vagrant.exe ssh default

You can then install Singularity on the virtual machine using the Linux instructions above.

Windows Subsystem for Linux (WSL)

Windows Subsystem for Linux (WSL) is a compatibility layer for running Linux natively on Windows. To install WSL on Windows, open a Powershell terminal and run the following commands:

> dism.exe /online /enable-feature /featurename:Microsoft-Windows-Subsystem-Linux /all /norestart
> wsl --install

This first enables WSL, then installs the software. Then, to install a Linux virtual machine, run the following commands:

> wsl --list --online
> wsl --install -d Ubuntu

The above would install the latest version of Ubuntu. To access the virtual machine, run the following command:

> wsl -d Ubuntu

You can then install Singularity on the virtual machine using the Linux instructions above.

Mac OSX

Similarly to Windows, Singularity is not available for Mac OSX, it can be installed in a Linux virtual machine. The easiest method is to use homebrew to install VirtualBox and Vagrant using the following commands in the terminal:

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
brew install --cask virtualbox vagrant

Then, Vagrant can be used to install a Linux virtual machine, and Singularity can be installed on the virtual machine using the Linux instructions above.

Example: Creating Singularity Containers

Singularity containers can be created from a definition file, which is a simple text file containing instructions for building the container, including the installation of software. For example, the following definition file creates a container with the latest version of Ubuntu, then installs the neovim text editor:

Bootstrap: docker
From: ubuntu:latest

%post
    apt update
    apt install -y neovim

%environment
    export EDITOR=nvim
    export LC_ALL=C

%runscript
    exec nvim "$@"

To build this container, save the above text to a file named neovim.def, then use the singularity build command:

singularity build neovim.sif neovim.def

This creates a container named neovim.sif from the definition file neovim.def, and creates a default action which can be run for the container. To run the container, use the singularity run command:

singularity run neovim.sif