So you want to put your scientific analysis in a nice package and run it on a cluster? You’ve probably heard of the container technology called Docker?
…too bad you can’t use it on your research cluster, because it introduces huge security issues. You’ve probably also heard of virtual machines, but most clusters won’t let you run those either. What options does this leave us? Oh wait, duh, you are reading this website. You already know the answer to this question.
The easiest thing to do is to install Singularity on your local workstation:
sudo apt-get update sudo apt-get -y install build-essential curl git sudo man vim autoconf libtool git clone https://github.com/singularityware/singularity.git cd singularity ./autogen.sh ./configure --prefix=/usr/local make sudo make install
If you are using a Mac, or just need a virtual machine, then you will want to follow the instructions here. Basically, you need to install vagrant, virtual box, and then do this:
vagrant init ubuntu/trusty64 vagrant up vagrant ssh -c /bin/sh <<EOF sudo apt-get update sudo apt-get -y install build-essential curl git sudo man vim autoconf libtool git clone https://github.com/singularityware/singularity.git cd singularity ./autogen.sh ./configure --prefix=/usr/local make sudo make install EOF vagrant ssh
Once you are in your Virtual Machine, or have Singularity up and running? Well, it’s time to go NUTS of course!
A little about Singularity Hub
Singularity Hub is an online registry for images. This means that you can connect a GitHub repo containing a build specification file to this website, and the image is going to build for you automatically, and be available programmatically! We can talk more about how that happens later. If you want some quick details, you should check out the Usage Docs on Singularity Hub.
Make and run containers
Run an image
For this little preview, we are going to be first running an image, directly from Singularity Hub. This image is called vsoch/singularity-images and it’s associated with the equivalent GitHub repository.
singularity run shub://vsoch/singularity-images
In the above, we use the Singularity Hub “unique resource identifier,” or
shub:// which tells the software to run an image from Singularity Hub.
Create an image
Running is great, but what if we want to mess around on the command line, using an image we’ve created ourselves? We can do that by creating an image:
sudo singularity create analysis.img sudo singularity import analysis.img docker://ubuntu:latest singularity shell analysis.img
In the above, we use the docker “unique resource identifier,” or
docker:// which tells the software to import a docker image.
If we wanted to shell into the image and make it writable, meaning that we can write files and save changes, we would do this:
sudo singularity shell --writable analysis.img
Note that we need sudo, and also note that you wouldn’t be able to do this on a research cluster, because you don’t have sudo.
Create a reproducible image
The problem with create an image, and then maybe writing stuff to it with
--writable is that your work isn’t properly saved anywhere. You COULD ship and share the entire image, but that still doesn’t help to say what was done to it, and this is problematic. To help with this, we encourage you to create a build specification file, a file called
Singularity. There are a few important sections you should know about. First, let’s look at a very simple file:
Bootstrap: docker From: ubuntu:latest %runscript echo "I can put here whatever I want to happen when the user runs my container!" exec echo "Hello Monsoir Meatball" "[email protected]" %post echo "Here we are installing software and other dependencies for the container!" apt-get update apt-get install -y git
The important things to note. The header section says that we want to
Bootstrap a docker image, specifically
From ubuntu:latest. No, you don’t actually need Docker installed to run this, because the layers are pulled from their API endpoint.
Once you have your bootstrap file, and you know how to use GitHub, you are really good to go. You can add the file to repository, connect it to Singularity Hub, and it will build automatically and be available via the
shub:// endpoint. That’s it!