Release 2.3.2 This dot release includes a fix for a change that Docker implemented to their registry RESTful API which broke compatibility with Singularity (among several other low minor fixes).
Where does Docker image metadata come from? The Docker Registry serves metadata about images via manifests, where each image has a manifest that tells Singularity important information like environment, labels, and entrypoints and running command. Importantly, the image manifest serves the actual layers that should be obtained to create the image. The manifests come in different flavors, or schema versions. Version 1 serves the primary load of metadata (labels, environment) while the first release of version 2 served layers, and had the addition of size. This addition made it possible to pull an image with Singularity and calculate the size on the fly for images generated with support for this manifest.
Singularity had an old default to retrieve the version 2 (done by way of a header asking for it), and ask it for layers. If the remote registry could only offer version 1, we were still able to obtain a list of layers (under a different key,
fsLayers instead of
layers) and metadata via the older (schema 1) manifest. However, with the update, the API version 2 schema can now return a list of manifests. This meant that when we checked the response for the keys
fslayers, they wouldn’t be found, because we needed to look under
manifests, and then do a second call to retrieve the manifest of interest based on a system architecture and OS. This of course broke most
exec, because all of these commands require retrieving the layers.
A super quick fix would have been to fall back to the version 1 manifest, always, but then we would lose the automatic calculation of size, which is important to many users. The “better” fix is obvious - the code needs to:
- check for a
- if found, select a default manifest to use
- retrieve it, and continue!
- if not found, fall back to version 1
This means that we’ve added environment variables
SINGULARITY_DOCKER_ARCHITECTURE to specify this choice, with defaults
amd64. This is a pretty exciting change, because it means down the line you (as a user!) can specify the specifics of the layers you want returned, given an image that has this metadata available. We can see the fix running as follows:
DEBUG *** STARTING DOCKER IMPORT PYTHON ****
This is the initialization of the Docker client, it’s parsing the image name provided
DEBUG Starting Docker IMPORT, includes env, runscript, and metadata. VERBOSE Docker image: ubuntu:14.04 VERBOSE2 Specified Docker ENTRYPOINT as %runscript. DEBUG Headers found: Content-Type,Accept VERBOSE Registry: index.docker.io VERBOSE Namespace: library VERBOSE Repo Name: ubuntu VERBOSE Repo Tag: 14.04 VERBOSE Version: None
The initial poke to Docker hub asks for tags, to determine if we need some kind of token
VERBOSE Obtaining tags: https://index.docker.io/v2/library/ubuntu/tags/list DEBUG GET https://index.docker.io/v2/library/ubuntu/tags/list
401 means that we do, and with the response the API sends back the scope and other details we need to make to request it
DEBUG Http Error with code 401
Here we are requesting it
DEBUG GET https://auth.docker.io/token?service=registry.docker.io&expires_in=9000&scope=repository:library/ubuntu:pull DEBUG Headers found: Content-Type,Authorization,Accept
And here is the new bit. Instead of blindly doing one call, we have a function to update two versions of the manifest - the older (with metadata) and some version of the newer (with layers and size) with a fallback to use the version 1
DEBUG Updating manifests. # Here is version 2+ DEBUG MANIFEST (Primary): not found, making initial call. VERBOSE Obtaining manifest: https://index.docker.io/v2/library/busybox/manifests/latest DEBUG GET https://index.docker.io/v2/library/busybox/manifests/latest # Here is version 1 (Metadata) DEBUG MANIFEST (Metadata): not found, making initial call. VERBOSE Obtaining manifest: https://index.docker.io/v2/library/busybox/manifests/latest DEBUG GET https://index.docker.io/v2/library/busybox/manifests/latest
Notice that the two calls are the same - that’s because the headers are actually different, to request different ones.
And here is the (new) indication that we found a list
DEBUG Image manifest version 2.2 list found.
Here is the default architecture / os
DEBUG Obtaining architecture: amd64, OS: linux
And the specific call to get it
VERBOSE Obtaining manifest: https://index.docker.io/v2/library/busybox/manifests/sha256:030fcb92e1487b18c974784dcc110a93147c9fc402188370fbfd17efabffc6af DEBUG GET https://index.docker.io/v2/library/busybox/manifests/sha256:030fcb92e1487b18c974784dcc110a93147c9fc402188370fbfd17efabffc6af
With this fix - we can again use
import, etc, and also have a working
singularity pull docker://busybox that estimates the size automatically.
Please report any additional bugs to our issues board.