Creating a Singularity image when Docker is misbehaving

Today’s task was to update a docker image by adding some data, committing the changes to the Docker-Hub and creating image files for Docker and for Singularity (version 2.4-dist).
I’m usually performing these tasks on an Ubuntu machine on the AWS cloud. Trying to follow my steps from a previous time, I first start my container and perform the update within:

sudo docker run -it --rm -m 4g --mount type=bind,source=/home/ubuntu,target=/tmp  --name projectname-1 $DOCKER_ID_USER/projectname:2 /bin/bash
# ...

Checking the container ID and committing and pushing the changes to Docker-Hub all seems fine:

sudo docker ps
sudo docker commit 6a024ac35ab5 $DOCKER_ID_USER/projectname:2
sudo docker push $DOCKER_ID_USER/projectname:2

But trying to pull down the image using Docker or Singularity fails repeatedly:

sudo docker pull $DOCKER_ID_USER/projectname:2
invalid reference format
singularity pull docker://$DOCKER_ID_USER/projectname:2
Importing: base Singularity environment
Importing: /home/ubuntu/.singularity/docker/sha256:223cbef2a1193a2c9ab9dac0195ff0dcbbe2067e724f46a5fbe8473dda842b71.tar.gz
gzip: /home/ubuntu/.singularity/docker/sha256:223cbef2a1193a2c9ab9dac0195ff0dcbbe2067e724f46a5fbe8473dda842b71.tar.gz: not in gzip format
tar: This does not look like a tar archive
tar: Exiting with failure status due to previous errors

The name and format of my project have not changed however! After some head-scratching I found that I could actually perform the tasks of creating the Singularity images on my local machine (an Apple MacBook Pro running High Sierra), following this Singularity page:

cd /Users/fsk/singularity-vm/
vagrant destroy
rm Vagrantfile
vagrant up --provider virtualbox

Checking with singularity –version shows version 2.4-dist is running successfully!
Pulling with Singularity nicely creates my Singularity img file from the Docker image:

sudo singularity pull docker://$DOCKER_ID_USER/projectname:2
Singularity container built: ./projectname-2.img

What are the alternatives?

  • Using docker2singularity created a file twice the size and was therefore not useful!
  • Re-building from a Docker file or from a Singularity recipe file would have been very tedious, but would probably work.

To produce the Docker images, I was able to export from the running container on the Ubuntu machine:

sudo docker export 6v034ac35ab5 | gzip > projectname-2.tar.gz

All is good…