Packaging docker for guix

For better or worse, not all software that I want to use is packaged in guix. Lots of self-hosted applications are distributed primarily as docker containers, or in the most complex case, as a complete docker-compose environment.

This release (unless I'm misreading something) does not even turn up in the docker release notes for version 20.10, and predates several important changes, especially the rewrite to Go and the reimplementation of docker-compose as a subcommand of docker.

This article notes down everything I had to learn to package a recent version of docker.

The source material

To package stuff, you have to know how it works, and what pieces it's made of. Everything written below is what I think, going into writing this. I might find out new stuff as we go.

Docker consists primarily of two components: docker engine and the docker cli. Both are distributed as single binaries. Docker engine is the binary that runs as a daemon when you run docker as a service, and docker cli is the binary you run when you issue commands to the docker engine. I expect docker engine to be far more complicated than docker cli. I have seen other components mentioned, including "BuildKit", "containerd" and "runc", but I don't know what they mean.

The engine has been generalised and made to work with other "container runtimes". The source code for docker engine is therefore at moby/moby on GitHub, while the CLI is at docker/cli.

Docker's build system is docker itself. The project distributes dockerfiles, which can be built to produce the binaries which are actually released. Their PACKAGERS.md file just defers to the dockerfile itself:

Build dependencies

The Dockerfile contains the most up-to-date list of build-time dependencies.

This has obvious problems for bootstrapping. What if I live on a system that does not have docker already, or that has an old version of docker that can't build recent versions?^[1] How can I "prove" to users that my docker binary is actually authentic? What version of docker did the upstream use to build their new docker version, and how do they know the build does not depend on the version of docker used?

Identifying build dependencies

Starting with moby, which is the most significant piece, it contains a Dockerfile, but as far as I can tell no actual instructions on how to use it. So, "it is what it does", and we will just have to read it line by line to identify the things it builds and how. Hypothetically, we could just build the container and try to peek around inside it to guess which things are used and needed, without understanding the high-level plan. I am not sure at the moment how to tell which things are relevant, though, without inspecting each of the docker layers (and I don't know how to do that).

This is a multi-stage Dockerfile, meaning that it creates several containers and copies files between them in order to produce the "final" image. I am used to just running docker build . in order to build a dockerfile, but there is a comment that suggests otherwise:

# usage:
# > docker buildx bake binary
# > DOCKER_STATIC=0 docker buildx bake binary
# or
# > make binary
# > make dynbinary
FROM scratch AS binary
COPY --from=build /build/ /

Ok. I switched to a laptop that has Debian, and installed docker in the default way. The given command does run, it takes over ten minutes. It looks like buildx is just a souped-up version of build that does parallel builds and lets you have multiple build outputs. It outputs a binary ./bundles/binary/dockerd which I can run with --verison.

My immediate feeling is that this Dockerfile seems huge and incredibly complicated. I can see nearly a hundred inter-container dependencies between FROM and COPY commands. There is a good chance that a lot of it will not be relevant, too. I had a go at running dockerfilegraph on the project, and I get the following graph:

IMAGE OF GRAPH

This makes it much easier to follow which containers are required by binary. It looks like the reduced dependency list is just:

xx -> base base -> build -> binary base -> gowinres -> build -> binary

with the "base" image being built very early as a dependency of many many other containers. I can't see any redefinition of any of these containers (I am not even sure if this is allowed). It follows that we should be able to delete anything between FROM instructions that do not refer to one of these very few containers. This should work since we went via the graph, which considers COPY and MOUNT dependencies, which I would not obviously detect by reading.

Doing this deletion leaves me with a 150 line Dockerfile with only 7 containers declared and 9 commands run (although the commands are not themselves trivial). This still builds, and creates a binary that reports its version correctly, but with a changed sha-sum (which could be a bad fact or a neutral fact). The resulting graph is very easy to read:

ABBREVIATED GRAPH

This is not the full dependency graph, since the Dockerfile contains several calls to apt-install. We'll need to make sure all the installed packages are also available in guix, although there is a good chance that they are. We will now need to look at each of the dependencies in detail.

The xx container

xx provides tools to support cross-compilation from Dockerfiles that understand the --platform flag passed in from docker build or docker buildx build. These helpers allow you to build multi-platform images from any architecture into any architecture supported by your compiler with native performance.

xx currently contains xx-info, xx-apk, xx-apt-get, xx-cc, xx-c++, xx-clang, xx-clang++, xx-go, xx-cargo, xx-verify. xx-clang (and its aliases) creates additional aliases, eg. ${triple}-clang, ${triple}-pkg-config, on first invocation or on xx-clang --setup-target-triple call.

If this is all just about cross-compilation, we should be able to completely ignore it, because guix handles cross-compilation for us.

The go-winres container

The only thing go-winres does is "a simple command line tool for embedding usual resources in Windows executables built with Go". Who cares about that? Bin it.

The base container

This begins from a standard Docker image for Go. If it's necessary, we can go read it, but hopefully all it does is install a Go environment. All docker does on top of that is configuring apt and Go telemetry (which is irrelevant to us) and install the file utility.

The build container

What remains is only about 70 lines of Dockerfile commands. Several of them are ARG lines, which I think we can ignore because I could not see anywhere they get set during a vanilla install. Presumably they exist so you can try Docker out in different environments.

As far as I can tell, the remaining structure is roughly:

WORKDIR /usr/src/moby
RUN apt-get install clang lld llcm icoutils
ARG TARGETPLATFORM
RUN xx-apt-get install gcc libc6-dev linux-libc-dev libnftables-dev
                       libseccomp-dev libsystemd-dev pkg-config
ENV PREFIX=/tmp
RUN inline-script-1
RUN --mount=type=bind,target=.,rw \
    --mount=type=cache ... \
    inline-script-2

Ok, so: install some dependencies, cross-compile-install some other dependencies, and run two scripts. I guess these are inputs vs native-inputs in guix terminology. inline-script-1 is incredibly simple,

RUN <<EOT
  set -ex
  # in bullseye arm64 target does not link with lld so configure it to use ld instead
  if [ "$(xx-info arch)" = "arm64" ]; then
    XX_CC_PREFER_LINKER=ld xx-clang --setup-target-triple
  fi
EOT

It seems to only contain cross-compilation information. I think we can ignore it completely.

The second snippet is more complicated. The type=cache mount is irrelevant, it is just there to make rerunning more performant (should happen in guix for free). The type=bind mount is significant, I think it mounts the source repo into the container. So at this point we have /usr/src/moby full of the source repo, with PREFIX=/tmp. I don't know if this env var matters to Go or not, but I assume it must.

Here are the contents of inline-script-2, where I have commented out anything I think we can ignore.

# set -e # causes early return on error, but we only care about successful compilation
  # I am only interested in a dynamic binary
# target=dynbinary
  # we do not need verify or exe support (see below)
# verify_flags=
# exe_suffix=
# if [ "$DOCKER_STATIC" = "1" ]; then
#     target=binary
#     verify_flags=--static
# fi
  # ignore windows
# if [ "$(xx-info os)" = "windows" ]; then
#     exe_suffix=.exe
# fi
# set -x # just changes the debug level of the shell
# xx-go --wrap # just enables go cross-compilation
  PKG_CONFIG=$(xx-go env PKG_CONFIG) ./hack/make.sh "$target"
  # xx-verify allows verifying that the cross-compile toolchain was
  # correctly configured and outputted binaries for the expected
  # target platform. xx-verify works by calling file utility and
  # comparing the expected output.
# xx-verify $verify_flags "/tmp/bundles/${target}-daemon/dockerd${exe_suffix}"
# if [ "$(xx-info os)" = "linux" ]; then
#     xx-verify $verify_flags "/tmp/bundles/${target}-daemon/docker-proxy"
# fi
  mkdir /build
  mv /tmp/bundles/${target}-daemon/* /build/

So, great, I think almost everything is irrelevant. We have reduced the 600-line Dockerfile to just one significant line of code:

PKG_CONFIG=$(xx-go env PKG_CONFIG) ./hack/make.sh "$target"

The script /hack/make.sh

There is a nice introductory comment.

This script builds various binary artifacts from a checkout of the docker source code.

Requirements:

1. The current directory should be a checkout of the docker source code (https://github.com/moby/moby). Whatever version is checked out will be built.
2. The VERSION file, at the root of the repository, should exist, and will be used as Docker binary version and package version.
3. The hash of the git commit will also be included in the Docker binary, with the suffix -unsupported if the repository isn't clean.
4. The script is intended to be run inside the docker container specified in the Dockerfile at the root of the source. In other words: DO NOT CALL THIS SCRIPT DIRECTLY.
5. The right way to call this script is to invoke "make" from your checkout of the Docker repository. the Makefile will do a "docker build -t docker ." and then "docker run hack/make.sh" in the resulting image.

Cool. Having read the Dockerfile, it seems "DO NOT RUN THIS" just means "make sure you have installed the build and runtime dependencies". The question now is of course "can we just run this in a container?".

EMACS CHANGES I WOULD LIKE - add a command to wrap highlighted text in a pollen command - hide attributes unless the cursor is inside the attribute list - prevent M-q from breaking lines inside an attribute list and do not count attributes in line length