How Docker 19. 03 makes it simple to build images for most hardware platforms.
There comes a point in time in every hacker’s career when the need to compile a program for an alternate CPU architecture arises. Perhaps you want to compile a program for yourRaspberry Pi project, create a custom image for an embedded device, or support your software across multiple platforms . Perhaps you just want to learn how the process works, or are curious what the assembly code looks like for architectures other than the ubiquitious x 86 – 64 / AMD 64 on desktop PCs.
Either way, you were typically required to pack your bags and attempt the programmer’s equivalent of a spiritual pilgrimage; but instead of ascending to a forlorn mountaintop, you would begin a hellish descent, a journey that took you from the sunny plains of application development down into the dark caverns of the computing stack: the elusive world of low-level systems and embedded programming. Given the questionable prospects of this trek, most hackers who attempt the journey end up hittingCtrl Zand dashing back to the surface, gasping madly for air as they warn their colleagues of the terrors of cross-compilation, QEMU, and chroots.
OK, maybe I’m exaggerating a bit. But the truth is, it’s still not straightforward to build your programs for other CPU architectures. Thankfully, this story has significantly improved with the recent introduction of a new experimental plugin in Docker 19. 03 that makes multi-arch builds easier than ever.
In order to appreciate the significance of Docker’s new multi-arch build support, we first need to learn a bit about building programs for foreign architectures.
Background – Methods for compiling programs for foreign architectures
Note: If you’re familiar with this concept already or just want to get some damn images built already, feel free to skip this section.
Let’s take a quick survey of the methods that exist today for compiling programs that target foreign architectures.
Method # 1 – Build on the target hardware itself
If you have access to hardware for the target architecture, and the OS has support for all the build tooling you need, you can just compile the program directly on the hardware itself.
For our specific use-case of building multi-arch Docker images, you could, for instance, install the Docker runtime on a Raspberry Pi and build your app’s Dockerfile directly on the Pi just like you normally would on your developer machine. This is possible because Raspbian, the Raspberry Pi’s official OS, supports installing Docker natively.
But what if you don’t have convenient access to your target hardware? Is there any way we can somehow build programs for non-native architectures directly on our developer workstation?
And that brings us to …
Method # 2 – Emulate the target hardware
Do you remember the good ‘ol 16 – bit days with the Super Nintendo? I was just a toddler at the time, but when I grew a little older, I discovered the reverance with which gamers would reminisce of classic games like Super Mario World and Chrono Trigger. I never had the chance to own a SNES, but thanks to the emulators likeZSNES, I was able to travel back in time and experience the delight of playing those classic games – all from the comfort of my 32 – bit PC.
It turns out that we can use emulation to not only play video games, but to build non-native binaries as well. Instead of using ZSNES, we can use a much more powerful and flexible emulator:QEMU. QEMU is a free and open-source emulator that supports many common architectures including ARM, Power-PC, and RISC-V. By running in full system emulation mode, you can run a generic ARM virtual machine that can boot Linux, set up your development environment as usual, and compile your app from within the VM.
But if you think about it, full system emulation seems a bit wasteful. In this mode, QEMU will emulate an entire system, including hardware like timers, memory controllers, bus controllers like SPI and I2C, etc. – but the binaries we’re compiling most of the time don’t care at all about these hardware-specific features. Can we do better?
Method # 3 – Emulate just the target’s user-space via binfmt_misc
On Linux, QEMU has an alternative operating mode that can run Linux binaries compiled for non-native architectures via user-mode emulation. This mode skips the overhead of emulating the entire target system hardware in Option # 2. Rather, QEMU will register binary format handlers with the Linux kernel viabinfmt_miscand interpret the foreign binary transparently when it is run, converting system calls from the target to the host system as needed. The end result for a user is that it appears that she can run foreign binaries “natively”.
Using user-mode emulation and QEMU, we could install a foreign Linux distribution via lightweight virtualization (chrootor container) and build our binary as though we were building natively on our target.
We will soon see that this has become the method of choice for multi-arch Docker images .
Method # 4 – Use a cross-compiler
Finally, we have the standard method used in the embedded systems community: cross-compilation.
Across-compileris a compiler that is specifically built to run on a given host architecture, yet output binaries for a different target architecture. For example, you may have a C cross-compiler for a an amd 64 host that targets an embedded device (perhaps a smartphone or something) that is aarch 64 (64 – bit ARM). To give you a real-world example of this, consider the billions of Android devices around the world that have their software built precisely with this method.
Performance-wise, this approach is just as efficient as building on the target hardware itself (Method # 1) since it runs without emulation. But the complexity of cross-compilation varies depending on your language (it’s super easy with Go).
Confused yet? It gets more complicated with Docker images …
Keep in mind that all of these compilation hoops we had to jump through were just to generate a single program binary. In the modern age of containers, when we throw Docker images into the mix, we are not only talking about building single binaries, we are talking about building an entire foreign container image! It’s even more annoying than before!
If all of this sounds like a pain in the ass, don’t feel bad, because it kind of is a pain in the ass to build binaries for non-native platforms. Add the complexity of Docker on top of that and it seems like something best left to the experts.
But thanks to a new experimental extension in the latest Docker runtime, building multi-arch images is now easier than ever.
Building multi-arch Docker images
To build multi-arch Docker images the easy way, we can make use of arecently announcedDocker extension calledbuildx. buildx is the next-generation frontend for the standarddocker build ...command that we are all so familiar with for building Docker images. buildx extends the standard functionality ofdocker buildby leveraging the full functionality ofBuildKit, the new backend build system for Docker.
Let’s see how we can use buildx to cook up some multi-arch images in just a few minutes.
Step 1 – Enable buildx
To usebuildx, make sure your Docker runtime is at least version 19. 03. buildx actually comes bundled with Docker by default, but needs to be enabled by setting the environment variableDOCKER_CLI_EXPERIMENTAL. Let’s enable it for our current terminal session by running:
$ export DOCKER_CLI_EXPERIMENTAL=enabledVerify that you now have access to buildx by checking your version:
$ docker buildx version github.com/docker/buildx v0.3.1-tp-docker 6db 68 D (C) a aa7adcba8e5a 344795 A7Optional: Build from source
If you want the bleeding edge release or in case settingDOCKER_CLI_EXPERIMENTALis not working for you (I couldn’t get it to work on Arch Linux for example), you can always build from source:
$ export DOCKER_BUILDKIT=1 $ docker build --platform=local -o. git: //github.com/docker/buildx $ mkdir -p ~ / .docker / cli-plugins && mv buildx ~ / .docker / cli-plugins / docker-buildxStep 2 – Enable binfmt_misc to run non-native Docker images
If you’re using Docker Desktop (Mac and Windows), you can skip this step because binfmt_misc is set up by default.
If you’re on Linux, you need to set up binfmt_misc. This is pretty easy in most distributions, but is even easier now that you can just run a privileged Docker container to set it up for you:
$ docker run --rm --privileged docker / binfmt: 66 f (C) A 8316 f (FFD) D7C 21 c1f6f 28 dVerify that binfmt_misc is set up correctly by inspecting the QEMU handlers:
$ ls -al / proc / sys / fs / binfmt_misc / total 0 drwxr-xr-x 2 root root 0 Nov 12 09: 19. dr-xr-xr-x 1 root root 0 Nov 12 09: 16 .. -rw-r - r-- 1 root root 0 Nov 12 09: (qemu-aarch) -rw-r - r-- 1 root root 0 Nov 12 09: 25 qemu-arm -rw-r - r-- 1 root root 0 Nov 12 09: (qemu-ppc) le -rw-r - r-- 1 root root 0 Nov 12 09: (qemu-s) x --w ------- 1 root root 0 Nov 12 (***************************************************************************************************************: 19 register -rw-r - r-- 1 root root 0 Nov 12 09: 19 statusAnd verify that the handlers are enabled, for example:
$ cat / proc / sys / fs / binfmt_misc / qemu-aarch 64 enabled interpreter / usr / bin / qemu-aarch 64 flags: OCF offset 0 magic 7f (C) B7 mask ffffffffffffff 00 fffffffffffffffffeffffStep 3 – Switch from the default Docker builder to a multi-arch builder
By default, Docker will use the old builder instance without multi-arch support.
To create a new builder with multi-arch support, run:
$ docker buildx create --use - name mybuilderVerify that our new builder is selected:
$ docker buildx ls NAME / NODE DRIVER / ENDPOINT STATUS PLATFORMS mybuilder * docker-container mybuilder0 unix: ///var/run/docker.sock inactive default docker default default running linux / amd 64, linux / arm 64, Linux / PPC 64 le, linux / s 390 x, linux / 386, linux / arm / v7, linux / arm / v6That’s it – now Docker will use our new builder that’s capable of building for multiple platforms.
Step 4 – Build a multi-arch image
OK, now we can finally build a multi-arch image! To do that, we’ll first need an example app.
Let’s create a simple Go program that echoes back the host’s runtime architecture:
$ cat hello.go package main import ( "fmt" "runtime" ) func main () { fmt.Printf ("Hello,% s! n", runtime.GOARCH) }And let’s create a Dockerfile to containerize this app:
$ cat Dockerfile FROM golang: alpine AS builder RUN mkdir / app ADD. / app / WORKDIR / app RUN go build -o hello. FROM alpine RUN mkdir / app WORKDIR / app COPY --from=builder / app / hello. CMD ["./hello"]This is a multi-stage Dockerfile that builds our app with the Go compiler and creates a minimal Alpine Linux image with the resulting binary.
Now let’s build a multi-arch image with buildx that supports arm, arm 64, and amd 64, and push it to Docker Hub all in one go:
$ docker buildx build -t mirailabs / hello -arch --platform=linux / arm, linux / arm 64, Linux / AMD 64. --pushYup, that’s it. We now have a multi-arch Docker image for arm, arm 64, and amd 64available on Docker Hub! When you rundocker pull mirailabs / hello-arch, Docker will take care of fetching the matching image for your host architecture.
How does this buildx magic work, you ask? Well, behind the scenes, buildx builds three Docker images (one for each of arm, arm 64, and amd 64) using QEMU and binfmt_misc as needed. When it’s done building, it will create a Dockermanifest listwhich contains pointers to the three images. In other words, a “multi-arch image” is really just a manifest list with links to images built per architecture.
Step 5 – Test the multi-arch image
Let’s quickly test our multi-arch image and make sure everything is working as expected. Since we have already set up binfmt_misc, we can actually run any of our images on our development machine, regardless of architecture.
First, we list the digests for each of our images:
$ docker buildx imagetools inspect mirailabs / hello- arch Name: docker.io/mirailabs/hello-arch:latest MediaType: application / vnd.docker.distribution.manifest.list.v2 json Digest: sha 256: BBB 246 e 520 A (e) B0C6D (b) ************************************************************************ (Eece) a 8407 eede (cff) ************************************************************************************************* (c) edce 96 Manifests: Name: docker.io/mirailabs/hello-arch:[email protected]: 5FB 57946152 D (e) C (AA) ******************************************************************** (fe) CD 5742 DC 13 a3fabc1a 890 ADFC 2683 df MediaType: application / vnd.docker.distribution.manifest.v2 json Platform: linux / arm / v7 Name: docker.io/mirailabs/hello-arch:[email protected]: CC6E 91101828 fa4e 464 f7eddec3fa7cdc 73089560 cfcfe4af 16 CCC 61743 AC 02 b MediaType: application / vnd.docker.distribution.manifest.v2 json Platform: linux / arm 64 Name: docker.io/mirailabs/hello-arch:[email protected]: cd0b 32276 cdd5af 510 fb1df5c (f) *************************************************************************** (e) ************************************************************************************* (fe) afe3cec5ff7da3f 80 f 27985 d MediaType: application / vnd.docker.distribution.manifest.v2 json Platform: linux / amd 64With these digests handy, we can run each image and observe the output:
$ docker run --rm docker.io / mirailabs / hello-arch: latest @ sha 256: 5fb (D) e (c) **************************************************************** (AA) FE (CD) ******************************************************************* (DC) a3fabc1a 890 ADFC 2683 df Hello, arm! $ docker run --rm docker.io/mirailabs/hello-arch:[email protected]: CC6E 91101828 fa4e 464 f7eddec3fa7cdc 73089560 cfcfe4af (CCC) AC 02 b Hello, arm 64! $ docker run --rm docker.io/mirailabs/hello-arch:[email protected]: cd0b 32276 cdd5af (fb1df5c) f (e) fe (afe3cec5ff7da3f) ******************************************************************************************* f) d Hello, amd 64!That was pretty easy, wasn’t it?
Conclusion
To recap, in this post we learnt about the challenges of supporting software on multiple CPU architectures, and how buildx, an experimental extension to Docker’s build engine, can solve some of these challenges for us. Using buildx, we were able to quickly build a multi-arch Docker image for arm, arm 64, and amd 64 without a single change to our Dockerfile, and push it up to Docker Hub, from where any Docker-supported platform could transparently pull down the correct image for its architecture.
In the future, it is likely that buildx capabilities will become part of the standarddocker buildcommand and we will end up taking these features for granted. The tales of descending into the depths of the com puting stack to cross-compile programs will soon be nothing more than ghost stories from a more primitive age.
Go forth and multi-arch without fear
GIPHY App Key not set. Please check settings