A big step towards multi-platform Docker images

Photo by Todd Lappin from Flickr; CC BY-NC 2.0 license.

You might have missed some interesting image format-related advancements in the Docker registry and engine if you weren’t already paying attention to work that has been in-flight since mid-2015. Specifically, earlier this year a new image format specification was included in the Docker v2.3 registry release, and the initial code to handle this new format was made available in the Docker v1.10 engine.

This new “Version 2, schema 2” image manifest specification added a new object type, a “manifest list,” for storing what are effectively pointers to existing image manifests wrapped by a “platform” definition, allowing the specification of operating system, CPU architecture, as well as details like a CPU variant (a commonly needed distinction for embedded CPU families) and a features list. With this new manifest list support, a set of existing architecture-specific images in a Docker registry can be combined into a single reference, and most importantly for end users, referred to with a common name:tag nomenclature that users of Docker are familiar with. It is still the case today, as I had shared in a lightning talk at DockerCon EU 2015, that image creators and packagers on non-Linux OSs and/or non-64bit Intel CPUs have used various workarounds to segregate their own image references in Docker Hub (and private registries) given the inability to share names with their widely used 64-bit Linux counterparts. So, for example, docker pull busybox becomes docker pull ppc64le/busybox for a 64-bit POWER little-endian CPU-based system. Other architectures and operating systems have used this and other methods for creating names that users of these platforms must simply know to find and use their images appropriately.

What has been missing from these great enhancements to the registry and engine since their appearance in February 2016 is a tool to create these new manifest list objects in a supported v2.3 or above Docker registry. While the future landing spot for such a tool is still under discussion, today I’m introducing a standalone tool for creating and viewing  manifest list entries in supported registries. Meanwhile community and Docker distribution & engine team discussions are continuing to determine the officially supported tooling around manifest lists.

Check out the manifest tool repository on GitHub: https://github.com/estesp/manifest-tool

Using this manifest tool, you can provide a YAML formatted input file which defines the names and tags of the existing images which you would like to assemble into a manifest list. For each image reference combined into your manifest list, you provide the platform section definition, which at a minimum must define an OS and architecture pair to associate with that image.

An example of creating a debian:jessie image in my own estesp namespace on Docker Hub is shown below:

image: estesp/debian:jessie
 manifests:
   -
     image: debian:jessie
     platform:
       architecture: amd64
       os: linux
   -
     image: ppc64le/debian:jessie
     platform:
       architecture: ppc64le
       os: linux
   -
     image: s390x/debian:jessie
     platform:
       architecture: s390x
       os: linux

Manifest tool internals

Running the manifest tool to create a manifest list from YAML input is fairly straightforward:

$ manifest [--debug] pushml debian-jessie.yaml

The manifest tool operates by parsing the input YAML and querying the registry specified (or Docker Hub if no hostname is prefixed) to find the digest of each image reference as an input to the manifest list creation process. If these images do not exist or are not valid v1 or v2 schema manifests, the tool exits with an error. The platform section specified in YAML is verbatim copied into the manifest list object, after validation that the architecture and OS values are reasonable. The features and variant sections cannot be validated at this time given there is no well-defined or exhaustive list of what those can contain for various architectures. Discussions are underway on how to best handle those fields in the engine as well.

Once the manifest list object is assembled from the input, we have one more step that is potentially required before we ask the registry to host our manifest list. Remember that at this point our manifest list could reference blobs (layers) and manifests which may not exist in our target repository namespace. For example, if I’m assembling content from a repository aarch64/busybox into a final target name of estesp/busybox, then the blob (layer) references either need to be uploaded into my target repository (like a traditional docker push), or, thanks to another recent feature of the Docker registry, I can ask the registry for a “cross-repository” blob mount into my target repository given I know the source location of these blobs (hint: I found them when I looked up the source images found in the input YAML). After compiling this list, the manifest list creation code will ask for all blobs not pre-existing in the target namespace to be mounted, and then will push the manifests of each source image referenced into a dummy tag before finally pushing the manifest list itself to the registry. At this point, the registry server will validate it can reach all the referenced digest hashes from the manifest list object, and if so, will finally successfully create my manifest list in the target repository. Sounds complicated, but the good news is that if done correctly, I now have a single name:tag reference that can support commands like docker pull and docker run on multiple architectures and operating systems!

What’s Left to Do?

Full Engine Support

When the engine-side changes were added to handle the v2.2 image specification, encountering a manifest list during a pull operation generated this behavior: the list of platform-specific references from the manifest list are traversed, searching for an exact match on the engine’s OS and architecture. If a match is found, then that matching manifest reference is pulled locally under that manifest list’s name:tag, and the engine continues processing. At this time, the other fields of the platform object are ignored in the engine. For a complete manifest list implementation in the engine, we need to determine what and how the variant and features fields within the platform object will be assessed against the host platform of the engine. On Linux these will most naturally match to existing fields from /proc/cpuinfo, but more consideration is required to validate whether image runtime requirements may have broader dependencies than simple CPU specifics. In addition, for the Docker on Windows implementation, two additional fields have been added to the platform object to represent os.version and os.features for Windows specific distinctions necessary for their platform.

Hopefully community participants who care about their constituent OS and architecture platforms can provide requirements and input to this process as the engine is expanded to handle the full platform specification in a manifest list object.

Manifest Tool Function

In addition to adding full capabilities of the manifest list object format on the engine side, the manifest tool itself is, in some sense, merely a “proof of concept” tool that needs more work to be fully functional for all use cases. The most glaring area would be image signing and Docker Content Trust (DCT). The code which handles DCT is currently encapsulated within the push and pull implementation in the Docker engine itself, and if the manifest tool remains as a standalone tool, a refactoring to allow incorporation of DCT capabilities will be required. At this point in time there is no way to utilize DCT function with manifest list objects.

Docker Hub UI

Currently the Docker Hub UI does not provide visibility into manifest list-type name:tag entries in the registry. Of course, Docker Hub’s support for the v2.3 registry enhancements is quite recent: manifest list objects were only supported in the API endpoints in the past few weeks. We can expect that as complete and official implementations are realized for manifest lists in both the creation tooling and engine, Docker Hub will also provide more official visibility into manifest lists.

So, how can I tell if a registry entry is a manifest list?

I’m glad you asked! Thanks to the code ancestry of the manifest-tool project (it’s based on fellow Docker engine maintainer Antonio Murdaca’s original skopeo project, which started life as a registry inspection client) I have kept that core functionality in the manifest client, allowing rudimentary inspection of manifests. This inspect support handles both traditional schema v1 and v2 single manifest objects as well as the new manifest lists.

Below you can see example output from looking at a manifest list entry I pushed as estesp/busybox:

$ manifest inspect estesp/busybox
estesp/busybox is a manifest list containing the following 5 manifest references:
1    Mfst Type: application/vnd.docker.distribution.manifest.v1+json
1       Digest: sha256:7820f9a86d4ad15a2c4f0c0e5479298df2aa7c2f6871288e2ef8546f3e7b6783
1  Mfst Length: 2094
1     Platform:
1           -      OS: linux
1           -    Arch: ppc64le
1           - Variant:
1           - Feature:
1     # Layers: 2
         layer 1: digest = sha256:f7858729407d65aadd0dbcd321921368cff669db30f80878b44879867c061054
         layer 2: digest = sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4

2    Mfst Type: application/vnd.docker.distribution.manifest.v1+json
2       Digest: sha256:ae1b0e06e8ade3a11267564a26e750585ba2259c0ecab59ab165ad1af41d1bdd
2  Mfst Length: 1922
2     Platform:
2           -      OS: linux
2           -    Arch: amd64
2           - Variant:
2           - Feature: sse
2     # Layers: 2
         layer 1: digest = sha256:f810322bba2c5f0a6dd58ba31eba0543baabb4533e479ab2db376aaa8064be55
         layer 2: digest = sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4

3    Mfst Type: application/vnd.docker.distribution.manifest.v1+json
3       Digest: sha256:e4c0df75810b953d6717b8f8f28298d73870e8aa2a0d5e77b8391f16fdfbbbe2
3  Mfst Length: 2084
3     Platform:
3           -      OS: linux
3           -    Arch: s390x
3           - Variant:
3           - Feature:
3     # Layers: 2
         layer 1: digest = sha256:39b68f8511f9f44f10fb90a8a195dacb5f6ed26bd575163db3610ea91160abb6
         layer 2: digest = sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4

4    Mfst Type: application/vnd.docker.distribution.manifest.v1+json
4       Digest: sha256:07ebe243465ef4a667b78154ae6c3ea46fdb1582936aac3ac899ea311a701b40
4  Mfst Length: 2084
4     Platform:
4           -      OS: linux
4           -    Arch: arm
4           - Variant: armv7
4           - Feature:
4     # Layers: 2
         layer 1: digest = sha256:8fd1b2a17acf6fce6ffd3e72c17140c6dae9075f4d926809572c786ef11d821f
         layer 2: digest = sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4

5    Mfst Type: application/vnd.docker.distribution.manifest.v1+json
5       Digest: sha256:fb2fc0707b86dafa9959fe3d29e66af8787aee4d9a23581714be65db4265ad8a
5  Mfst Length: 2090
5     Platform:
5           -      OS: linux
5           -    Arch: arm64
5           - Variant: armv8
5           - Feature:
5     # Layers: 2
         layer 1: digest = sha256:770408447083eb668060e860a062a1e627019a3fd523b0bd345de21fdb2330eb
         layer 2: digest = sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4

The output format is somewhat rough at the moment, but gives the user a reasonable amount of information about the contents of a registry manifest, including a full level of detail on manifest list objects.

Thanks!

There’s a lot more to do as we continue marching towards full-featured multi-architecture support in the Docker engine and registry, but I think we’ve made significant progress in 2016 and I hope the manifest tool project is another stepping stone along that path. However, this is not only my work, and I want to thank those who have been involved all along the way:

• The Docker distribution team who have met weekly or bi-weekly for much of the past year, with “multi-arch registry support” sometimes being the only topic as we’ve hashed out many of the details together, and specifically created the v2.2 schema. There was a joke at one point that the meeting should be renamed “distribution team meets up with Phil”
• Good friend and fellow engine maintainer “runc0m” a.k.a Antonio Murdaca for creating his skopeo project which is now refactored and hosted under projectatomic/skopeo.  He did much of the hard work of factoring out the necessary registry interactions into a codebase that I could use as a starting point.
• Fellow IBMers Harshal Patil and Mathew Koshy who both performed a lot of the early leg-work, helped think through the schema definition with use cases, and created the initial prototype of the manifest tool.

We would love to get community feedback on where we are so far and if you have an interest in multi-architecture or OS support in Docker, now’s the time to get involved!

References:

• Docker Registry v2.3.0 release notes
• Docker engine 1.10 release (distribution section)
• Docker engine PR #18785: Manifest v2 support
• V1 and V2 schema format specifications
• manifest-tool GitHub repository