So you have started to look at alternative architectures such as ARM, IBM Power or maybe you have a s390 mainframe and you want to leverage the power of containers, along with the management and orchestration of Kubernetes and OpenShift. You build your OpenShift cluster and deploy your first app and you are greeted with an error:
exec container process: Exec format error
What is this error, and why can’t I just run my app? The problem is, the container you are trying to run was built for a specific architecture (most likely x86_64), and can not be run on your shiny new ARM or IBM Power server. You may also run into this if you are building your containers on a modern Apple M1 (or newer) machine as this is going to build ARM64 based containers by default which will not run on your typical x86_64 server. So what is a multi-arch engineer to do? Well, in this case we need to start to build architecture specific container images. When we create and publish these images we have two options:
- Create a named image for each architecture. eg.
ghcr.io/xphyr/k8s_memuser_advanced:AMD64v1 - Create a manifest list which contains multiple architectures under one image name. eg.
ghcr.io/xphyr/k8s_memuser_advanced:v1
Support for multi-arch images has been around for a while, and their use and popularity is starting to grow as more enterprises look to optimize their datacenters for improved power usage, higher core counts and potential cost savings in many cloud providers like AWS, Google and Azure.
For the purposes of this discussion we will be using k8s_memuser_advanced a small tool I wrote awhile back to test load scaling on OpenShift. As of the writing of this blog post, it is published as a single architecture image (AMD64), and will work as a great test case for creating a multi-architecture image. In this article, we will focus on the use of podman for creating our container images. If you are looking to use the Moby/Docker tools, there are multiple articles that explain the use of BuildX which can also create multi-architecture images. See the references at the end of this post for some examples.
Manifest vs Manifest Lists
So what is a Manifest List? Its just what it sounds like, a LIST of all the manifests that are available for a given container image based on the architecture and OS type. We can see this by using podman manifest inspect command to see if a manifest is a “list” or just a regular manifest and if so, what architectures are available. We will start with the ghcr.io/fedora/fedora image to see what a multi-architecture manifest list looks like:
$ podman manifest inspect quay.io/fedora/fedora
{
"schemaVersion": 2,
"mediaType": "application/vnd.oci.image.index.v1+json",
"manifests": [
{
"mediaType": "application/vnd.oci.image.manifest.v1+json",
"size": 504,
"digest": "sha256:c056a81c332cefc846c689f149a1e51158abb64ede01bf0c83fb0f49a3a725fb",
"platform": {
"architecture": "arm64",
"os": "linux"
}
},
{
"mediaType": "application/vnd.oci.image.manifest.v1+json",
"size": 504,
"digest": "sha256:8dca26cef159c4a16f2ad35dd6d5696567957eae2270b1d102ed9136e40c9b64",
"platform": {
"architecture": "ppc64le",
"os": "linux"
}
},
{
"mediaType": "application/vnd.oci.image.manifest.v1+json",
"size": 504,
"digest": "sha256:684ade5948482b78f8f383ae06a61d53ccd23e77e077ba49fe77612f8f95fd30",
"platform": {
"architecture": "s390x",
"os": "linux"
}
},
{
"mediaType": "application/vnd.oci.image.manifest.v1+json",
"size": 504,
"digest": "sha256:af5dd88dbd971e6c19d83a6899b7eed3229003876051d79165fe2e4b503f4a39",
"platform": {
"architecture": "amd64",
"os": "linux"
}
}
]
}
Here we can see what a manifest list looks like. In this case the quay.io/fedora/fedora:latest image has support for four different architectures, arm64, ppc64le, s390x and amd64.
NOTE: “amd64” is the same as x64, x86-64, Intel 64, basically anything using the x86 64 bit instruction set.
OK, now lets take a look at the image that we want to run ghcr.io/xphyr/k8s_memuser_advanced:latest:
$ podman manifest inspect ghcr.io/xphyr/k8s_memuser_advanced:latest
WARN[0000] The manifest type application/vnd.docker.distribution.manifest.v2+json is not a manifest list but a single image.
{
"schemaVersion": 2,
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"manifests": null
}
As we can see from the output, our ghcr.io/xphyr/k8s_memuser_advanced:latest image is not a manifest list at all, and thus does not support multiple architectures. In order to find what platform the image does support we need to run podman inspect ghcr.io/xphyr/k8s_memuser_advanced:latest. Inspecting the output shows “Architecture”: “amd64”, and this is why we get the exec format error in our logs on our ARM64 OpenShift worker node.
$ podman image inspect --format "architecture: {{ .Architecture }}" ghcr.io/xphyr/k8s_memuser_advanced:latest
architecture: amd64
Let’s move onto the next step, and fix this issue so that our k8s_memuser_advanced tool can run on alternative architectures such as ARM.
Creating the manifest
We will start by creating our manifest list. This will be an empty list, that we will manually add architecture specific container images to at a later point. Instead of using the latest tag, we will use a new tag multi so we can go back and compare what we create here against the “latest” tag.
$ podman manifest create ghcr.io/xphyr/k8s_memuser_advanced:multi
a7a38454171eed886de226807c25d9cf09dd52cd903c73d21c0576e55bd07fee
We can check and see what our new manifest list looks like
$ podman manifest inspect ghcr.io/xphyr/k8s_memuser_advanced:multi
{
"schemaVersion": 2,
"mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
"manifests": []
}
Now that we have a better understanding of manifests vs manifest lists and how to query them, lets go ahead and build the containers for each of our target platforms next.
Manually Creating Platform Specific Images
We will use three different machines to build our images… an ARM Linux Workstation, an x86-64 Linux Workstation, and a Windows Workstation. The commands for building the images on both Linux workstations will be the same, and on our Windows workstation we will use docker to build a Windows native container image.
AMD64 Linux Workstation\
We will start with building our AMD64 container image. We will clone the code and then use the container build process to build our binary and then build our container image. There is nothing special that we need to do in building this container, however we will create a platform specific tag (eg. AMD64), that we will use later when adding it to our Manifest List.
$ git clone https://github.com/xphyr/k8s_memuser_advanced
$ cd k8s_memuser_advanced
$ podman build -t ghcr.io/xphyr/k8s_memuser_advanced:amd64 .
[1/2] STEP 1/4: FROM docker.io/golang:alpine AS builder
[1/2] STEP 2/4: WORKDIR /build/src
[1/2] STEP 3/4: ADD . /build/src/
[1/2] STEP 4/4: RUN CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo -ldflags '-extldflags "-static"' -o memuser .
[2/2] STEP 1/5: FROM alpine
[2/2] STEP 2/5: COPY --from=builder /build/src/memuser /app/
[2/2] STEP 3/5: WORKDIR /app
[2/2] STEP 4/5: EXPOSE 8080
[2/2] STEP 5/5: CMD ["/app/memuser", "-fast", "-maxmemory", "1000"]
[2/2] COMMIT ghcr.io/xphyr/k8s_memuser_advanced:amd64
Successfully tagged ghcr.io/xphyr/k8s_memuser_advanced:amd64
17d9bd9f88d0a0c4d5fc370ff6ef7f5d819ffeda27da2b751b823ed23582db9d
$ podman push ghcr.io/xphyr/k8s_memuser_advanced:amd64
Getting image source signatures
Copying blob b73cf5519a7f done |
Copying blob fd2758d7a50e done |
Copying config 17d9bd9f88 done |
Writing manifest to image destination
ARM Linux Workstation
Just like our AMD container image, we will clone the code and use podman to build our container using a tag of arm64.
$ git clone https://github.com/xphyr/k8s_memuser_advanced
$ cd k8s_memuser_advanced
$ podman build -t ghcr.io/xphyr/k8s_memuser_advanced:arm64 .
[1/2] STEP 1/4: FROM docker.io/golang:alpine AS builder
[1/2] STEP 2/4: WORKDIR /build/src
[1/2] STEP 3/4: ADD . /build/src/
[1/2] STEP 4/4: RUN CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo -ldflags '-extldflags "-static"' -o memuser .
[2/2] STEP 1/5: FROM alpine
Resolved "alpine" as an alias (/etc/containers/registries.conf.d/shortnames.conf)
Trying to pull docker.io/library/alpine:latest...
Getting image source signatures
Copying blob d69d4d41cfe2 skipped: already exists
Copying config 2abc5e8340 done
Writing manifest to image destination
Storing signatures
[2/2] STEP 2/5: COPY --from=builder /build/src/memuser /app/
[2/2] STEP 3/5: WORKDIR /app
[2/2] STEP 4/5: EXPOSE 8080
[2/2] STEP 5/5: CMD ["/app/memuser", "-fast", "-maxmemory", "1000"]
[2/2] COMMIT ghcr.io/xphyr/k8s_memuser_advanced:arm64
Successfully tagged ghcr.io/xphyr/k8s_memuser_advanced:arm64
1ab6bacb63433b8fbef0e25904e504d23a9ff4aba675cc2b5edd272841f982e7
$ podman push ghcr.io/xphyr/k8s_memuser_advanced:arm64
Getting image source signatures
Copying blob 1231a673589a done
Copying blob 0b0fd91b6c87 done
Copying config 1ab6bacb63 done
Writing manifest to image destination
Storing signatures
and now for Fun, lets create a Windows container as well!
AMD64 Windows Workstation
When working with Windows native containers, the tooling is similar and most commands are the same, however Podman is not supported for building Windows Native containers. We will instead be using the stevedore project to install the opensource version of tools to build our image. You can also check out the article I wrote on using stevedore for building Windows native containers here: Windows Containers on Windows 10 or 11, without Docker Desktop. Just like before we will pull our codebase and then use docker to build our container. We will use the tag amd64win this time to differentiate the container image from our amd64 Linux based container.
PS> git clone https://github.com/xphyr/k8s_memuser_advanced
PS> cd k8s_memuser_advanced
docker build . -t ghcr.io/xphyr/k8s_memuser_advanced:amd64win -f Dockerfile.Windows
Sending build context to Docker daemon 111.6kB
Step 1/9 : FROM golang:nanoserver as gobuild
Step 2/9 : COPY . /code
Step 3/9 : WORKDIR /code
Step 4/9 : RUN go build -o memuser.exe
Step 5/9 : FROM mcr.microsoft.com/windows/nanoserver:ltsc2022
ltsc2022: Pulling from windows/nanoserver
Step 6/9 : RUN mkdir c:\apps
Step 7/9 : COPY --from=gobuild /code/memuser.exe /apps/memuser.exe
Step 8/9 : EXPOSE 8080
Step 9/9 : CMD [ "c:/apps/memuser.exe" ]
Successfully built 105a5721fb6c
Successfully tagged ghcr.io/xphyr/k8s_memuser_advanced:amd64win
PS > docker push ghcr.io/xphyr/k8s_memuser_advanced:amd64win
The push refers to repository [ghcr.io/xphyr/k8s_memuser_advanced]
aa57595afdda: Pushed
37b652dbe546: Pushed
fd636c4aaab7: Pushed
67b23ce172d9: Pushed
44b913d145ad: Pushed
amd64win: digest: sha256:6037c5cf715d6b7dd120eb9f4eb120013788519b7e45561f4a3116f7e7b466c7 size: 1366
Let’s move onto the next step and make all of these images we built available under one manifest list.
Adding our images to the list
We now have three separate images that have been pushed to our image registry:
- ghcr.io/xphyr/k8s_memuser_advanced:arm64
- ghcr.io/xphyr/k8s_memuser_advanced:amd64
- ghcr.io/xphyr/k8s_memuser_advanced:amd64win
Now, we need to add them to the manifest list that we created earlier using the podman manifest add command making sure to specify the manifest list we want to add our images to and the location of the image to add:
$ podman manifest add ghcr.io/xphyr/k8s_memuser_advanced:multi docker://ghcr.io/xphyr/k8s_memuser_advanced:arm64
a7a38454171eed886de226807c25d9cf09dd52cd903c73d21c0576e55bd07fee
$ podman manifest add ghcr.io/xphyr/k8s_memuser_advanced:multi docker://ghcr.io/xphyr/k8s_memuser_advanced:amd64
a7a38454171eed886de226807c25d9cf09dd52cd903c73d21c0576e55bd07fee
$ podman manifest add ghcr.io/xphyr/k8s_memuser_advanced:multi docker://ghcr.io/xphyr/k8s_memuser_advanced:amd64win
a7a38454171eed886de226807c25d9cf09dd52cd903c73d21c0576e55bd07fee
Before we push the manifest list to GHCR.IO, lets take a look at it:
podman manifest inspect ghcr.io/xphyr/k8s_memuser_advanced:multi
{
"schemaVersion": 2,
"mediaType": "application/vnd.oci.image.index.v1+json",
"manifests": [
{
"mediaType": "application/vnd.oci.image.manifest.v1+json",
"size": 1003,
"digest": "sha256:fd9c56022e1678e5af09e6fdd97b62ecffd228d29c6b95c8cf6237c808fcfd60",
"platform": {
"architecture": "arm64",
"os": "linux"
}
},
{
"mediaType": "application/vnd.oci.image.manifest.v1+json",
"size": 1189,
"digest": "sha256:ce6dffdccddc0a6ecd4a2c104db727a29c6e58fe48ef43be6f25fe52f5562a8e",
"platform": {
"architecture": "amd64",
"os": "linux"
}
},
{
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"size": 1366,
"digest": "sha256:6037c5cf715d6b7dd120eb9f4eb120013788519b7e45561f4a3116f7e7b466c7",
"platform": {
"architecture": "amd64",
"os": "windows",
"os.version": "10.0.20348.3807"
}
}
]
}
We can see that we now have a manifest list that contains not just Multiple Architectures (AMD64 and ARM64), and also multiple OSes (both linux and windows).
The final step is to push the manifest list to the registry:
$ podman manifest push ghcr.io/xphyr/k8s_memuser_advanced:multi
Getting image list signatures
Copying 3 images generated from 3 images in list
Copying image sha256:fd9c56022e1678e5af09e6fdd97b62ecffd228d29c6b95c8cf6237c808fcfd60 (1/3)
Getting image source signatures
Copying blob b2d9e4e6b537 skipped: already exists
Copying blob 027e9609cfd0 skipped: already exists
Copying config 1ab6bacb63 done |
Writing manifest to image destination
Copying image sha256:ce6dffdccddc0a6ecd4a2c104db727a29c6e58fe48ef43be6f25fe52f5562a8e (2/3)
Getting image source signatures
Copying blob 0b7bc71b010c skipped: already exists
Copying blob c3426593330f skipped: already exists
Copying config 17d9bd9f88 done |
Writing manifest to image destination
Copying image sha256:6037c5cf715d6b7dd120eb9f4eb120013788519b7e45561f4a3116f7e7b466c7 (3/3)
Getting image source signatures
Copying blob 4b0c1eaac276 skipped: already exists
Copying blob 96acbf1c6d5b skipped: already exists
Copying blob 8aec05d8206a skipped: already exists
Copying blob 59c7384b0ab0 skipped: already exists
Copying blob c4ac71a068e9 skipped: already exists
Copying config 105a5721fb done |
Writing manifest to image destination
Writing manifest list to image destination
Storing list signatures
You will note that we are not pushing blobs, as those already exist, we are only validating the image definitions are available and then updating the manifest list ghcr.io/xphyr/k8s_memuser_advanced:multi.
Testing our Work
With our manifest list pushed to the registry, we can now go ahead and try running the container image, using the same container image name on each architecture (and OS).
Linux ARM
[markd@linuxarm]$ podman run --name k8s_mem_user -d ghcr.io/xphyr/k8s_memuser_advanced:multi
Trying to pull ghcr.io/xphyr/k8s_memuser_advanced:multi...
Getting image source signatures
Copying blob b2d9e4e6b537 skipped: already exists
Copying blob 027e9609cfd0 skipped: already exists
Copying config 1ab6bacb63 done
Writing manifest to image destination
Storing signatures
d5e48f7797cabb0d9a5ded1b2023140846eb937fa98c93a4e81d9af662dc4019
[markd@linuxarm]$ podman ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
d5e48f7797ca ghcr.io/xphyr/k8s_memuser_advanced:multi /app/memuser -fas... 3 seconds ago Up 4 seconds ago k8s_mem_user
[markd@linuxarm]$ podman logs k8s_mem_user
2025/06/12 21:31:56 Starting k8s_memuser_advanced revision: devel
2025/06/12 21:31:56 Listening on port: :8080
Linux AMD64
[markd@linuxamd64]$ podman run --name k8s_mem_user -d ghcr.io/xphyr/k8s_memuser_advanced:multi
Trying to pull ghcr.io/xphyr/k8s_memuser_advanced:multi...
Getting image source signatures
Copying blob c3426593330f skipped: already exists
Copying blob 0b7bc71b010c skipped: already exists
Copying config 17d9bd9f88 done |
Writing manifest to image destination
af481332fe815e9f29a8fdef37f40cc9c0e704981a553077b3736fe80011ccca
[markd@linuxamd64]$ podman ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
af481332fe81 ghcr.io/xphyr/k8s_memuser_advanced:multi /app/memuser -fas... 3 seconds ago Up 3 seconds 8080/tcp k8s_mem_user
[markd@linuxamd64]$ podman logs k8s_mem_user
2025/06/12 21:33:34 Starting k8s_memuser_advanced revision: devel
2025/06/12 21:33:34 Listening on port: :8080
Windows AMD64
PS > docker run --name k8s_mem_user -d ghcr.io/xphyr/k8s_memuser_advanced:multi
Unable to find image 'ghcr.io/xphyr/k8s_memuser_advanced:multi' locally
multi: Pulling from xphyr/k8s_memuser_advanced
Digest: sha256:002b0ac6a6e85e7f0acb6d524bbaf0d72cda62ed1fa90629aaee551d52dfb580
Status: Downloaded newer image for ghcr.io/xphyr/k8s_memuser_advanced:multi
1ebe6ebb643cdc867ad15cad86fa812f4c9fcdc5215b30a2f315d0df6da19e9c
PS > docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
1ebe6ebb643c ghcr.io/xphyr/k8s_memuser_advanced:multi "c:/apps/memuser.exe" 44 seconds ago Up 21 seconds 8080/tcp k8s_mem_user
PS > docker logs k8s_mem_user
2025/06/12 17:37:21 Starting k8s_memuser_advanced revision: devel
2025/06/12 17:37:21 Listening on port: :8080
BONUS ROUND: The Podman “Easy Button” for Multi-arch (but not Multi OS) images
Both podman and the docker have tools that can automate much of what we did above. While it can not handle the multi-OS portion where we created a Windows native Container, it can handle cross-compiling Linux container builds for multiple target architectures.
It should be noted that targeting multiple architectures requires that your Docker (Container) Image be built such that it can be cross-compiled for those architectures (e.g., by utilizing QEMU-based emulation, or through the Go tool chain cross compile capability.).
$ podman manifest create ghcr.io/xphyr/k8s_memuser_advanced:easybutton
$ podman build --platform linux/amd64,linux/arm64 --manifest ghcr.io/xphyr/k8s_memuser_advanced:easybutton .
[linux/amd64] [1/2] STEP 1/4: FROM docker.io/golang:alpine AS builder
[linux/amd64] [1/2] STEP 2/4: WORKDIR /build/src
[linux/amd64] [1/2] STEP 3/4: ADD . /build/src/
[linux/amd64] [1/2] STEP 4/4: RUN CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo -ldflags '-extldflags "-static"' -o memuser .
[linux/amd64] [2/2] STEP 1/5: FROM alpine
[linux/amd64] [2/2] STEP 2/5: COPY --from=builder /build/src/memuser /app/
[linux/amd64] [2/2] STEP 3/5: WORKDIR /app
[linux/amd64] [2/2] STEP 4/5: EXPOSE 8080
[linux/amd64] [2/2] STEP 5/5: CMD ["/app/memuser", "-fast", "-maxmemory", "1000"]
[linux/arm64] [1/2] STEP 1/4: FROM docker.io/golang:alpine AS builder
17d9bd9f88d0a0c4d5fc370ff6ef7f5d819ffeda27da2b751b823ed23582db9d
Trying to pull docker.io/library/golang:alpine...
Getting image source signatures
Writing manifest to image destination
[linux/arm64] [1/2] STEP 2/4: WORKDIR /build/src
[linux/arm64] [1/2] STEP 3/4: ADD . /build/src/
[linux/arm64] [1/2] STEP 4/4: RUN CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo -ldflags '-extldflags "-static"' -o memuser .
[linux/arm64] [2/2] STEP 1/5: FROM alpine
Resolved "alpine" as an alias (/etc/containers/registries.conf.d/000-shortnames.conf)
Trying to pull docker.io/library/alpine:latest...
Getting image source signatures
Copying blob d69d4d41cfe2 skipped: already exists
Copying config 2abc5e8340 done |
Writing manifest to image destination
[linux/arm64] [2/2] STEP 2/5: COPY --from=builder /build/src/memuser /app/
[linux/arm64] [2/2] STEP 3/5: WORKDIR /app
[linux/arm64] [2/2] STEP 4/5: EXPOSE 8080
[linux/arm64] [2/2] STEP 5/5: CMD ["/app/memuser", "-fast", "-maxmemory", "1000"]
[linux/arm64] [2/2] COMMIT
--> 4c324af06fdb
4c324af06fdbef53c45597b8e77da1bd9215d9bb722fa5c54fbc68260c123cf9
$ podman manifest push ghcr.io/xphyr/k8s_memuser_advanced:easybutton
We now have a manifest list that has been created with one command with two architectures availabled. We can confirm this by inspecting the manifest for this image:
$ podman manifest inspect ghcr.io/xphyr/k8s_memuser_advanced:easybutton
{
"schemaVersion": 2,
"mediaType": "application/vnd.oci.image.index.v1+json",
"manifests": [
{
"mediaType": "application/vnd.oci.image.manifest.v1+json",
"size": 1189,
"digest": "sha256:ce6dffdccddc0a6ecd4a2c104db727a29c6e58fe48ef43be6f25fe52f5562a8e",
"platform": {
"architecture": "amd64",
"os": "linux"
}
},
{
"mediaType": "application/vnd.oci.image.manifest.v1+json",
"size": 1188,
"digest": "sha256:ab4c403bff7542eb17ccf758b06fe3834decc432cbc4b5b9bb0b9b9da9efe03a",
"platform": {
"architecture": "arm64",
"os": "linux"
}
}
]
}
With our multi-image manifest list created, we can now go ahead and push this to the github Repo and know that for anyone who wants to run the k8s_memuser_advanced image it is available to them on x86_64 and ARM64 based platforms.
Multi-arch considerations
Should you decide to go down the route of building multi-architecture images be aware that you are opening your application up to a new type of bug, the “platform-specific” bug, which may not show up on one processor architecture but be prevalent on another. Catching these types of bugs will require that you have a well thought out application test plan and testing infrastructure in place.
Conclusion
Adding support for multiple architectures to your container images takes additional work, and should not be taken lightly. You will need to ensure that you can test your application on each architecture you support, or expect that you end users will be testing for you and might find bugs you are not aware of. However by supporting multiple architectures you open up your end user base to a wider audience and it may help with the long term adoption of your application.