# Publishing a New Container Image ## About the Container Image Firecracker uses a [Docker container](https://www.docker.com/) to standardize the build process. This also fixes the build tools and dependencies to specific versions. Every once in a while, something needs to be updated. To do this, a new container image needs to be built locally, then published to the [AWS ECR](https://aws.amazon.com/ecr/) registry. The Firecracker CI suite must also be updated to use the new image. ## Prerequisites 1. Access to the [`fcuvm` ECR repository](https://gallery.ecr.aws/firecracker/fcuvm). 1. The `docker` package installed locally. You should already have this if you've ever built Firecracker from source. 1. Access to both an `x86_64` and `aarch64` machines to build the container images. 1. Ensure `aws --version` is >=1.17.10. ## Steps ### **[optional]** Update `poetry.lock` This step is optional but recommended, to be on top of Python package changes. ```sh ./tools/devtool shell --privileged poetry update --lock --directory tools/devctr/ ``` This will change `poetry.lock`, which you can commit with your changes. ### `x86_64` 1. Login to the Docker organization in a shell. Make sure that your account has access to the repository: ```bash aws ecr-public get-login-password --region us-east-1 \ | docker login --username AWS --password-stdin public.ecr.aws ``` For non-TTY devices, although not recommended a less secure approach can be used: ```bash docker login --username AWS --password \ $(aws ecr-public get-login-password --region us-east-1) public.ecr.aws ``` 1. Navigate to the Firecracker directory. Verify that you have the latest container image locally. ```bash docker images REPOSITORY TAG IMAGE ID CREATED SIZE public.ecr.aws/firecracker/fcuvm v26 8d00deb17f7a 2 weeks ago 2.41GB ``` 1. Make your necessary changes, if any, to the [Dockerfile](https://docs.docker.com/engine/reference/builder/). There's one for all the architectures in the Firecracker source tree. 1. Commit the changes, if any. 1. Build a new container image with the updated Dockerfile. ```bash tools/devtool build_devctr ``` 1. Verify that the new image exists. ```bash docker images REPOSITORY TAG IMAGE ID CREATED SIZE public.ecr.aws/firecracker/fcuvm latest 1f9852368efb 2 weeks ago 2.36GB public.ecr.aws/firecracker/fcuvm v26 8d00deb17f7a 2 weeks ago 2.41GB ``` 1. Tag the new image with the next available version `X` and the architecture you're on. Note that this will not always be "current version in devtool + 1", as sometimes that version might already be used on feature branches. Always check the "Image Tags" on [the fcuvm repository](https://gallery.ecr.aws/firecracker/fcuvm) to make sure you do not accidentally overwrite an existing image. As a sanity check, run: ```bash docker pull public.ecr.aws/firecracker/fcuvm:vX ``` and verify that you get an error message along the lines of ``` Error response from daemon: manifest for public.ecr.aws/firecracker/fcuvm:vX not found: manifest unknown: Requested image not found ``` This means the version you've chosen does not exist yet, and you are good to go. ```bash docker tag 1f9852368efb public.ecr.aws/firecracker/fcuvm:v27_x86_64 docker images REPOSITORY TAG IMAGE ID CREATED public.ecr.aws/firecracker/fcuvm latest 1f9852368efb 1 week ago public.ecr.aws/firecracker/fcuvm v27_x86_64 1f9852368efb 1 week ago public.ecr.aws/firecracker/fcuvm v26 8d00deb17f7a 2 weeks ago ``` 1. Push the image. ```bash docker push public.ecr.aws/firecracker/fcuvm:v27_x86_64 ``` ### `aarch64` Login to the `aarch64` build machine. Steps 1-4 are identical across architectures, change `x86_64` to `aarch64`. Then continue with the above steps: 1. Build a new container image with the updated Dockerfile. ```bash tools/devtool build_devctr ``` 1. Verify that the new image exists. ```bash docker images REPOSITORY TAG IMAGE ID CREATED public.ecr.aws/firecracker/fcuvm latest 1f9852368efb 2 minutes ago public.ecr.aws/firecracker/fcuvm v26 8d00deb17f7a 2 weeks ago ``` 1. Tag the new image with the next available version `X` and the architecture you're on. Note that this will not always be "current version in devtool + 1", as sometimes that version might already be used on feature branches. Always check the "Image Tags" on [the fcuvm repository](https://gallery.ecr.aws/firecracker/fcuvm) to make sure you do not accidentally overwrite an existing image. As a sanity check, run: ```bash docker pull public.ecr.aws/firecracker/fcuvm:vX ``` and verify that you get an error message along the lines of ``` Error response from daemon: manifest for public.ecr.aws/firecracker/fcuvm:vX not found: manifest unknown: Requested image not found ``` This means the version you've chosen does not exist yet, and you are good to go. ```bash docker tag 1f9852368efb public.ecr.aws/firecracker/fcuvm:v27_aarch64 docker images REPOSITORY TAG IMAGE ID public.ecr.aws/firecracker/fcuvm latest 1f9852368efb public.ecr.aws/firecracker/fcuvm v27_aarch64 1f9852368efb public.ecr.aws/firecracker/fcuvm v26 8d00deb17f7a ``` 1. Push the image. ```bash docker push public.ecr.aws/firecracker/fcuvm:v27_aarch64 ``` 1. Create a manifest to point the latest container version to each specialized image, per architecture. ```bash docker manifest create public.ecr.aws/firecracker/fcuvm:v27 \ public.ecr.aws/firecracker/fcuvm:v27_x86_64 public.ecr.aws/firecracker/fcuvm:v27_aarch64 docker manifest push public.ecr.aws/firecracker/fcuvm:v27 ``` 1. Update the image tag in the [`devtool` script](https://github.com/firecracker-microvm/firecracker/blob/main/tools/devtool). Commit and push the change. ```bash PREV_TAG=v26 CURR_TAG=v27 sed -i "s%DEVCTR_IMAGE_TAG=\"$PREV_TAG\"%DEVCTR_IMAGE_TAG=\"$CURR_TAG\"%" tools/devtool ``` ## Troubleshooting Check out the [`rust-vmm-container` readme](https://github.com/rust-vmm/rust-vmm-container) for additional troubleshooting steps and guidelines. ### I can't push the manifest ```bash docker manifest is only supported when experimental cli features are enabled ``` See [this article](https://medium.com/@mauridb/docker-multi-architecture-images-365a44c26be6) for explanations and fix. ### How to test the image after pushing it to the Docker registry Either fetch and run it locally on another machine than the one you used to build it, or clean up any artifacts from the build machine and fetch. ```bash docker system prune -a docker images REPOSITORY TAG IMAGE ID CREATED SIZE tools/devtool shell [Firecracker devtool] About to pull docker image public.ecr.aws/firecracker/fcuvm:v15 [Firecracker devtool] Continue? ``` ### I don't have access to the AWS ECR registry ```bash docker push public.ecr.aws/firecracker/fcuvm:v27 The push refers to repository [public.ecr.aws/firecracker/fcuvm] e2b5ee0c4e6b: Preparing 0fbb5fd5f156: Preparing ... a1aa3da2a80a: Waiting denied: requested access to the resource is denied ``` Only a Firecracker maintainer can update the container image. If you are one, ask a member of the team to add you to the AWS ECR repository and retry. ### I pushed the wrong tag Tags can be deleted from the [AWS ECR interface](https://aws.amazon.com/ecr/). Also, pushing the same tag twice will overwrite the initial content. ### I did everything right and nothing works anymore If you see unrelated `Python` errors, it's likely because the dev container pulls `Python 3` at build time. `Python 3` means different minor versions on different platforms, and is not backwards compatible. So it's entirely possible that `docker build` has pulled in unwanted `Python` dependencies. To include **only your** changes, an alternative to the method described above is to make the changes *inside* the container, instead of in the `Dockerfile`. Let's say you want to update [`cargo-audit`](https://github.com/RustSec/cargo-audit) (random example). 1. Enter the container as `root`. ```bash tools/devtool shell -p ``` 1. Make the changes locally. Do not exit the container. ```bash cargo install cargo-audit --force ``` 1. Find your running container. ```bash docker ps CONTAINER ID IMAGE COMMAND CREATED e9f0487fdcb9 fcuvm:v14 "bash" 53 seconds ago ``` 1. Commit the modified container to a new image. Use the `container ID`. ```bash docker commit e9f0487fdcb9 fcuvm:v15_x86_64 ``` ```bash docker image ls REPOSITORY TAG IMAGE ID CREATED fcuvm v15_x86_64 514581e654a6 18 seconds ago fcuvm v14 c8581789ead3 2 months ago ``` 1. Repeat for `aarch64`. 1. Create and push the manifest.