Welcome to LWN.net

The following subscription-only content has been made available to you by an LWN subscriber. Thousands of subscribers depend on LWN for the best news from the Linux and free software communities. If you enjoy this article, please consider subscribing to LWN. Thank you for visiting LWN.net!

The bootc project allows users to create a bootable Linux system image using the container tooling that many developers are already familiar with. It is an evolution of OSTree (now called libostree), which is used to create Fedora Silverblue and other image-based distributions. While creating custom images is still a job for experts, the container technology simplifies delivering heavily customized images to non-technical users.

Image-based desktop operating systems, such as Fedora Silverblue or the Universal Blue project’s Bluefin and Bazzite, have been around for a while now. LWN covered Bluefin in 2023. I started installing Fedora Silverblue on my relatives’ computers after a package-based update on my niece’s laptop went awry. It shut down unexpectedly during a DNF transaction, which caused a corruption in a systemd binary, rendering the system dead. The image-based systems (also called “atomic” systems) effectively prevent this kind of problem by managing updates as a single transaction that can be rolled back if needed.

Fedora’s image-based operating systems use OSTree and rpm-ostree to create images. However, there has been interest in using container tooling to create bootable images for Fedora and CentOS instead. LWN covered the bootc project in June 2024, and it has since become a CNCF Sandbox project.

When I finally decided to switch to atomic desktops on my computers, I stumbled upon Valentin Rothberg’s presentation (YouTube video) on bootable containers and decided that this is the technology I want to use.

Benefits of bootable containers

A bootc image is basically an Open Container Initative (OCI) image that also includes a kernel and other bits needed to boot a system. The running system has /usr mounted as read-only; updated images are downloaded in the background, but updates are performed atomically when the system reboots—the system boots into the new image, and the old image is preserved for rollbacks. Internally, bootc still depends on libostree, the heart of Fedora’s atomic and immutable operating systems. It is built using steps defined in a Containerfile (or Dockerfile), just like a regular Linux container. Unlike a normal container, it can be easily deployed as a virtual machine or installed on bare metal.

Since bootable containers are built as regular containers, the final images can be pushed to a container registry; from there, the image can be deployed to all bootc-enabled systems by running “bootc upgrade” on the host that is being updated (this is usually done automatically by a systemd timer). This brings several key benefits.

Since bootc uses existing container tooling, users familiar with creating container images using Docker or Podman will be able to start creating custom operating system images without learning new tools or workflows. Users are also fully in control over the image contents; therefore, they can essentially create their own heavily customized Linux distribution by installing custom packages, enabling kernel modules, providing kernel arguments, and configuring other parts of the system.

Using containers also allows local image testing before deployment and the ability to integrate it inside continuous-integration/continuous-deployment (CI/CD) pipelines. After the container builds, it can be started and tested as a regular container, which shares the kernel with the host system. This should be sufficient for testing most of the time, but it does not make use of the kernel (or its configuration) in the container image. It is also possible to easily start the image as a virtual machine with podman-bootc or its successor, the bootc virtualization kit (bcvk), to test the bootc container’s kernel as well. When all tests pass, it can be pushed to a container registry that it can be deployed from.

Preparing an image with a desktop environment

Even though bootable containers are built more or less the same way as any other containers, they require a special base image that is prepared for this particular use case containing a kernel, bootloader, and OSTree. There are bootc base images available for AlmaLinux, CentOS Stream, Fedora, and Red Hat Enterprise Linux at the moment. The Bluefin distribution, which is based on Fedora, uses bootc images as well.

There are also unofficial base images for several other distributions like Debian, Gentoo, Ubuntu, and others (this bootc ticket contains some tips and discussion on creating the base images). The official images usually do not come with a graphical environment, so they require a bit of customization to make them ready for desktop use cases such as installing desktop environments and favorite applications. Rothberg has a Containerfile for Fedora Workstation that can be used as an example for creating one’s own.

There is one catch that needs to be remembered when building the bootc container image: the build process is the same as for other containers. The kernel that is running on the host where “podman build” is run is not necessarily the kernel that will be installed in the bootc image. This poses difficulties when installing additional kernel modules (or other code that depends on the running kernel).

Each kernel module is built against the kernel that is supposed to load the module. Normally, this is the kernel that is currently booted, however when building a container image, we need to build the module against the kernel that is installed inside the image which is often different. Today kernel modules are often packaged by distributions in dynamic formats like akmods (for Fedora-derived distributions) or DKMS (other distributions). The akmods tooling automatically rebuilds the installed kernel modules against the currently running kernel (they essentially look at “uname -a”) and simplifies packaging and kernel updates.

This requires some additional work when installing kernel modules—akmods must be built manually with akmodsbuild, and other modules may need to be compiled directly from source. While this adds more complexity to the container definition file, it can be beneficial once the base bootc image switches to a new kernel that is not yet supported by one of the modules. A great example is the OpenZFS filesystem module that occasionally fails to build on newer kernels. No new images will be created until the modules build correctly, which means users aren’t served broken images that would cause them problems.

Deploying bootc images

When the bootable container image is available, it needs to be deployed to bare metal or a virtual machine. There are several ways to do that. One way is the “bootc install” command that can be run from the container image to install the image into a block device or an existing filesystem. The best way to install for the first time using “bootc install” is to boot into a live system, such as a Fedora Workstation live image.

A higher-level tool, “bootc-image-builder”, can output multiple image types, including an Anaconda unattended-installation ISO, and virtual-machine formats such as vmdk or qcow2, that can be deployed on other computers. However, for systems that support Anaconda and kickstart installations, I found the easiest way to deploy the image for the first time is kickstart’s ostreecontainer command. This can be used to prepare an installation flash disk including the kickstart file included to deploy the image on the target computers.

Once the image is installed, we have a bootc-enabled system that is managed and updated by the bootc tool.

Adopting bootc

Adopting bootc systems may require some getting used to, unless one has previous experience with atomic systems, as there are several differences, especially with regards to filesystem permissions. On the other hand, for those already used to atomic systems, it will feel like home.

The system is booted with a read-only /usr (which means it is not possible to install new packages at will). The expectation is that all packages are already installed within the bootc image. If additional packages are needed, they should be installed as a container, as a Flatpak application. Another alternative is using Toolbx or Distrobox, which are tools that let users run applications from containers that are fully integrated with the rest of the user’s system.

While /usr is read-only and most of the applications should be preinstalled, installed as Flatpaks, or inside a Toolbx/Distrobox container, it is possible to create a transient writable file system (OverlayFS) over the read-only /usr by calling “bootc usr-overlay”. Once the overlay is created, it is possible to install additional packages or make other changes to /usr; however, all changes are discarded after reboot. To make the change permanent, it is necessary to update the Containerfile to include the new package, rebuild the image, run “bootc upgrade” to deploy it, and then reboot the updated host to apply the new image. The overlay can be used to test the package functionality before updating the Containerfile and rebuilding the image.

If, by any chance, the image gets through pre-deployment testing and introduces a breaking change to the environment, it is always possible to roll back to the previous version. If the terminal can be accessed, it is possible to call “bootc rollback –apply” or if the system fails to boot, it is possible to choose the older version from the GRUB boot menu.

Keeping /etc in sync

One of the challenges for standard Linux systems is that configuration in /etc is modified over time, and tracking those changes is hard. This is especially true if users tend to work on multiple computers or laptops. Bootc, by default, sets up a persistent /etc, which means that every configuration change is kept after reboot, and there is a /etc merge process on each image upgrade; new configuration files are added, files that were not modified locally are updated, and files that were modified are kept. Note that bootc does not attempt to merge changes between a new configuration file supplied by an update and an existing configuration file.

However, it is possible to set up a transient /etc that will keep /etc writable so it is possible to experiment with configuration changes, but all changes are discarded after reboot as /etc contents from the image are reloaded. This makes it possible to write the desired configuration into the container image and deploy it on the computers through the image updates and thus keep all deployments completely in sync.

The command “ostree admin config-diff” can be handy when relying on the transient /etc feature, as it shows the difference between the current contents of /etc and the image version. It will quickly identify files that have been added, modified, or deleted. This makes it simple to experiment with configuration changes and then commit these changes to the container image without losing track of them.

Encrypting the image

While the bootable container images can be saved on the computer’s local storage, it is expected that they will be published to a public or private container registry; this will make it easier to share and to deliver updates for multiple machines. When doing so, it is desirable to encrypt the image’s contents to avoid leaking secrets to any third-party provider. The bootc project does not currently integrate encryption into its tooling, unfortunately. The project does have an open ticket to track this feature.

However, container images can be encrypted with OCIcrypt, which is integrated into Podman. It is possible to use Podman to pull the images into a local store and then point bootc to fetch updates from it. This solution keeps the encrypted image inside a remote container registry but stores it unencrypted in a local storage, from which it is read by bootc.

$ sudo podman pull --decryption-key private.pem quay.io/path/to/image:encrypted
$ sudo bootc switch --transport containers-storage quay.io/path/to/image:encrypted

Conclusion

Modifications to the OSTree/rpm-ostree-based systems, such as Fedora Silverblue, are possible: but using bootc makes it much more straightforward. However, building an image requires technical expertise and is not as simple as it might be. With the popularity of containers, though, it is quite likely that many users are already familiar with the technology. Once created, a bootc system can be used safely by non-technical users, and should make it less likely that an update will cause problems.

What I like about bootable containers most is the ease with which I manage all of the fifteen computers for my family. I no longer have to be on-site to fix things or include additional software; I do not even require a direct network connection to the machine. I can easily get access to an identical environment when needed. Furthermore, I have full control over the contents and base configuration of those systems, while my family can still install user-facing applications without worrying about doing any damage to the underlying operating system.

Index entries for this article
GuestArticles	Březina, Pavel

---