Adventures with bootc: Upgrading to Fedora 42

Twice a year, I eagerly await the new Fedora release and typically move to it on my systems during the beta phase. I was particularly excited about trying this with F42 because my setup *should* let me change the tag on my image to from :41 to :42 and then all of my “child images” should get automatically rebuilt, and then all upgraded. I’ve been a user of various rpm-ostree distros for many years now. I typically tell people that once you go through a major upgrade, that’s it – you’ll never go back. As you might imagine this post probably wouldn’t exist if everything was smooth sailing. Don’t get me wrong everything worked out fine, but I thought it might be helpful to others if I documented a few things about my experience.

3rd Party Repos

The biggest blocker I faced was the lag with the NVIDIA repos: https://developer.download.nvidia.com/compute/cuda/repos/ I wanted to use both the graphics and cuda drivers from this repo as that’s where they’re commonly pulled on the RHEL side of the house. I haven’t been watching them long enough to know how long the historic lag time is, but I can see some packages dated about ~30 days after F41 was released. That’s actually not bad, but for my main systems I really don’t want to be beholden to a 3rd party to update my OS. This reason is why the rpmfusion repos are so valuable. I’m glad I documented the NVIDIA repos in my last blog, as that should be valuable to RHEL users.

The path forward for my fedora systems is to put cuda in containers – which is nice because it makes the base OS **huge**, and get graphics drivers from rpmfusion. This should make future upgrades that much smoother.

Kmods & base images

My next challenge was around the base image content vs packages in the repo. Ultimately, base image builds are done frequently and are automated with CI. I don’t expect this to be a common issue for others, but here’s what I hit. When F42 was announced as GA, we had a gap in shipping a new base image and my build was picking up multiple kernels when installing kernel-devel and failing to build the kmods due to my script not being able to handle multiple kernels. This isn’t something you’ll hit if you try it today, but it was a minor hiccup.

The akmod command fails with F42 hitting a permissions issue in /var/tmp. I didn’t have time to look into why this happens, and fortunately, it was super easy to just work around w/ chmod. Also, the rpmfusion drivers use akmods, I so I commented out the dkms section. Here is my updated script for building kmods with the container:

#!/bin/bash
set -euox pipefail

kver=$(cd /usr/lib/modules && echo *)

cat >/tmp/fake-uname <<EOF
#!/usr/bin/env bash

if [ "\$1" == "-r" ] ; then
  echo ${kver}
  exit 0
fi

exec /usr/bin/uname \$@
EOF
install -Dm0755 /tmp/fake-uname /tmp/bin/uname

#workaround for akmod permission issue
chmod 777 /var/tmp

#PATH=/tmp/bin:$PATH dkms autoinstall -k ${kver}
PATH=/tmp/bin:$PATH akmods --force --kernels ${kver} 

chmod 755 /var/tmp

Containerfiles

My standard operating environment, or “SOE”, base just adds a few global things I want for all my systems, and also streamlines some of the layer sharing and build process. This one *was* super simple to update and had the intended result for my home server & work laptop layered images. I really only had to tweak w/ desktop setup.

FROM quay.io/fedora/fedora-bootc:42

COPY etc etc
COPY usr usr

RUN dnf install -y btop cockpit cockpit-podman cockpit-storaged cockpit-ws cockpit-machines cockpit-selinux bwm-ng firewalld git htop lm_sensors nss-mdns pcp pcp-selinux sysstat tree tuned wget vim-enhanced && dnf clean all

RUN systemctl enable fstrim.timer podman-auto-update.timer cockpit.socket

#let's set the timezone
RUN ln -s /usr/share/zoneinfo/America/Chicago /etc/localtime && \
	rm -rf /var/*

#added linting to catch basic issues
RUN bootc container lint

Here’s my main daily driver Containerfile:

FROM [my_registry]/fedora-soe-bootc:42

#copy configs
COPY etc etc
RUN mkdir -p /var/roothome /data

#install packages, enable services, grab fonts
RUN dnf -y install https://mirrors.rpmfusion.org/free/fedora/rpmfusion-free-release-$(rpm -E %fedora).noarch.rpm https://mirrors.rpmfusion.org/nonfree/fedora/rpmfusion-nonfree-release-$(rpm -E %fedora).noarch.rpm && dnf install -y rpmfusion-free-release-tainted && \
dnf install -y gcc-c++ nvidia-container-toolkit && \
dnf install -y akmod-nvidia xorg-x11-drv-nvidia && \
dnf group install -y kde-desktop virtualization && \
dnf install -y android-tools bcache-tools cups cups-browsed firefox fuse-exfat gamemode gdb guvcview gvfs input-leap kamera k3b kernel-headers libguestfs libvirt libvirt-daemon openrgb-udev-rules powertop qemu-kvm samba steam-devices subscription-manager thermald virt-install virt-manager vulkan-tools v4l2loopback v4l-utils xdpyinfo && \
dnf clean all && \
dnf group install -y multimedia –setopt=”install_weak_deps=False” –exclude=PackageKit-gstreamer-plugin && \
dnf install -y ffmpeg libva-nvidia-driver nvidia-vaapi-driver libva-utils vdpauinfo && \
dnf swap -y mesa-va-drivers mesa-va-drivers-freeworld && \
dnf swap -y mesa-vdpau-drivers mesa-vdpau-drivers-freeworld && \
dnf remove -y –no-autoremove plasma-discover-offline-updates plasma-discover-packagekit tracker tracker-miners && \
dnf clean all && \
systemctl enable lm_sensors sysstat tuned libvirtd.socket && \
systemctl set-default graphical.target && \

DOWNLOAD_URL=$(curl https://api.github.com/repos/githubnext/monaspace/releases/latest | jq -r ‘.assets[] | select(.name| test(“.*.zip$”)).browser_download_url’) && \
curl -Lo /tmp/monaspace-font.zip “$DOWNLOAD_URL” && \
unzip -qo /tmp/monaspace-font.zip -d /tmp/monaspace-font && \
mkdir -p /usr/share/fonts/monaspace && \
mv /tmp/monaspace-font/monaspace-v*/fonts/variable/* /usr/share/fonts/monaspace/ && \
rm -rf /tmp/monaspace-font* && \
fc-cache -f /usr/share/fonts/monaspace && \
curl –output-dir /tmp -LO https://github.com/ryanoasis/nerd-fonts/releases/download/v3.2.1/FiraCode.zip && \
mkdir -p /usr/share/fonts/fira-nf && \
unzip /tmp/FiraCode.zip -d /usr/share/fonts/fira-nf && \
fc-cache -f /usr/share/fonts/fira-nf && \
fc-cache -f /usr/share/fonts/ubuntu && \
fc-cache -f /usr/share/fonts/inter

#Configure Default SDDM background
COPY usr usr

#Build kmods
COPY –chmod=755 kmod.sh /tmp
RUN /tmp/kmod.sh

#workaround for selinux policy, clean /var
RUN
rm -rf /var/*

#added linting to catch basic issues
RUN bootc container lint

Anyway, it was ultimately a very smooth experience. I did have to iterate a few times on my main containerfile, but 1) it didn’t take very long 2) nothing interrupted my system from working 3) I very much appreciated I could rollback, but it wasn’t necessary this go around. A huge thanks to everyone who works on this wonderful distro and bootc technology. I love it.