Rebuilding Container Images with systemd timers & podman

In general, it’s considered a best practice when running containers to ensure that the images are being rebuilt on a regular basis to pickup security/bug fixes. In a real production environment, it’s common to use something like jenkins, github actions, or some type of automation or CI/CD workflow to keep the images fresh. ….but here at my house, I only have a single server that runs containers and my use case doesn’t really warrant a more serious CI/CD setup. This blog will show you how to setup a simple “perpetual motion” machine to automatically rebuild container images and then auto-update them. It’s also pretty easy to setup and works great too!

First, let’s understand that podman can auto update containers based on a registry tag or using the local image store. Most of the time, I rely on new versions landing in the registry, but since this is a single server where we’re rebuilding, I think it makes more since to use the local storage option. To use auto-updates we need to label the container with --label io.containers.autoupdate=local If you want to read more on this subject, I recommend this blog post. Once this is in place, enable the auto-updates timer and adjust the timer unit to your desired time to run. systemctl enable --now podman-auto-update.timer.

I prefer using Quadlets to define my applications. Notice the line below invokes AutoUpdate=local

cat /etc/containers/systemd/caddy.container

[Unit]
Description=Caddy Quadlet

[Container]
Image=localhost/caddy:latest
ContainerName=caddy
AutoUpdate=local
EnvironmentFile=/etc/containers.environment
Volume=caddy-data.volume:/data
Volume=caddy-config.volume:/config
Volume=/etc/Caddyfile:/etc/caddy/Caddyfile:Z
PublishPort=80:80
PublishPort=443:443

[Service]
Restart=always
TimeoutStartSec=900

[Install]
WantedBy=multi-user.target default.target

The caddy.container file above will ensure that this container is always running, and the podman-auto-update.timer is set to check for nightly updates. Honestly that’s pretty amazing on it’s own – especially if you remember the early days of Linux containers. From here all we need to do is create unit files to rebuild the image and automatically run it.

This unit will basically just run podman build and create our image. A nice option in podman is the –pull which just like docker will grab the latest image from the registry. I chose to run a prune command to prevent the system storage from filling up w/ too many builds. Some people may not want the prune line.

# /etc/systemd/system/caddy-build.service
[Unit]
Description=Rebuild the caddy container image

[Service]
Type=oneshot
ExecStart=/bin/sh -c '/usr/bin/podman build --pull -f /etc/caddy/caddy.cf -t caddy:latest && /usr/bin/podman image prune  -f'

From here, we just need to create a timer unit w/ the same name as the unit above and pick a time for it to run. I chose to run this weekly as it’s not super urgent for me to rebuild this frequently. For my situation monthly would be fine and I may tweak this over time as I’m still relatively new to using caddy.

# /etc/systemd/system/caddy-build.timer
[Unit]
Description=Rebuild caddy once a week

[Timer]
OnCalendar=weekly
AccuracySec=1h
Persistent=true
RandomizedDelaySec=100min

[Install]
WantedBy=timers.target

Simply enable the above timer with systemctl enable --now caddy-build.timer and your perpetual motion machine will be set in motion! I have no doubts that using a more modern setup like github actions or gitlab runner would enable something smarter. I do plan to move to this model at some point, but for now this seems to be working well enough and it was painless to setup.

UPDATE! I also added a simple healthcheck to this setup to ensure the update is functioning. There may be a better way to do this, but I just added a handler that responds OK in my Caddyfile:

        @healthcheck host healthcheck.mydomain.com
        handle @healthcheck {
                respond "OK" 200
        }

Then a simple script to verify Caddy is responding and COPY this inside the containerfile when it’s built:

cat healthcheck.sh 
#!/bin/sh

CHECK=$(curl -s https://healthcheck.mydomain.com)

if [ "$CHECK" = OK ];
then
    exit 0
fi
exit 1

Then finally add these lines to the quadlet file above:

HealthCmd=/healthcheck.sh
HealthInterval=2m
HealthOnFailure=kill
HealthRetries=3

i wouldn’t say this gives us perfect resiliency, but it does add a good layer. This is a good blog post if you want more details. One thing I don’t have working yet is auto-rollback and I believe I need to switch to a registry for this to work.

At first, I tried googling for a similar setup and couldn’t find anything. I’m sharing this in case it’s useful for others, so please copy/paste and tweak for your own images and please leave any recommendations in the comments for improvements. I’m sure others will benefit. Cheers!