{"id":2557,"date":"2024-03-14T17:32:50","date_gmt":"2024-03-14T23:32:50","guid":{"rendered":"https:\/\/mrguitar.net\/?p=2557"},"modified":"2024-06-12T16:21:15","modified_gmt":"2024-06-12T22:21:15","slug":"rebuilding-container-images-with-systemd-timers-podman","status":"publish","type":"post","link":"https:\/\/mrguitar.net\/?p=2557","title":{"rendered":"Rebuilding Container Images with systemd timers & podman"},"content":{"rendered":"\n
\"\"<\/figure>\n\n\n\n

In general, it’s considered a best practice when running containers to ensure that the images are being rebuilt on a regular basis to pickup security\/bug fixes. In a real production environment, it’s common to use something like jenkins, github actions, or some type of automation or CI\/CD workflow to keep the images fresh. ….but here at my house, I only have a single server that runs containers and my use case doesn’t really warrant a more serious CI\/CD setup. This blog will show you how to setup a simple “perpetual motion” machine to automatically rebuild container images and then auto-update them. It’s also pretty easy to setup and works great too!<\/p>\n\n\n\n\n\n\n\n

First, let’s understand that podman can auto update containers<\/a> based on a registry tag or using the local image store. Most of the time, I rely on new versions landing in the registry, but since this is a single server where we’re rebuilding, I think it makes more since to use the local storage option. To use auto-updates we need to label the container with --label io.containers.autoupdate=local<\/code> If you want to read more on this subject, I recommend this blog post<\/a>. Once this is in place, enable the auto-updates timer and adjust the timer unit to your desired time to run. systemctl enable --now podman-auto-update.timer<\/code>.<\/p>\n\n\n\n

I prefer using Quadlets to define my applications. Notice the line below invokes AutoUpdate=local<\/code><\/p>\n\n\n\n

cat \/etc\/containers\/systemd\/caddy.container\n\n[Unit]\nDescription=Caddy Quadlet\n\n[Container]\nImage=localhost\/caddy:latest\nContainerName=caddy\nAutoUpdate=local\nEnvironmentFile=\/etc\/containers.environment\nVolume=caddy-data.volume:\/data\nVolume=caddy-config.volume:\/config\nVolume=\/etc\/Caddyfile:\/etc\/caddy\/Caddyfile:Z\nPublishPort=80:80\nPublishPort=443:443\n\n[Service]\nRestart=always\nTimeoutStartSec=900\n\n[Install]\nWantedBy=multi-user.target default.target\n<\/code><\/pre>\n\n\n\n
The caddy.container<\/code> file above will ensure that this container is always running, and the podman-auto-update.timer<\/code> is set to check for nightly updates. Honestly that’s pretty amazing on it’s own – especially if you remember the early days of Linux containers. From here all we need to do is create unit files to rebuild the image and automatically run it.<\/p>\n\n\n\n
 This unit will basically just run podman build and create our image. A nice option in podman is the –pull which just like docker will grab the latest image from the registry. I chose to run a prune command to prevent the system storage from filling up w\/ too many builds. Some people may not want the prune line.<\/p>\n\n\n\n
# \/etc\/systemd\/system\/caddy-build.service\n[Unit]\nDescription=Rebuild the caddy container image\n\n[Service]\nType=oneshot\nExecStart=\/usr\/bin\/podman build \u2013pull -f \/etc\/caddy\/caddy.cf -t caddy:latest\nExecStart=\/usr\/bin\/podman image prune -f<\/code><\/pre>\n\n\n\n
From here, we just need to create a timer unit w\/ the same name as the unit above and pick a time for it to run. I chose to run this weekly as it’s not super urgent for me to rebuild this frequently. For my situation monthly would be fine and I may tweak this over time as I’m still relatively new to using caddy.<\/p>\n\n\n\n
# \/etc\/systemd\/system\/caddy-build.timer\n[Unit]\nDescription=Rebuild caddy once a week\n\n[Timer]\nOnCalendar=weekly\nAccuracySec=1h\nPersistent=true\nRandomizedDelaySec=100min\n\n[Install]\nWantedBy=timers.target<\/code><\/pre>\n\n\n\n
Simply enable the above timer with systemctl enable --now caddy-build.timer<\/code> and your perpetual motion machine will be set in motion! I have no doubts that using a more modern setup like github actions or gitlab runner would enable something smarter. I do plan to move to this model at some point, but for now this seems to be working well enough and it was painless to setup. <\/p>\n\n\n\n
UPDATE!<\/strong> I also added a simple healthcheck to this setup to ensure the update is functioning. There may be a better way to do this, but I just added a handler that responds OK in my Caddyfile:<\/p>\n\n\n\n
        @healthcheck host healthcheck.mydomain.com\n        handle @healthcheck {\n                respond \"OK\" 200\n        }\n<\/code><\/pre>\n\n\n\n
Then a simple script to verify Caddy is responding and COPY<\/code> this inside the containerfile when it’s built:<\/p>\n\n\n\n
cat healthcheck.sh \n#!\/bin\/sh\ntest \"$(curl -s https:\/\/healthcheck.mydomain.com)\" = OK\n<\/code><\/pre>\n\n\n\n
Then finally add these lines to the quadlet file above:<\/p>\n\n\n\n
HealthCmd=\/healthcheck.sh\nHealthInterval=2m\nHealthOnFailure=kill\nHealthRetries=3\n<\/code><\/pre>\n\n\n\n
i wouldn’t say this gives us perfect resiliency, but it does add a good layer. This is a good blog post<\/a> if you want more details. One thing I don’t have working yet is auto-rollback and I believe I need to switch to a registry for this to work. <\/p>\n\n\n\n
At first, I tried googling for a similar setup and couldn’t find anything. I’m sharing this in case it’s useful for others, so please copy\/paste and tweak for your own images and please leave any recommendations in the comments for improvements. I’m sure others will benefit. Cheers!<\/p>\n","protected":false},"excerpt":{"rendered":"
In general, it’s considered a best practice when running containers to ensure that the images are being rebuilt on a regular basis to pickup security\/bug fixes. In a real production environment, it’s common to use something like jenkins, github actions, or some type of automation or CI\/CD workflow to keep the images fresh. ….but here … <\/p>\n
Continue reading “Rebuilding Container Images with systemd timers & podman”<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[6],"tags":[124,122,106,123],"class_list":["post-2557","post","type-post","status-publish","format-standard","hentry","category-open-sourcenerd-stuff","tag-auto-update","tag-caddy","tag-podman","tag-systemd"],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/mrguitar.net\/index.php?rest_route=\/wp\/v2\/posts\/2557","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mrguitar.net\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/mrguitar.net\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/mrguitar.net\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/mrguitar.net\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=2557"}],"version-history":[{"count":5,"href":"https:\/\/mrguitar.net\/index.php?rest_route=\/wp\/v2\/posts\/2557\/revisions"}],"predecessor-version":[{"id":2573,"href":"https:\/\/mrguitar.net\/index.php?rest_route=\/wp\/v2\/posts\/2557\/revisions\/2573"}],"wp:attachment":[{"href":"https:\/\/mrguitar.net\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=2557"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/mrguitar.net\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=2557"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/mrguitar.net\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=2557"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}