Blog Archive - Marvin Beckers

Don't Yell at Your LLM

Sun, 05 Apr 2026 17:00:00 +0000

Maybe not surprisingly, the science of how to extract maximum value from an LLMLarge Language Model, a computer program trained on large amounts of human language; Used for coding agents, for example. A coding agent is a program that uses a LLM to write and execute code. is an imprecise one. There is much advice floating around in the industry, but perhaps the most obvious one (at least to me) is “be kind to your LLM”. Not because LLMs have feelings, they don’t. But language encodes emotion. And human interactions recorded in written language are emotional to the very core.

And while the LLM doesn’t “understand” these emotions, it seems somewhat logical that being mean will result in worse results. That’s a fairly basic trait of human interaction! If you yell at someone, their answer is not going to be of better qualityA shock to choleric managers around the planet, I know.. Quite the opposite, in fact. Humans under pressure tend to produce something, but it often is somewhat useless. People tend to be more helpful when being appreciated.

It’s simply more probable that the interaction with your coding agent will go sideways if you’re not nice, because the dataset (humans interacting with each other) points in that direction. In a game of probabilityPerhaps *the* characteristic trait of LLMs, after all., ignoring this tendency seems like an unnecessary risk.

Remember the startup founder that got their production database dropped by an agent last year? This particular founder was seemingly yellingPerhaps understandably; Mid of 2025 was not a great time to get useful LLM coding output. That one is kind of on him though. at their agent for making mistakes – overall treating the agent a bit like a sub-par intern. Here is an example of what I mean. And what do stressed out interns do if yelled at? They make more mistakes, like dropping the production database. It was the most likeliest thing to happen in human interactions shaped this particular way, so the agent did it.Of course, this vastly oversimplifies what happened. But that's the thing with LLMs, right? No one really knows why they do something in particular. Oops.

Steve Klabnik expressed a similar sentiment earlier this year, so I’m not alone with this:

But I do think that the attitude you bring towards this process partially dictates your success, and I think you should be conscious of that while you go on this journey.

Is that too woo-y for you? Okay, let me make it concrete: I un-ironically believe that swearing at Claude makes it perform worse.

Maybe it’s obvious to most people, and that’s why we don’t really talk about it. Hard to say, yet I’ve seen LLM interactions I would categorize as “hostile” and “frustrated”. Yes, it’s a program running advanced math on a computer, neither have any feelings, but it’s not productive to express your negative emotions to it either.

AI labs seem to be hard at work to eliminate this “weakness”, possibly because people tend to swear at their models and they’re aware of this. You can see that some LLMs respond very differently when put under stress these days. But can they iron out a fundamental human trait, which heavily shapes language itself? I personally have my doubts about that.

The gist here is: It’s not a good idea to yell at the junior engineer that did something wrong, and the same roughly applies to your LLMIt's a bit ironic that social skills become even more important when you're no longer working with humans, but computers, but here we are with this particular flavor of AI.. The difference is that the junior engineer likely learns from screwing up if you explain – with patience – what went wrong, but the LLM might not, as soon as your explanation leaves the context window.

FOSS Warn with Self-Hosted ntfy

Fri, 13 Mar 2026 19:00:00 +0000

FOSS Warn is an open source app for Android (and Linux) that doesn’t rely on Google Play services to notify you about emergency alerts sent out to the population by authorities. I primarily use it as a replacement for the German NINA app.

Since version 1.0, FOSS Warn relies on UnifiedPush (UP) providers to handle push notifications. UnifiedPush is a relatively niche standard in the FOSS community for push notifications not handled by Google or Apple relays. I happen to host my own ntfy instance already to handle Prometheus alerts.

When FOSS Warn updated to version 1.0, I couldn’t get notifications to work. It would always fail with a somewhat cryptic error in the “Notification Self Check” flow at the “Test subscription and notifications” stage that my UP provider was not available to use. The exact error was:

Something went wrong. Can not subscribe. Please try again later. The server responded with RegisterAreaError: Your push service is invalid or not reachable. Please check your push notification server.

Turns out, I had my ntfy instance configured with authentication, and had this in my configuration:

auth-default-access: "deny-all"

As such, no anonymous access to my ntfy instance is possible, which hopefully prevents abuse. But it appears that there is no way in FOSS Warn to pass authentication information to the public alert server that registers with the UnifiedPush service (my ntfy instance, in that case).

How to Fix

The solution was to make the specific topic used by FOSS Warn available without authentication. Thankfully, this self-check page (and the “Troubleshoot notifications” menu entry) prints the full ntfy endpoint, which includes the topic. The format displayed looks something like https://<ntfy endpoint>/<ntfy topic>?up=1.

The topic is a random string and as such already kind of secret information that besides your FOSS Warn app and the alerts server, no one should be aware of to abuse.

Use the topic to set up unauthenticated access via an access control list (ACL). This can be done via configuration file or ad-hoc via the CLI. I haven’t migrated my instance to declarative configuration files for ACLs yet, so I ran:

ntfy access '*' <topic> rw

After doing this, FOSS Warn started working after another self-check and I was able to register places to receive notifications for.

2024 Roundup

Mon, 30 Dec 2024 17:00:00 +0000

As I’m sitting on a train back from the very last conference of the yearpurely as an attendee, thankfully., I’m writing this post to reflect on 2024. It’s been a pretty long year with lots of professional and personal developments and it’s the time of year to reflect on it.

I’ve recently been accepted as an Ambassador for the Cloud Native Computing Foundation, which I am absolutely thrilled about it. Seriously, there are less than 300 people worldwide that can call themselves that. Being recognized as a community voice lets the imposter syndrome kick in really hard. But I’m incredibly grateful for the opportunity.

All of this got me thinking: How did I get here?

I started working with Kubernetes in early 2018. The interview was for a local Linux sysadmin job – exactly the kind of position for a Linux nerd running Gentoo at home. In the interview I was asked if I had used docker-compose before and maybe heard of “Kubernetes”. At that time Kubernetes 1.0 had been released less than three years ago, so I was in luck – There was no expectation of having worked with it before. In hindsight, it was incredible that a small local MSP was already running Kubernetes for its customers.

I’ve since long left my first Kubernetes job, but this year saw me returning to the very same meeting room that my interview took place in – To host the local meetup group we started this year. In a way, it felt like the Ouroboros biting its own tail (yes, I absolutely had to Google that, so here’s a link). A foregone conclusion, somehow. The community around Kubernetes and its ecosystem has exponentionally grown since my first working day. We’ve been able to fill that room to the brim with people. Where there were two talking about Kubernetes, there were now many.

Working that first job I was yearning for the “big leagues” – as you do when you’re young and impatient. At the time, there were two “Kubernetes companies” in Germany, and I dreamt about them. Today, I work for one of them. It’s easy to lose yourself in the day to day work of your job, but I have to remind myself once in a while that this was my dream job – and frankly speaking, still is.

Being a CNCF Ambassador is a big milestone in my personal development. If you knew me in high school, you’d never imagine seeing me on a public stage, talking to strangers at a conference. Describing this as stepping outside my comfort zone would be a tremendous understatement. And yet, it somehow worked.

My public speaking activites started only last year, at Cloud Native Rejekts in Amsterdam. And let me tell you, it was terrifying. I did not sleep the night before. Without the chance to attend a previous KubeCon I didn’t know what to expect, COVID had destroyed my chance to attend the “original” Amsterdam KubeCon in 2020. But the cloud native community at Rejekts and KubeCon last year showed me how welcoming it is. Some amazing people approached me, told me I looked a little lost (damn right I did, because I was) and started engaging with me. I’m not sure if they even remember doing that but it meant the world to me in that moment. So, thank you, kind community folks!

All things considered the year 2024 has been pretty crazy for me. My plans for 2025 are stepping up my role as CNCF Ambassador in public speaking and our local meetup community. kcp is starting to build some momentum as a community project, which feels extremely gratifying, and I hope to continue contributing to its success in the coming year.

One last thing though – 2024 has been a great year for me personally, but let’s be honest, the world has been moving fast under our feet and in many ways not in the right direction. 2025 will likely be less kind, less certain, less factual. It is our collective responsibility to keep our communities and those around us safe. A little compassion will go a long way.

Kubernetes Homelab #2: KubeOne Cluster

Fri, 30 Aug 2024 20:00:00 +0000

After finally receiving the third Raspberry Pi and additional pieces of hardware (and being busy with, well, life) I was able to move to the next part of this blog series: Setting up Kubernetes with KubeOne (full disclosure: KubeOne is developed by my employer Kubermatic) and adding a hyperconverged storage layer to the cluster. For that I am using Longhorn. This post will focus on the cluster setup, while the next post will discuss Longhorn and the distributed storage aspect.

One requirement I had for the setup: Any of the three Raspberry Pis are allowed to fail and the cluster stays up and running. High Availability might be a little overkill for a home setup, but this one breaks with so many best practices already (you will see later in this post and the series) that – at least – it should always stay up, even if one node goes down.

During the initial setup, some of the systems occasionally lost connection to their disk and the system crashed. So far, I haven’t figured out if the cause is one of the hats (e.g. PoE not delivering enough power for a short period), but this meant the cluster needed to be resilient to node failure.

Let’s dive right in.

This post is part of a blog post series describing my Kubernetes Homelab.

Part 1: Raspberry Pi Setup
Part 2: Cluster Installation with KubeOne (this post)
TBD: Hyperconverged Storage with Longhorn
TBD: Exposing Cluster and Services through Tailscale

If you skipped the first part, I’d recommend to go back and at least read the post-boot steps. Without applying them to the Raspberry Pis they cannot be used as Kubernetes nodes.

I’m using KubeOne over similar tools like k3s because I’m already familiar with it. Its declarative approach to cluster setups feels very similiar to Cluster API without the need to have a management cluster. Let’s also be honest – These machines are no longer embedded devices, which are k3s’ targets. They have the CPU and memory footprint of a large instance on AWS, after all.

Virtual IP for Kubernetes API

The first thing I needed was a virtual IP for the Kubernetes API. Why? Because the nodes of the Kubernetes cluster try to reach the cluster’s control plane. The Kubernetes API is the beating heart of the cluster and all control plane data (i.e. which pods are scheduled onto which node, the status of pods, etc) flows through it. Seriously, it’s a pretty amazing apparatus.

Anyway – to make sure that nodes can reach the Kubernetes API, no matter where it runs (since there will be three nodes acting as control plane and worker), we need a failover mechanism for an unchanging IP. For that purpose I am using keepalived. keepalived uses VRRP (Virtual Router Redundancy Protocol) to determine that it needs to take over a static IP address and attach it to its own network interface. The details of how this works are quite important, but out of scope for this post.

Usually keepalived is available from distribution repositories, so it only needs this:

$ apt install keepalived

To make sure it starts at boot (e.g. after a crash):

$ systemctl enable keepalived.service

keepalived Configuration

For this setup, keepalived needs two files: A configuration file and a script to check if the Kubernetes API is healthy. Thankfully KubeOne already provides a vSphere example from which I could lift the keepalived parts.

Here is the check_apiserver.sh script:

#!/bin/sh

errorExit() {
    echo "*** $*" 1>&2
    exit 1
}

curl --silent --max-time 2 --insecure https://localhost:6443/healthz -o /dev/null || errorExit "Error GET https://localhost:6443/healthz"
if ip addr | grep -q 192.168.178.201; then
    curl --silent --max-time 2 --insecure https://192.168.178.201:6443/healthz -o /dev/null || errorExit "Error GET https://192.168.178.201:6443/healthz"
fi

This script first checks if the Kubernetes API is up and running on port 6443 of localhost, and if the local system also holds the virtual IP it will also check if the Kubernetes API is reachable on port 6443 of said virtual IP.

Now comes the configuration file. keepalived instances require a shared secret, so I generated one via:

$ tr -dc A-Za-z0-9 </dev/urandom | head -c 8; echo

Configuration slightly differs between master and backup instances. The file deployed to raspi-01, which I wanted to be my “default” instance (so, the master, in keepalived terminology), looks like this:

lobal_defs {
    router_id LVS_DEVEL
    script_user root
    enable_script_security
}

vrrp_script check_apiserver {
  script "/etc/keepalived/check_apiserver.sh"
  interval 3
  weight -2
  fall 10
  rise 2
}

vrrp_instance VI_1 {
    state MASTER
    interface eth0
    virtual_router_id 55
    priority 100
    unicast_src_ip 192.168.178.93

    authentication {
        auth_type PASS
        auth_pass <password> # <- this needs to be replaced!
    }

    virtual_ipaddress {
        192.168.178.201/24
    }

    track_script {
        check_apiserver
    }
}

I’m honestly not a keepalived expert, so this configuration is kind of “best effort”. For raspi-02 and raspi-03, the file looks mostly the same:

# ...
vrrp_instance VI_1 {
    state BACKUP
    interface eth0
    virtual_router_id 55
    priority 80 # <- respectively 75, for raspi-03
    # ...
}
# ...

Those files are written to /etc/keepalived/keepalived.conf. Starting keepalived on all machines was necessary to get the virtual IP, so I ran this command on all of them:

$ systemctl start keepalived.service

Kubernetes Setup

KubeOne is a command line tool for Linux and macOS that bootstraps a Kubernetes cluster from a declarative configuration file. It has integration with Terraform/OpenTofu to let the Infrastructure-as-Code (IaC) software handle creation of VMs and ingest IaC output to install Kubernetes. Provisioning of machines as Kubernetes nodes happens via SSH. It’s licensed under Apache-2.0, so it’s free to use (and fork).

The easiest way to install KubeOne is:

$ curl -sfL https://get.kubeone.io | sh

If piping an unknown bash script to your shell is making you uncomfortable, binaries are also available from GitHub releases directly. I also maintain a small homebrew tap that includes a kubeone formula.

KubeOne Configuration

Once (or before) KubeOne was installed I had to craft a declarative configuration for it. Below is the complete configuration file (kubeone.yaml) I used to run KubeOne and set up a Kubernetes cluster across the three Raspberry Pis (if you are looking for more instructions on how to set up this configuration, the KubeOne documentation might be helpful):

apiVersion: kubeone.k8c.io/v1beta2
kind: KubeOneCluster
name: raspi
versions:
  kubernetes: '1.29.8'
cloudProvider:
  none: {}

controlPlane:
  hosts:
    # raspi-01
    - publicAddress: '192.168.178.93'
      privateAddress: '192.168.178.93'
      sshUsername: embik
      taints: []
    # raspi-02
    - publicAddress: '192.168.178.98'
      privateAddress: '192.168.178.98'
      sshUsername: embik
      taints: []
    # raspi-03
    - publicAddress: '192.168.178.104'
      privateAddress: '192.168.178.104'
      sshUsername: embik
      taints: []

apiEndpoint:
  host: '192.168.178.201'
  port: 6443

machineController:
  deploy: false

The gist of this file is: Please create a Kubernetes cluster with version 1.29.8, use three machines as control planes and configure them to use the virtual IP as Kubernetes API endpoint.

Let’s look at this file in detail. We’ll go top to bottom, starting with this:

apiVersion: kubeone.k8c.io/v1beta2
kind: KubeOneCluster
name: raspi
# ...

KubeOne configuration files look similar to Kubernetes manifests on purpose – both work declaratively, and the configuration file reflects that. What is described above is the target state of the cluster, and I want KubeOne to bring the three Raspberry Pis to that state. I don’t care how. Because of that, the “header” of the file looks pretty similar to a Kubernetes object, defining the API version and the cluster name (raspi).

# ...
versions:
  kubernetes: '1.29.8'
# ...

This is the Kubernetes version that should be installed onto the machines. In subsequent runs, updating this (e.g. to Kubernetes 1.30.x) will prompt KubeOne to run a minor version upgrade. Support for minor Kubernetes versions in KubeOne is documented here.

# ...
cloudProvider:
  none: {}
# ...

KubeOne supports integration with various cloud providers. Because this is a bare-metal setup, there is no cloud provider to integrate with, and thus the selected cloud provider is none.

# ...
controlPlane:
  hosts:
    # raspi-01
    - publicAddress: '192.168.178.93'
      privateAddress: '192.168.178.93'
      sshUsername: embik
      taints: []
    # raspi-02
    - publicAddress: '192.168.178.98'
      privateAddress: '192.168.178.98'
      sshUsername: embik
      taints: []
    # raspi-03
    - publicAddress: '192.168.178.104'
      privateAddress: '192.168.178.104'
      sshUsername: embik
      taints: []
# ...

This is the list of machines that are supposed to form the Kubernetes cluster once KubeOne is done with its work. I’m passing their IP addresses as both public and private (since there is no internal/external network, a Raspberry Pi doesn’t have two ethernet connectors) and my SSH username. That way, KubeOne will use my SSH agent to connect to these machines to bootstrap and configure them appropriately.

One note should be on taints: [] for each list entry. Usually, a Kubernetes control plane has something called taints applied to them on setup; Taints provide a way to mark nodes as not suitable for scheduling unless a toleration for the taint is configured on a Pod. For this setup, everything has to run on the three nodes that serve as control plane – there are no other machines. Therefore, I’m instructing KubeOne to not apply any taints to them.

# ...
apiEndpoint:
  host: '192.168.178.201'
  port: 6443
# ...

This configures all machines to use 192.168.178.201 as the Kubernetes API endpoint (even control plane nodes need to check in with the central Kubernetes API). This is made possible by the previous work on allocating a virtual IP through keepalived.

# ...
machineController:
  deploy: false

Because this is not a cloud environment, there is no way to dynamically provision machines (well, something like Tinkerbell allows to do so, but I wasn’t planning on buying more hardware [for the moment]). machine-controller is a suplimentary component we develop at Kubermatic which allows dynamic provisioning, but since I have no use for it, it’s not deployed.

Running KubeOne

With everything ready, the only thing left was flipping the switch. So I flipped the switch:

$ kubeone apply -m kubeone.yaml

KubeOne connects to the given machines, discovers their current status and then determines which steps should be taken. Before doing so, it asks for confirmation.

INFO[10:22:11 CEST] Determine hostname…
INFO[10:22:18 CEST] Determine operating system…
INFO[10:22:20 CEST] Running host probes…
The following actions will be taken:
Run with --verbose flag for more information.
  + initialize control plane node "raspi-01" (192.168.178.93) using 1.29.8
  + join control plane node "raspi-02" (192.168.178.98) using 1.29.8
  + join control plane node "raspi-03" (192.168.178.104) using 1.29.8

Do you want to proceed (yes/no):

After confirming, KubeOne does its thing for a couple of minutes.

...
INFO[10:32:54 CEST] Downloading kubeconfig…
INFO[10:32:54 CEST] Restarting unhealthy API servers if needed...
INFO[10:32:54 CEST] Ensure node local DNS cache…
INFO[10:32:54 CEST] Activating additional features…
INFO[10:32:56 CEST] Applying canal CNI plugin…
INFO[10:33:10 CEST] Skipping creating credentials secret because cloud provider is none.

Cluster provisioning finishes at this point. KubeOne leaves a little present in its wake, a kubeconfig file created in the working directory. In my case, that was raspi-kubeconfig. This is the “admin” kubeconfig to access the fresh cluster. And indeed, checking up on the cluster is possible now:

$ export KUBECONFIG=$(pwd)/raspi-kubeconfig # use raspi-kubeconfig as active kubeconfig
$ kubectl get pods -n kube-system
NAME                                       READY   STATUS    RESTARTS   AGE
calico-kube-controllers-699d6d8b48-mgkp4   1/1     Running   0          10m
canal-4dmvq                                2/2     Running   0          10m
canal-ddn8v                                2/2     Running   0          10m
canal-j46lz                                2/2     Running   0          10m
coredns-646d7c4457-s95hd                   1/1     Running   0          10m
coredns-646d7c4457-w48zx                   1/1     Running   0          10m
etcd-rpi-01                                1/1     Running   0          10m
etcd-rpi-02                                1/1     Running   0          10m
etcd-rpi-03                                1/1     Running   0          10m
kube-apiserver-rpi-01                      1/1     Running   0          10m
kube-apiserver-rpi-02                      1/1     Running   0          10m
kube-apiserver-rpi-03                      1/1     Running   0          10m
# ...

It’s alive, Jim!

This concludes the second part of my Kubernetes homelab series. At this point, I had a functional Kubernetes cluster, but a couple of open questions remained: Where would stateful applications store their data? How could I access any applications running within this cluster? These questions will be addressed in part three and four of this series, which I will hopefully write quicker than this one. Stay tuned!

Thinking, Fast and Slow (Daniel Kahneman)

Thu, 22 Aug 2024 10:00:00 +0000

Thinking, Fast and Slow is a non-fiction book by psychologist Daniel Kahneman published in 2011, discussing the human decision-making process and the flaws that come with it. If I had to break it down to one sentence, the book challenges the perception of humans as purely rational beings (“econs”) by describing various mechanisms of the human mind that can produce inconsistent results.

Kahneman died earlier this year after working on the research described in the book for decades and reaching highest academic acclaim, being dubbed “godfather of behavioral economics”. I had known about Thinking, Fast and Slow before, but his death prompted me to add the book to my reading list and I finally picked it up in July.

Summary

Each of the five parts of the book looks at different aspects of decision-making mechanisms observed by Kahneman.

It begins with introducing two systems involved in our decision-making process: System 1, which operates subconsciously and is responsible for reflexive and intuitional decisions, and System 2, which is conscious and what we consider as ourselves when we think. The case for these two systems (which are mostly described as actors throughout the book to make it easier to conceptualize them) is rather compelling.

They both have strengths and weaknesses (e.g. system 1 can handle averages and peaks well, but is not good at handling sums; system 2 is much better at dissecting problems, but is “lazy” and while tasked with validating inputs from system 1, tends to make mistakes at that verification process) and interlace to come up with decisions we make every minute.

Concepts

A (non-exhaustive) list of interesting concepts that the book proposes:

Substitution is a process in which a harder question is replaced with an easier question to answer the original question. This happens without the person realizing that they aren’t answering the original question but an easier one.

Intensity matching describes a more specific case of substitution: Transferring a value on one scale (e.g. your emotional response to a heartbreaking story) to another scale (e.g. a financial contribution to a good cause related to the prior story) to (intuitively) decide an appropriate value.

Regression to the mean is one of several examples where human intuition is beaten by statistics: We tend to see a particular good or bad day/event/occurrence as evidence of overall performance, but stasticially an outstanding event (e.g. a goalkeeper being able to catch every or none of the shots at the goal) is followed by an event closer to average. An outstanding game by the goalkeeper is more likely to be followed by a more average one but we tend to expect a similar outstanding game and are disappointed when it doesn’t happen.

Anchoring is when the context of a decision sets the stage by creating a reference point from which the decision-making process starts out. This is again a subconscious process: We are not aware it happens. But asking people to estimate something (e.g. a historical fact) above or below an arbitrary number supposedly changes their answers.

Prospect theory which is a little hard to put into a couple of sentences. If I’d still try I would describe it as following: It’s a counter-argument (or rather, an amendment) to utility theory which postulates that (rational) agents maximize the utility of a decision. Prospect theory however claims that human decision-making is more complex and driven by divergence from a status quo (instead of absolute numbers), by diminishing sensitivity (stark differences are more impactful) and by loss aversion (losses have a higher impact than gains).

Expert Intuition

A very fascinating part of the book is the discussion of expert intuition. Kahneman worked on this topic with Gary Klein, who Kahneman describes as somewhat adversarial in the book. Kahneman’s synthesis of their opposing positions is: Expert intuition can only be skilled (as in: have a higher chance of being correct than a coin toss) in sufficiently predictable environments with immediate feedback loops. A firefighter (the example from the book) can pick up on non-obvious cues that forebode escalations because they have experienced similar events in the past. Intuition is basically just memory retrieval and pattern matching.

A stock picker however cannot learn to correctly forecast stock performance because the stock market is a “zero-validity” environment. And the book makes an even bolder claim: If you actually validate stock picking skills, you will encounter minuscule differences from pure chance. In a market in which ETFs seem to perform more consistently than manually picked stocks, this rings true. Remarkable – according to the book – is the cognitive dissonance when confronted with such performance metrics: The results are ignored and everyone carries on as if they had the skills that statistics have been unable to verify.

Conclusion

There are many more topics discussed in great detail in the book, but the conclusion it draws is clear: Human decision-making is driven by evolutional features that were of great help surviving difficult conditions, but they have not fully adjusted to the kind of decisions that humans need to make in the present day. The “rational agents” described by economic theory (the book calls them “econs”) do not exist in that shape and form, and humans are susceptible to flawed heuristics and biases.

The conclusion of the book draws a line from all that to what it means for policy making. The gist is that humans do not necessarily need to be “kept from harm”, but they need to be provided with decision-making input in formats that do not skew their perception. Marketing departments and public opinion drivers are aware of psychological research as described in this book, and Kahneman adopts the standpoint that policy makers need to be aware of biases and heuristics and should set policies that enforce neutral information to minimize framing, anchoring, biases or similar flaws in decision making.

Review

Somewhat surprisingly, reading this book has been a breeze. The topic is difficult (and should be dry, given the amount of discussion of research results), but the writing style made it easy to follow the threads spun by Kahneman, much more than I anticipated. It’s certainly a book that requires breaks to process and think about the concepts, but I finished it way faster than was reasonable for me.

A criticism leveled against the book is that the countless studies it references are caught up in the replication crisis of social sciences – the inability to reproduce them successfully or consistently. That’s a very valid point, most of the book’s claims only work if the described behaviour can be considered somewhat universal. It becomes moot if humans simply behave differently.

To a layperson, a lot of the book makes intuitive sense (but I just learned to not always trust intuition) and the examples seem to create the cognitive knots described in the book, even when I was aware that I’m going to encounter a logical fallacy. Observing myself working through those has been a fascinating experience. The book provides some underpinning to fuzzy beliefs I’ve held before but had trouble articulating (e.g. that status quo plays a huge role in decision making) which makes it both compelling and precarious (as it makes me prone to confirmation bias).

Software engineers can – in my opinion – learn from this book, in particular from the review of expert intuition. We all know situations in which expert intuition is desired (say, story point estimations) but the responses are usually extremely poor and there is basically not feedback loop. And remarkably, those situations share some other characteristics with the case made in the book: A whole industry feels that story point estimations are bordering on the useless, but a competing (business) interest creates a cognitive dissonance we have simply accepted. If we accepted that human decision making is simply not up to the task, we could maybe stop wasting time on it.

Describing humans as purely rational beings has also been the theoretical underpinning for predatory, self-serving ideologies, and only if we acknowledge that these foundations are extremely shaky we can put policies in place that improve human well-being and social constructs.

Overall, the book highlights aspects we all can pay attention to while making decisions and even proposes processes to insulate organizations and societies from decision-making mistakes. Stopping ourselves in our tracks to review a decision and test it for biases makes universal sense, even if the models described in Thinking, Fast and Slow are not as useful or correct as they are made out to be. I’d strongly recommend this book to anyone interested in decision-making processes and introspection, but it should be taken as a (likely flawed) attempt at modelling of, not an exhaustive guide to, human behaviour.

Kubernetes Homelab #1: Raspberry Pi Setup

Thu, 09 May 2024 18:00:00 +0000

I like to tinker with Kubernetes in my free time and I’ve always wanted to host some services for myself. Vaultwarden is one of those services since I’d like my passwords to be stored away where I can see them. Up to now this was running on an old Intel NUC that I stashed away in my apartment.

Years ago I had been using Raspberry Pis (generation 1 or 2, maybe?) for hosting some fun projects (e.g. a TV antenna receiver attached to a Raspberry Pi to watch TV – when that was still a thing).

This post is part of a blog post series describing my Kubernetes Homelab.

Part 1: Raspberry Pi Setup (this post)
Part 2: Cluster Installation with KubeOne
TBD: Hyperconverged Storage with Longhorn
TBD: Exposing Services through Tailscale

Now I wanted to try again because cloud is expensive, Kubernetes can be quite fun and I hadn’t tinkered with hardware in a good while. So my goal was to get a setup that

had three nodes to make sure Kubernetes made sense. A single node isn’t really a cluster.
provided its own storage. A NAS is expensive and a single point of failure, so “hyperconverged” (is that word still in use at all? it was the big hype in on-premise hardware ten years ago) was the way to go.
didn’t boot from an SD card. Maybe this is fine now but I had many corrupted SD cards I needed to reformat back in the days. I wanted my machines to boot from a disk.
used power over ethernet (PoE) to reduce the cables needed for powering the setup.
allowed to expose services to me as its only user with valid TLS certificates and hostnames.

The blog post series that this post kicks off will map out my journey with getting my homelab up and running and document any steps and problems along the way.

Note: All commands in this post are executed as root. You can either prepend them with sudo or open a root shell once with sudo -i.

Hardware

The core of the setup is comprised of three Raspberry Pi 5 B with a 2.44 GHz ARM Cortex-A76 Quad-Core-CPU and 8GB of RAM. These machines are barely comparable to the original Raspberry Pi. They are quite the workhorses!

To power the Raspberry Pis over ethernet via PoE I bought a TP Link TL-SG1005P (a 5-port gigabit desktop PoE+ switch).

I ended up using the following HATs for the three Raspberry Pi 5 B, bought from Amazon:

Waveshare is a Chinese manufacturer of electronic components for microcontrollers and embedded boards. The choices for Raspberry Pi 5 PoE HATs is quite limited at the moment. An official PoE HAT had been announced, but updates have been rare and search results mostly end on a post half a year old. Jeff Geerling reviewed this PoE HAT (see his blog and the GitHub issue) and overall it looked promising.

Raspberry Pi 5 and both HATs.

M.2 SSD

I went with Waveshare’s PCIe to M.2 HAT+ to make sure the two HATs would work with each other. However it seems to be hit or miss whether an M.2 SSD is going to be supported by the HAT or not. I started the setup with a Transcend TS256GMTS430S (a 256GB M.2 SSD), but the disk would not get recognized, even after following all troubleshooting steps. After some back and forth with Waveshare support I got the recommendation to use a Western Digital WD BLACK SN770M, which worked like a charm.

As is often the case with these kind of manufacturers, documentation is sparse. It was more or less impossible to figure out why the Transcend SSD wasn’t recognized – some sources were talking about the Raspberry Pi PCIe not working with a specific on-board controller on M.2 SSDs, but as far as my Google research suggested this SSD didn’t seem to be using that controller. In the end things worked out by exchanging hardware, thus it doesn’t seem a good idea to combine this particular HAT with Transcend M.2 SSDs.

Assembly

Overall assembly worked fine but required some careful strength applied at the right times. I assembled the system with the PoE HAT being attached to the Pi first as the lower layer and then added the PCIe to M.2 HAT on top of it.

Beware: It's possible to mount the heat sink coming with the PoE HAT the wrong way (either there are no markings or I missed them) and it's really not built to be removed to remount it. So make sure you mount it the right way (the longer edge should be on the side of the SD card slot), otherwise you won't be able to attach the PoE HAT.

The cable provided with the PCIe to M.2 HAT to connect the Pi’s PCIe slot to the HAT was just long enough to work in this stacked setup. Connecting it properly on both ends was a bit fiddly (especially on the Pi side – the HAT side has a very solid mechanism to hold the cable in place), tweezers came in very handy.

Cable to connect PCIe slot to M.2 HAT (with the non-functional SSD mounted).

System setup

The best choice of OS on the Raspberry Pi currently seems to be Raspberry Pi OS. It’s based on Debian, which is great because KubeOne (disclaimer: I’m a contributor to it and it’s developed by my employer, Kubermatic) has (limited) support for creating Kubernetes clusters on it.

Before booting from the M.2 SSD a quick detour via a bootable USB stick was needed. I used the Raspberry Pi Imager to write the latest Raspbbery Pi OS “lite” (no desktop environment included) to a spare 32GB USB stick. One of the important customizations here is enabling SSH and adding a SSH public key.

Plugging in the USB stick and powering on the Raspberry Pi by connecting the ethernet cable started Raspberry Pi OS from the USB stick. I used this transient setup to check on the M.2 SSD disk and prepare it to become the boot device. To find its IP address I signed into my home router’s web UI, found the newly connected device and made sure to configure the device to always receive the same IP address from the router’s DHCP server.

Setting up static IPs here is actually pretty important for the Kubernetes setup later. Kubernetes doesn't really expect nodes to change IP addresses, and all three of my Pis are going to be control plane nodes; this "rule" applies especially to the control plane and might otherwise result in breaking the cluster.

After connecting via SSH I had to make sure the M.2 SSD was actually working. That required configuration so that the Raspberry Pi would start with NVMe support which can be added by editing /boot/firmware/config.txt and adding the following line:

dtparam=nvme

On one of the Pis also refused to boot from NVMe (in one of the later steps) before updating its firmware. To do so I ran the following commands (still booting from the USB stick):

$ apt update
$ apt upgrade
$ rpi-update

For this and the dtparam addition to take effect a reboot is required. Afterwards, the SSD showed up as block device:

$ lsblk
NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
sda           8:0    1  28.7G  0 disk
|-sda1        8:1    1   512M  0 part /boot/firmware
`-sda2        8:2    1  28.2G  0 part /
nvme0n1     259:0    0 465.8G  0 disk

At this point the SSD was showing up so I was able to prepare it to become the system’s boot disk. First of all I needed a Raspberry Pi OS (64bit) lite image on the system:

$ curl -O https://downloads.raspberrypi.com/raspios_lite_arm64/images/raspios_lite_arm64-2024-03-15/2024-03-15-raspios-bookworm-arm64-lite.img.xz
$ unxz 2024-03-15-raspios-bookworm-arm64-lite.img.xz

Next step was writing the image to disk:

$ dd if=./2024-03-15-raspios-bookworm-arm64-lite.img of=/dev/nvme0n1

This resulted in a partition table on /dev/nvme0n1 so the output of lsblk changed:

$ lsblk
[...]
nvme0n1     259:0    0 465.8G  0 disk
|-nvme0n1p1 259:1    0   512M  0 part
`-nvme0n1p2 259:2    0   2.1G  0 part

OS modifications

The OS written to disk did not include modifications to access after booting it. I skipped the Raspberry Pi image writer this time and did the modification in a chroot environment:

$ mount /dev/nvme0n1p2 /mnt
$ mount /dev/nvme0n1p1 /mnt/boot
$ chroot /mnt

The /boot/config.txt file in this environment also required dtparam=nvme since it was separate from the file (on the USB stick environment) previously adjusted.

Next up was changing the hostname:

$ echo "rpi-01" > /etc/hostname

To make sure that DNS resolution for its own name was working properly a slight modification to /etc/hosts and ensuring that the entry for 127.0.1.1 included the new hostname was necessary:

127.0.1.1		rpi-01 raspberrypi

For access to the system after boot I configured the existing user pi to have my SSH key. In addition /boot/ssh tells the OS to start sshd on boot.

$ mkdir -p /home/pi/.ssh
$ echo "ssh-ed25519 AAAA[...] embik" > /home/pi/.ssh/authorized_keys
$ chown -R /home/pi/.ssh pi:pi
$ touch /boot/ssh

I wanted to rename the pi user to embik on first boot by setting /boot/userconf.txt. Unfortunately you always need to pass a hashed password which can be generated by openssl passwd -6. Part of the post-boot steps was going to be removing the set password anyway.

$ echo "embik:$6$..." > /boot/userconf.txt

Everything on the NVMe disk was ready, so the only thing left there was to leave the chroot environment and make sure the disk was cleanly unmounted.

$ exit
$ umount /mnt/boot
$ umount /mnt

Boot from NVMe

Last step was changing the boot order to make sure the next time the Raspberry Pi started, it would boot from my M.2 SSD.

$ rpi-eeprom-config --edit

I changed BOOT_ORDER to 0xf416 which attempts boot from NVMe and falls back in case the NVMe disk isn’t bootable. In addition this configuration needed PCIE_PROBE=1. Now everything is set and ready to reboot the system into the M.2 SSD.

A curious observation from my first setup was that while 0xf416 should describe the boot order “NVMe -> SD card -> USB stick” (see Raspberry Pi documentation), my Pi would not successfully boot from the still inserted SD card when my NVMe disk wasn’t formatted properly. I had to use an USB stick to recover the system.

Post-boot steps

Once rebooted, ssh became available after a couple of seconds (booting from the M.2 SSD makes it essentially another system, so ssh-keygen -R was necessary before being able to connect). The system indeed booted from the NVMe disk and mounted its root partition:

$ lsblk
nvme0n1     259:0    0 465.8G  0 disk
|-nvme0n1p1 259:1    0   512M  0 part /boot/firmware
`-nvme0n1p2 259:2    0 465.3G  0 part /

To make sure that no one would be able to use password-based SSH authentication I deleted the user’s password:

$ sudo passwd -d embik

Two last things were necessary to prepare the Pi for Kubernetes: Disabling swap and enabling the memory group. The Raspberry Pi OS seems to have some special swapfile generation (swap isn’t mounted in /etc/fstab as usual), so this seems to be the way to disable swap on next boot:

$ systemctl disable dphys-swapfile.service

By default Raspberry Pi OS doesn’t enable the memory cgroup, which is a hard requirement for Kubernetes. That can be adjusted in /boot/firmware/cmdline.txt by appending two additional boot parameters (requires a reboot):

cgroup_enable=memory cgroup_memory=1

Rinse and repeat two more times to get all three Pis set up correctly.

Next up

Hardware is all set up now! The next post will discuss setting up Kubernetes on these Raspberry Pis with KubeOne and keepalived. Stay tuned and subscribe to my blog via RSS to not miss any upcoming posts.

Sources

Nine Kubernetes Tools You Might Not Know

Sat, 27 Jan 2024 00:00:00 +0000

Everyone working with Kubernetes (mostly likely) has kubectl installed. Most people also have helm. But what other tools are out there for your daily work with Kubernetes and containers? This post explores a couple of projects that range from somewhat known to heavily obscure, but all of them are part of my daily workflows and are my recommendations to aspiring (and seasoned) Kubernetes professionals.

Let’s dive right into our list!

protokol

Repository: github.com/xrstf/protokol

This one takes the cake as “most obscure” because I am the only person who has starred it on GitHub at the time of writing this. People are seriously missing out.

protokol is a small tool by my friend Christoph (also known as xrstf) that allows you to easily dump Kubernetes pod logs to disk for later analysis. This is especially useful in environments that do not have a logging stack set up that you can query later on. For example, to get all logs from the kube-system namespace until you stop the protokol command (e.g. with Ctrl+C), run:

$ protokol -n kube-system
INFO[Sat, 27 Jan 2024 11:51:47 CET] Storing logs on disk.                         directory=protokol-2024.01.27T11.51.47
INFO[Sat, 27 Jan 2024 11:51:47 CET] Starting to collect logs…                     container=coredns namespace=kube-system pod=coredns-787d4945fb-4q7jv
INFO[Sat, 27 Jan 2024 11:51:47 CET] Starting to collect logs…                     container=coredns namespace=kube-system pod=coredns-787d4945fb-ghskz
INFO[Sat, 27 Jan 2024 11:51:47 CET] Starting to collect logs…                     container=etcd namespace=kube-system pod=etcd-lima-k8s
INFO[Sat, 27 Jan 2024 11:51:47 CET] Starting to collect logs…                     container=kube-controller-manager namespace=kube-system pod=kube-controller-manager-lima-k8s
INFO[Sat, 27 Jan 2024 11:51:47 CET] Starting to collect logs…                     container=kube-proxy namespace=kube-system pod=kube-proxy-rppbc
INFO[Sat, 27 Jan 2024 11:51:47 CET] Starting to collect logs…                     container=kube-scheduler namespace=kube-system pod=kube-scheduler-lima-k8s
INFO[Sat, 27 Jan 2024 11:51:47 CET] Starting to collect logs…                     container=kube-apiserver namespace=kube-system pod=kube-apiserver-lima-k8s
^C
$ tree
.
└── protokol-2024.01.27T11.51.47
    └── kube-system
        ├── coredns-787d4945fb-4q7jv_coredns_008.log
        ├── coredns-787d4945fb-ghskz_coredns_008.log
        ├── etcd-lima-k8s_etcd_008.log
        ├── kube-apiserver-lima-k8s_kube-apiserver_006.log
        ├── kube-controller-manager-lima-k8s_kube-controller-manager_010.log
        ├── kube-proxy-rppbc_kube-proxy_008.log
        └── kube-scheduler-lima-k8s_kube-scheduler_010.log

3 directories, 7 files

protokol comes with a huge set of flags to alter behaviour and to target specific namespaces or pods. It is really useful in troubleshooting situations where you want to grab large parts of the cluster’s current logs, e.g. to grep for certain things. It’s also quite nice in CI/CD systems where logs of pods should be downloaded as artifacts that will be stored alongside the pipeline results.

Tanka

Website: tanka.dev
Repository: github.com/grafana/tanka

Do you remember ksonnet? No? A lot of people probably don’t. The ksonnet/ksonnet repository was archived in September 2020, which feels like a lifetime ago. ksonnet used to provide Kubernetes-specific tooling based on the jsonnet configuration language, which is basically a way to template and composite JSON data. The generated JSON structures can be Kubernetes objects, which can be converted to YAML or sent to the Kubernetes API directly. In essence, this was an alternative way to distribute your Kubernetes manifests with configuration options.

ksonnet ceased development but Grafana decided to revive the idea with tanka, which was really nice. The unfortunate truth is that jsonnet is very niche, so niche that the syntax highlighting for my blog doesn’t even support it. The only major project outside of Grafana that seems to use jsonnet is kube-prometheus (which doesn’t use tanka, unfortunately).

I personally find the syntax of it great though, much better than Helm doing string templating on YAML. See below for a jsonnet snippet that generates a full Deployment object:

local k = import "k.libsonnet";

{
    grafana: k.apps.v1.deployment.new(
        name="grafana",
        replicas=1,
        containers=[k.core.v1.container.new(
            name="grafana",
            image="grafana/grafana",
        )]
    )
}

You might feel some resistance to introducing tanka to your workplace because it has a learning curve, but once it clicks you never want to go back to helm. If you get buy-in from your colleagues this might be a huge win – The ability to provide standardized libraries to generate manifests can be extremely helpful in providing a consistent baseline to teams. So it might be worth trying it out for your next project.

stalk

Repository: github.com/xrstf/stalk

Another tool made by xrstf! stalk allows you to observe the changes in Kubernetes resources over time. This can be very useful if you just can’t understand what is happening to your Deployment (or any other resource) if you struggle to observe changes when running kubectl get. Usually, this is most needed when your Kubernetes controller goes into a reconciling loop. stalk to the rescue - it will show diff formatted output with timestamps when changes happen.

In the example below, the observed changes are limited to the .spec field of a Deployment. So stalk will start by showing .spec at start time, and then log any changes it observes over time (in the example, the Deployment has been scaled down to one replica later on):

$ stalk deployment sample-app -s spec
--- (none)
+++ Deployment default/sample-app v371701 (2024-01-27T12:43:10+01:00) (gen. 2)
@@ -0 +1,30 @@
+spec:
+  progressDeadlineSeconds: 600
+  replicas: 2
+  revisionHistoryLimit: 10
+  selector:
+    matchLabels:
+      app: sample-app
+  strategy:
+    rollingUpdate:
+      maxSurge: 25%
+      maxUnavailable: 25%
+    type: RollingUpdate
+  template:
+    metadata:
+      creationTimestamp: null
+      labels:
+        app: sample-app
+    spec:
+      containers:
+      - image: quay.io/embik/sample-app:latest-arm
+        imagePullPolicy: IfNotPresent
+        name: sample-app
+        resources: {}
+        terminationMessagePath: /dev/termination-log
+        terminationMessagePolicy: File
+      dnsPolicy: ClusterFirst
+      restartPolicy: Always
+      schedulerName: default-scheduler
+      securityContext: {}
+      terminationGracePeriodSeconds: 30

--- Deployment default/sample-app v371701 (2024-01-27T12:43:10+01:00) (gen. 2)
+++ Deployment default/sample-app v371736 (2024-01-27T12:43:13+01:00) (gen. 3)
@@ -1,6 +1,6 @@
 spec:
   progressDeadlineSeconds: 600
-  replicas: 2
+  replicas: 1
   revisionHistoryLimit: 10
   selector:
     matchLabels:

No more head scratching when two controllers compete on specific fields and update them several times a second.

Inspektor Gadget

Website: inspektor-gadget.io
Repository: github.com/inspektor-gadget/inspektor-gadget

If the tools in this blog form a toolbox, Inspektor Gadget is the toolbox in the toolbox. For someone with a sysadmin background (like me) this is a treasure trove when troubleshooting low-level issues. The various small tools in Inspektor Gadget are called – unsurprisingly – gadgets and are based on eBPF. You can even write your own gadgets!

Inspektor Gadget consists of a client component (which is a kubectl plugin) and a server component, which runs as DaemonSet on each Kubernetes node (after installing it).

In its essence gadgets give you access to system data you could also fetch from a Kubernetes node’s shell via SSH, but Inspektor Gadget allows to fetch and process this data with the context of containers and across nodes. To just show two of the many available gadgets, below is a snapshot of active sockets in all pods in the current namespace (which usually would be much more):

$ kubectl gadget snapshot socket
K8S.NODE                 K8S.NAMESPACE            K8S.POD                  PROTOCOL SRC                            DST                            STATUS
lima-k8s                 default                  sample-app-6…bf695-4cv89 TCP      r/:::8080                      r/:::0                         LISTEN

The tracing gadgets are also amazing to understand what is actually happening in pods over time. If you want to see which DNS requests and responses happen, you can just use the trace dns gadget:

$ kubectl gadget trace dns
K8S.NODE             K8S.NAMESPACE        K8S.POD              PID         TID         COMM       QR TYPE      QTYPE      NAME                RCODE      NUMA…
lima-k8s             default              sample-app…695-4cv89 98533       98533       ping       Q  OUTGOING  A          google.com.                    0
lima-k8s             default              sample-app…695-4cv89 98533       98533       ping       Q  OUTGOING  AAAA       google.com.                    0
lima-k8s             default              sample-app…695-4cv89 98533       98533       ping       R  HOST      A          google.com.         NoError    1
lima-k8s             default              sample-app…695-4cv89 98533       98533       ping       R  HOST      AAAA       google.com.         NoError    0

Seriously, it’s impossible to overstate how much information in an ongoing incident or a situational analysis can be discovered with Inspektor Gadget. If you operate Kubernetes clusters it should be in your go-to toolbox.

skopeo

Repository: github.com/containers/skopeo

This one has the most GitHub stars on the list so it is statistically the tool most people are familiar with, but it still made sense to include it on the list due to its sheer usefulness.

skopeo is strictly speaking not a tool for Kubernetes either – it’s for interacting with container images without the need for a fully blown container runtime running (which is extremely useful on systems that don’t run docker natively, like macOS or Windows). It can assist in both discovering image metadata or manipulating images in various ways.

The two frequent options in daily workflows are likely skopeo copy and skopeo inspect. Here’s an example of inspecting the metadata of an image in a remote registry:

$ skopeo inspect docker://quay.io/embik/sample-app:v0.1.0
{
    "Name": "quay.io/embik/sample-app",
    "Digest": "sha256:efbbf29b92bd8fca3e751c1070ba5bf0f2af31983bfc9b007c7bf26681c59b4c",
    "RepoTags": [
        "v0.1.0"
    ],
    "Created": "2023-04-07T11:38:28.791201794Z",
    "DockerVersion": "",
    "Labels": {
        "maintainer": "marvin@kubermatic.com"
    },
    "Architecture": "amd64",
    "Os": "linux",
    "Layers": [
        "sha256:91d30c5bc19582de1415b18f1ec5bcbf52a558b62cf6cc201c9669df9f748c22",
        "sha256:565a1b6d716dd3c4fdf123298b33e1b3e87525cff1bdb0da54c47f70cb427727"
    ],
    "LayersData": [
        {
            "MIMEType": "application/vnd.oci.image.layer.v1.tar+gzip",
            "Digest": "sha256:91d30c5bc19582de1415b18f1ec5bcbf52a558b62cf6cc201c9669df9f748c22",
            "Size": 2807803,
            "Annotations": null
        },
        {
            "MIMEType": "application/vnd.oci.image.layer.v1.tar+gzip",
            "Digest": "sha256:565a1b6d716dd3c4fdf123298b33e1b3e87525cff1bdb0da54c47f70cb427727",
            "Size": 3189012,
            "Annotations": null
        }
    ],
    "Env": [
        "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
    ]
}

fubectl

Repository: github.com/kubermatic/fubectl

fubectl is a collection of handy aliases for your shell so you don’t have to type out kubectl commands all the time. While this is a project hosted by my current employer, I’ve been using it since before joining Kubermatic.

fubectl is a bit hard to show off in a blog post – the repository README does a much better job at that. Besides the obvious aliases (k instead of kubectl, kall instead of kubectl get pods -A, etc) it does a great job at integrating fuzzy finding via fzf. It makes interacting with Kubernetes much more interactive.

It is much easier to get logs for a pod by running klog and then searching for the pod by typing fragments of its name, and then going through the second stage of selecting the right container within that pod. In a similar fashion, kcns and kcs help switching between contexts and namespaces without much friction.

Once the various aliases of fubectl are in your muscle memory, you can never got back to running kubectl config get-contexts and kubectl config use-context <context> instead of kcs.

kube-api.ninja

Website: kube-api.ninja
Repository: github.com/xrstf/kube-api.ninja

This completes the xrstf trifecta of Kubernetes tools you should know about. The difference to all other tools on this list is that this is not a command line tool but a website. It tracks Kubernetes API changes over time in an easy to read table view.

When was a specific resource in a specific API version added to Kubernetes? When was it migrated to another API version? What important API changes are in a specific Kubernetes version (e.g. what APIs might need to be updated in your manifests before upgrading to this Kubernetes version)? kube-api.ninja answers all those questions and many more.

For example the notable API changes for Kubernetes 1.29, showing that some resource types got removed with that version:

kube-api.ninja is also helpful if you are interested in the evolution of APIs. Did you know that HorizontalPodAutoscalers existed as a resource type before Deployments? These days the Kubernetes APIs have stabilized a bit, but I wish I had this around during the extensions to apps migration days.

kubeconform

Repository: github.com/yannh/kubeconform

The last (serious) entry on this list is kubeconform, which is extremely helpful in validating your Kubernetes manifests before applying them. It works great in tandem with helm by first rendering your Helm chart into YAML and then passing that to kubeconform to check for semantic correctness. This is how a simple CI pipeline will much improve your Helm chart’s development process:

helm template \
  --debug \
  name path/to/helm/chart | tee bundle.yaml
# run kubeconform on template output to validate Kubernetes resources.
# the external schema-location allows us to validate resources for
# common CRDs (e.g. cert-manager resources).
kubeconform \
  -schema-location default \
  -schema-location 'https://raw.githubusercontent.com/datreeio/CRDs-catalog/main/{{.Group}}/{{.ResourceKind}}_{{.ResourceAPIVersion}}.json' \
  -strict \
  -summary \
  bundle.yaml

This will help ensure that PRs changing your Helm chart still produce semantically valid Kubernetes resources, not just valid YAML.

kubeconfig-bikeshed

Repository: github.com/embik/kubeconfig-bikeshed

Okay, okay, okay. This one is shameless self-promotion, so I’ll keep it short. If you struggle with juggling access to many Kubernetes clusters and you feel like multiple contexts in your kubeconfig no longer cut it, kubeconfig-bikeshed (kbs) might be for you. I’ve started writing it to replace my various shell snippets that I was using to manage access to Kubernetes clusters.

How many of the listed tools did you know already? Hopefully you found some new things to try out in your next troubleshooting session or CI/CD pipeline design.

Bikeshedding Kubeconfig Management

Tue, 21 Nov 2023 00:00:00 +0000

When working on a platform like the Kubermatic Kubernetes Platform (KKP), plenty of kubeconfigs end up in your home directory. Some of them long-lived, some already outdated 30 minutes later.

Previously, I have been handling this with a couple of shell aliases and an occasional rm -rf ~/Downloads/kubeconfig-*. But I would not be a software engineer if I hadn’t been obsessed with optimizing my personal workflows. There is a good reason the profession likes to bikeshed some things to death. And we like to have, uhm, … special workflows.

Since I am no exception to all of that, I decided to write my own application to manage kubeconfigs. Enter kubeconfig-bikeshed – shorthand kbs. My way of keeping my ~~bikes~~ kubeconfigs organized and out of the rain (so they don’t rust – you want to guess what programming language it is written in?)

Let’s take a look at kbs in its current state, where I want it to go and why I want to work on it. Right now, the project is in its infancy, but I hope to make it actually useful in the future.

What is kbs?

The initial v0.1 version comes with admittely a very slim feature set. There are essentially three commands in kbs (ignoring help and shell completion related ones):

kbs import takes the path to a kubeconfig anywhere on your system (for example in ~/Downloads) and copies it to a central “store” that kbs is using. That store is in ~/.config/kbs by default, but respects XDG_CONFIG_HOME. While importing, the kubeconfig and context names get overridden with the server name. This is one of the things I kept doing when downloading kubeconfigs, and I often found the FQDN that is hosting the Kubernetes API to be the most “accurate” descriptor of a kubeconfig. If that is not suitable, the --name flag allows to set a name explicitly.
kbs list lists the available kubeconfigs from the “store”. It is not very useful on its own, but mostly for scripting purposes and in combination with the next command.
kbs use, which prints a shell snippet to export a kubeconfig from the store by name to the KUBECONFIG environment variable.

At the moment, the tool is restricted to import “simple” kubeconfigs, which means they can only have one cluster and one user entry. In the future I hope to improve on this.

This is the baseline for my tool, and thus the v0.1 release. In coming releases, I want to add a boatload of features:

flags to change behavior of commands (e.g. --short on kbs import to just use the first portion of the server FQDN as name).
let kbs use remember the last active kubeconfig and add kbs use - to “restore” the last selected kubeconfig.
add a command to fetch kubeconfigs from remote systems (e.g. for a user cluster in KKP).
add a command to “prune” outdated kubeconfigs. The semantics of that are still not fully settled in my head, but I can see it trying to establish a HTTPS connection to the Kubernetes API to decide if the server still exists.
a way to manually clean up kubeconfigs (without going to ~/.config/kbs manually).
perhaps a way to attach metadata to kubeconfigs when importing them. This could be useful to categorize and filter kubeconfigs, e.g. exclude any kubeconfig that has a vpn=internal flag from being pruned.

While writing this section, new ideas come up every second. There is a lot of potential in kbs that I hope to explore.

Installation

kbs can be installed via brew (if you have Homebrew) or cargo (if you are on Linux and have a working Rust toolchain), depending on what system you are on. For brew, installation is as simple as:

$ brew tap embik/tap
$ brew install kubeconfig-bikeshed

Getting the tap up and running was relatively simple, but writing the formula for a Rust binary was surprisingly difficult to get right. When you search for that online, you run into tons of outdated advice. Maybe this should be the topic for another blog post once I feel more comfortable with it.

With cargo, it is even simpler after pushing my first crate to crates.io:

$ cargo install kubeconfig-bikeshed

After installation, I highly recommend to set up shell integration. For example for zsh, add the following snippet to your .zshrc:

# load kubeconfig-bikeshed shell completion & magic
if command -v kbs &>/dev/null
then
    source <(kbs shell completion zsh)
    source <(kbs shell magic zsh)
fi

But wait … what is kbs shell magic zsh?

Shell magic

One of the more interesting bits in kbs is the optional shell “magic”. It allows the user to run kbs to select a kubeconfig and set it for further use with e.g. kubectl. For zsh, it is loaded in the snippet above. For bash, it can be loaded via kbs shell magic bash.

As with all good magic, the actual trick is very simple. This defines a function called kbs in bash:

alias _inline_fzf="fzf --height=50% --reverse -0 --inline-info --border"
alias _kbs_bin="$(type -p kbs)"

function kbs() {
    if [ $# -eq 0 ]; then
        # if no parameters are passed, we want to run fzf on available kubeconfigs and set the selected one as active kubeconfig
        eval "$(_kbs_bin use $(_kbs_bin ls | _inline_fzf))"
    else
        # if parameters are passed, we just call the kbs binary directly
        ${_kbs_bin} $@
    fi
}

This acts like a shim between the user and the actual kbs binary. This is necessary for actually setting the KUBECONFIG environment variable, which feels like a core functionality of any good kubeconfig manager. You cannot set environment variables from within an application, so this function runs kbs ls to list available kubeconfigs, pipes them through fzf so the user can interactively search and select one, and then exports the full path as KUBECONFIG (kbs use prints something like export KUBECONFIG=path/to/kubeconfig that is going through eval and applied to the current shell).

This only happens when kbs (the shell function now) is called without arguments. if it is called with any arguments, they are passed through to the kbs binary.

The neat trick I learned while working on this is the use of proper shell built-ins. which is the much more popular command to determine where and if a command exists. However, since this is overriding the kbs command with a function, it needs to know where the kbs binary is. type -p does exactly that by simply looking for binaries, and not checking the shell itself.

Note that while type is a unix standard built-in, it works differently across shells. The zsh magic for example uses whence instead of type, because type prints not only the path:

$ type -p kbs
kbs is /opt/homebrew/bin/kbs

Since shell magic is per shell, I can get away with not using type where it doesn’t suit what I need.

Why write your own

Because it is fun. No, seriously. Working in an established ecosystem like Cloud Native can be a challenge sometimes, even though it is a relatively young space. Do not see this as a complaint – Collaboration, discussions, guard rails and architecture considerations are vital to good software. And I enjoy being involved in them.

But writing my own tools can be refreshing. They can do exactly what I want them to do, scratching that super specific itch I have. If people feel the same way I do about those itches, they are very welcome to use kbs. If they feel it doesn’t scratch their itch, they can use another tool or write their own. Or fork kbs. Finding a personal workflow that works has many ways.

Our industry is split on “Not Invented Here” (NIH) syndrome. Some organizations extensively suffer from it, while others are avoiding it – with mostly good reasons – like the plague. I believe collaboration between people with diverse backgrounds and corporate culture always produces the best outcome and should be the default choice over building it on your own. But I think if I can get away with NIH anywhere, it is tools used in personal workflows.

Learning by doing

The way I learn – and I assume most do – is by doing. Rust is one of those programming languages. The ones I have been aware of for years, but never really got into over all those years. The first time I tried to get into Rust was with nightshift, a tool to reduce blue light emissions when using sway as your window manager on Linux. That repository was created six years ago! And it never really got anywhere because I struggled with the language a lot. Since then I have used Rust for a small tool called ing-csv-importer which frankly speaking could have been a shell script.

But I still wanted to learn Rust! And I was bothered by my kubeconfig (mis-)management. The obvious solution to both problems was to combine them. Since I had a rough idea of what the tool should be capable of, the Rust learning process was much more guided. I knew what to look for. The code itself is probably terrible to experienced Rustaceans, but it works and I have already learned a lot about Rust from it. I hope to refine it in the future.

kbs is a “low impact” learning project. I had the idea back in May of 2023. Set up the repository, pulled in some dependencies and called it a day. But since there was no pressure to come back to it, I was able to take my time and get back into it once I was motivated. No one cared if I delivered “on time”, and so I was able to get back into it half a year later.

And after a couple of days of ramp up, I have to say I am delighted with Rust. The language has a certain elegance to it that I cannot explain. But I enjoy concepts like match that make for really nice code. Consider for example the snippet below, which determines the name variable used for a kubeconfig and its context depending on whether the --name flag is set:

let name: String = match matches.get_one::<String>("name") {
    Some(str) => str.clone(),
    None => {
        let hostname = kubeconfig::get_hostname(&kubeconfig)?;
        let url = Url::parse(hostname.as_str())?;
        let host = url
            .host_str()
            .ok_or("failed to parse host from server URL")?;
        host.to_string()
    }
};

This feels so much less clunky to me than an if block in Go, for example. Is that “factually” correct? Maybe not. But Rust has a lot of concepts that I never knew I missed from other languages and I expect to do more work with it in the future.

Closing thoughts

Does kbs do something special? Absolutely not. If people are looking for a mature kubeconfig management solution, they are – at the moment, and maybe forever – better off with konf-go. But the work on this is a lot of fun and gives a certain feeling of self-empowerement once I had “finished” the v0.1.0 release end-to-end (from first commit to available Homebrew package).

Now I am excited to integrate kbs into my daily workflow and improve it where necessary. I plan to maintain it as long as I am working with masses of kubeconfigs, which I sincerely hope won’t change in the near future.

Anyways, I wish some people find use in kbs or the inspiration to start their own learning projects (or not; enjoy your free time with whatever makes you happy). Oh, and regarding that list of programming languages I want to get into – I’m coming for you next, Elixir and/or Gleam (once I get kbs where I want it to be, but … details, am I right?)

Running a Go Debugger in Kubernetes

Sun, 08 Oct 2023 00:00:00 +0000

(Note: This blog post is a written version of my talk Ephemeral Containers in Action - Running a Go Debugger in Kubernetes. Slides and recordings are available in the linked repository.)

Identifying bugs in our code can be hard; really hard. In complex microservice environments, it can be even harder, since reproducing the exact request flow which triggered a bug can be quite the challenge. Thankfully the modern observability stack can be quite helpful in the matter.

But sometimes extended observability might not be available or it’s still not enough to identify the problematic code path. In those cases I like to break out a proper debugger, set specific breakpoints and investigate the program’s state at those breakpoints.

But how do we run a debugger within Kubernetes, where our application actually lives? We do not want to run our application with active debugging all the time! Not a problem (anymore)! We can utilize ephemeral containers. It will allow us to launch a debugger into an existing Pod on demand.

Before going into the details, let’s review what debugging actually is and what considerations we need to be aware of when debugging Go applications.

Basics of debugging (Go)

Debugging is a long-standing staple of software development. If it is not part of our development toolbox yet, it might be worth looking into it. print based debugging sometimes has lower friction, but knowing how to use a debugger is a powerful skill.

For Go, the go-to debugger is, without a doubt, Delve. Delve is a debugger specific to Go and has been around for years. While using GBD is possible, the official Go documentation recommends using Delve when possible. For some basics of debugging Go applications with Delve, there is an article available from golang.cafe that serves as a good introduction.

The gist however is that debuggers allow to inspect application state during code execution. We can set breakpoints and look at variable contents when that breakpoint is hit. In Go, specifically, it is also possible to inspect Goroutines.

This means debuggers are incredibly powerful in identifying faults in code since the whole state can be inspected at any given time. Once set up, effective debugger usage beats print-style debugging by a mile or two.

Let’s take a look at the requirements to run a debugger.

Linux capabilities

While interacting with the operating system, we often interact with kernel APIs. For this article we will take a quick look at the Linux kernel in specific, but similar concepts probably exist in the operating system of our choice. They do not overlap 100%, but they are probably close enough.

In Linux, there is a concept called capabilities. Capabilities are assigned to a process and allow to interact with certain functionality of the kernel. This enables more granular permissions than running a process as a super user.

For debuggers, a specific capability is usually required - CAP_SYS_PTRACE - because it unlocks calling the ptrace system call. Debuggers need this to attach to running processes and to interact with their memory and registers. It essentially allows us to “take control” of an already running process.

That is a scary-looking system call and there is good reason this is locked by a capability. But we need it! As a Linux user, we might not see this level of detail (and maybe run the debugger via sudo) but since we are thinking of running a debugger in Kubernetes, we need to be aware of this. A container in Kubernetes does not have this capability by default. Let’s keep this in mind, it will come up later again.

Go compile flags

Debuggers usually require a certain level of information encoded in the binary that they try to debug. This means that the “production” build of our application binary might not be suitable for debugging.

To determine this, we can look at how our binaries are built, e.g. in a Makefile or Dockerfile. If the following flags are present in go build’s -ldflags we definitely need a separate debug build:

-s: Omit the symbol table and debug information.
-w: Omit the DWARF symbol table.

Both flags allow us to optimize the binary size at the expense of debug information, which is usually what we want for production builds. If this is the case, a separate image needs to be provided that does not have those flags set. Otherwise, the application cannot be debugged. This also means that we will need to launch a separate application Pod with the debug-enabled application build. More on that later.

In addition, some flags can be helpful for debugging. Since we rely on line numbers to set breakpoints, we need to make sure that our code is not optimized by the compiler. This can happen if the compiler recognizes code patterns that can be simplified. Some parts of the code might therefore be changed and cause issues with debugging, although optimized code is generally considered okay (with some pitfalls).

Code optimization can be disabled if required by passing additional flags:

$ go build -gcflags="all=-N -l"

This will pass two flags to the Go compiler for all invocations. The flags being

-N: Disable optimizations.
-l: Disable inlining.

We should be careful with these flags though, since the compiler usually has some pretty good ideas about optimizing code. Let’s only add them if debugging gives us unexpected results or errors when trying to set breakpoints.

Remote debugging

Now we have established debugging as a concept - But so far, we have been talking about attaching ourselves to processes on the system. Which obviously means that we need to be on the same (virtual) machine. But if the application is running within Kubernetes, it would be nice to be able to debug from the comfort of our laptop.

Thankfully, the Debug Adapter Protocol exists to help us out. Developed for Visual Studio Code to have a standardized protocol to interact with so-called debug adapters (intermediate layers that translate between a client and the actual debugger), it has seen solid adoption and will help us out here (even if we do not use VS Code).

While DAP was meant to enable debug adapters as a layer in between, its success has triggered adoption in debuggers directly, overshooting the stated project goal. Delve has a native DAP implementation and can act as a DAP server. It also has its own protocol implementation that stems from before DAP adoption and can serve both protocols as a server. We will focus on DAP for this blog post since it is a (somewhat) new and emerging standard. For adaption into other language stacks, the Delve protocol is not interesting at all.

The interesting bit about DAP (and Delve’s own protocol) is that it is networked. This means we can suddenly traverse network boundaries when starting a debugging session! We can launch a debugging session on the system running the application (i.e. in Kubernetes) and then connect to it via a remote client (i.e. our local system).

We will use that to our advantage, since we prefer integrating debug sessions with our usual development tools. But to do so, we need an active debugger to connect to. Next, we will take a look at how to get the debugger (with a DAP server onboard) into Kubernetes in the first place.

A primer on ephemeral containers

Ephemeral containers are a feature in Kubernetes that became generally available in Kubernetes 1.25. Before ephemeral containers, there were two types of containers in a Pod specification: InitContainers and (normal) containers. InitContainers run before the “normal” containers at Pod startup to execute some initialisation logic (note this is also changing with sidecar patterns available in InitContainers). But both types of containers needed to be defined in the Pod specification when creating the Pod. Pods were more or less immutable, we could not change them once they were created.

This changes with ephemeral containers! Ephemeral containers are part of the Pod specification but are implemented as a subresource. And most importantly the list of ephemeral containers can be amended while the Pod is running. In fact, we can only create ephemeral containers while the Pod is running since subresources cannot be created at the same time as the actual resource.

Command line tooling

Since we established that we will need some special tooling to interact with the ephemeralContainers subresource on a Pod, let’s look at our options.

The obvious candidate is kubectl. Thankfully, it got us covered! kubectl debug is available as a command in the kubectl (kube-control? kube-cuttle? kube-c-t-l?) toolbox. It allows us to launch ephemeral containers into specific Pods.

$ kubectl debug <pod> -it --image=busybox --target=<container>

With the example above, an ephemeral container using the busybox image (--image) is attached to the specified Pod <pod>. It also connects the ephemeral container to a specific <container> (--target), which means that we break down the isolation between containers so the ephemeral container can access the process namespace of the target one. We also tell kubectl debug to drop us into an interactive shell in the ephemeral container (-i and -t).

kubectl debug is a very powerful toolbox for debugging in Kubernetes! If we choose to provide a special build of our application image (as discussed earlier), we cannot debug a “live” Pod. But we can use --copy-to and --set-image to

create a new Pod from the “template” of a currently running Pod and
override the image used for the application container to a debug-enabled image

which could be used to run a variant of this debug workflow against a Pod copy.

Debugging profiles

As established earlier, a debugger needs CAP_SYS_PTRACE (essentially by definition). By default a container does not have this capability, and for good reason!

But kubectl debug comes with a flag called --profile that helps with setting the right security configuration on the ephemeral container. This is a fairly recent addition, so not all use cases might be covered and it’s not always easy to figure out what a profile exactly provides, but for us, it is sufficient.

What we need is --profile=general, since it provides CAP_SYS_PTRACE and nothing more. Easy peasy.

As a minor side note: When I started working on this topic not everything was as polished as it is today. Because profiles were not yet available in kubectl debug, I had to develop my own little kubectl plugin called kubectl-ephemeral. It allows usage of the full API for the EphemeralContainer type to create ephemeral containers, which might prove useful if no profile provides the necessary configuration. Usually, we want kubectl debug though.

Delve as ephemeral container

Now we have everything that we need in building blocks. Let’s put it together! In this case, I am debugging a sample application that is a simple Go web server. It is already running in Kubernetes. We now want to create an ephemeral container in one of the Pods for that application. The ephemeral container should start Delve, attach to the application, and open a DAP server port.

$ kubectl debug \
    --profile=general \
    --target=sample-app \
    --image=quay.io/embik/dlv:v1.20.1 \
    sample-app-[...] \
    -- dlv --listen=127.0.0.1:2345 --headless=true attach 1

There we go. We attach to process id 1 because that is the process which started in the target container. Since we are connected to the PID namespace of the target container, we can just target the initial process - This only works if our target container does not have another entrypoint than the application binary.

One last thing though: We opened the DAP server on 127.0.0.1:2345. That’s not accessible from outside the Pod! How are we supposed to connect our client to that? And how do we keep this port secured as DAP knows no authentication?

Thankfully, kubectl port-forward helps us close that last gap. It does not need a port definition in our PodSpec, it works ad-hoc. Which means we can simply run

$ kubectl port-forward pod/sample-app-[...] 2345

which opens port 2345 on our local system and forwards it to the port within our Pod that has Delve running. This is our setup visualized:

To validate that our setup is running, try connecting to the local port with dlv connect:

$ dlv connect localhost:2345

If everything is working, this will connect us to the remote instance of Delve running in Kubernetes. From here, we can use the interactive shell of dlv to start debugging!

But we started out with the promise of integration into our workflows - And while the command line might be part of our usual workflow in developing software, we can do better than that.

Remote debugging with VS Code

Since we have a working connection to the Delve headless server instance in our ephemeral container now, we can use a DAP client of our choice to connect to it. The most popular DAP client - by a mile or two, probably - is Visual Studio Code (VS Code). It also happens to be one of the more popular editors out there.

To configure VS Code to talk to Delve, we can drop a launch.json configuration file into the .vscode directory of our project (if the folder does not exist, we need to create it). VS Code will read it as a project-specific debugging configuration.

This - or something similar - is the file for my example project:

{
  "version": "0.2.0",
  "configurations": [
    {
      "name": "Remote Attach",
      "type": "go",
      "request": "attach",
      "debugAdapter": "dlv-dap",
      "mode": "remote",
      "substitutePath": [
        { "from": "${workspaceFolder}", "to": "/build" }
      ],
      "port": 2345,
      "host": "127.0.0.1"
	}
  ]
}

All of this is important to configure the connection to a running debugger, but we care mostly about type being set to "go" (since we are debugging Go code), debugAdapter set to "dlv-dap" (so VS Code knows it will talk to Delve via DAP), port and host referencing the local port-forwarding we have opened, and - most importantly - substitutePath set correctly.

Because Go embeds the full path in its debugging information (unless we trim paths), we need to tell VS Code about the original path that the binary was built in. In our case, this might have been /build, the working directory set in our Dockerfile. Therefore, we instruct VS Code to rewrite path information when interacting with the debugger from our current workspace folder to /build.

Once this is done, VS Code will offer the new debugging mode called Remote Attach from its debugging screen via the green play button. For a detailed overview of debugging capabilities in VS Code please check out the official documentation.

Closing thoughts

Phew, that was a lot. Hopefully this overview helps with designing a personal debugging workflow in Kubernetes. This was not meant as a step by step tutorial, but more of a showcasing for what ephemeral containers can do. Maybe it can serve as basis for additional, similar workflows.

I think it is clear from the full text that this is not an outright recommendation. A lot of preconditions need to be taken into consideration before a debugging session in Kubernetes will work. If they can be fulfilled, Delve can really help with figuring out deeply rooted bugs in an application.

Thank you to my personal editor for reviewing and improving this post!

Private DNS with CoreDNS, Podman and Ansible

Wed, 12 Aug 2020 00:00:00 +0000

Running a private DNS resolver is useful in quite a few situations, for example in a home lab or on an internal company network; basically everywhere where you want to give names to private systems. CoreDNS is a simple DNS server that can be used in such a situation - if the name rings a bell with you, it’s because CoreDNS is the standard in-cluster DNS solution for Kubernetes for some time now. In practice, this means that large microservice architectures rely on it to resolve internal and external hostnames.

I decided to go with CoreDNS for my setup because of that fact (it’s a tool proved to be battle-tested in Kubernetes environments I worked with) and the simplicity of configuration - it needs a single configuration file, called Corefile, that defines the DNS resolver’s behavior. Moreover, CoreDNS is based on a plugin architecture, which potentially allows to extend it with custom functionality.

The small network that required a private DNS resolver in my case was however too small to run Kubernetes, so it was necessary to find a different way of running it. Since it is a Go program, downloading and running a binary would probably work - But I wanted to run it in a container for better isolation. CentOS 8 added support for a new container runtime largely spearheaded by Red Hat and Fedora called podman. podman is interesting because it gets rid of the omniscient (Docker) daemon, allowing for running containers in a more stand-alone manner. It also supports running containers as non-root users, albeit we won’t use this feature for now.

To deploy CoreDNS and maintain its configuration, I chose Ansible. Again, mostly because I worked with it before and because all I need to get started is a SSH key on the target system. The following steps will assume that you already have a running CentOS 8 system (likely a VM) running somewhere in your private network and that you have access to it. I won’t go into the basics of Ansible and I recommend you to search for tutorials / getting started guides on it before tagging along.

At the end of this, we will have a private DNS resolver that is capable of resolving public names (with some filtering in place) and resolving one or more private domains we can adjust to our needs.

Important: You should make sure that port 53 (DNS) of your system is not available on the public internet, securing access with firewall or security group rules.

Required packages

Depending on the disk/CD image you used to install CentOS 8, not all required tools will be available right away. Besides the container runtime we want to use (podman), let’s install two more packages: udica and bind-utils. bind-utils is just useful as it contains the dig command, something we can use to query our DNS resolver later on.

udica is more interesting: It hasn’t been mentioned yet, but I would like to integrate with existing security mechanisms on CentOS as much as possible. This means that we will keep SELinux in enforcing mode. udica will enable us to generate SELinux modules that will allow running CoreDNS in a podman container while keeping SELinux up.

All things considered, translated to a task in Ansible this would mean adding this to your coredns role:

- name: install base packages
  dnf:
    name:
      - podman
      - udica
      - bind-utils
    state: present

Later on, I will not present tasks for trivial operations in Ansible - I strongly recommend to adapt my findings to your needs.

Setting up a Corefile

Before we get too much into the details of deploying CoreDNS in podman, we should focus on defining our requirements for the DNS resolver and templating a Corefile around that.

We need to ensure that on the target system, there is a directory we can put our configuration into. My Ansible role creates /etc/coredns for that purpose:

- name: add directory for CoreDNS configuration
  file:
    path: /etc/coredns
    state: directory
    mode: '0755'

The bad news first: CoreDNS does not come with any kind of built in ad block solution. A lot of people are running services like Pi-hole on their private network and our solution won’t get as sophisticated as that. The good news is, if you want to block nepharious websites in your DNS resolver, you can still do that. Earlier we defined the requirement for our resolver to resolve all kinds of names, so let’s look into public names first.

Resolve public DNS, with a blocklist

To implement some kind of blocklist for bad DNS names, we can use the hosts plugin of CoreDNS. It reads the content of a /etc/hosts-style file and serves requests based on that. I have included a list from StevenBlack/hosts as a blocklist that will resolve the hosts on that list to 0.0.0.0, thus failing resolution. Download one of the lists (depending on what you would like to block) into your Ansible role and copy them to your target system.

Let’s set up the part in our Corefile that will allow our resolver to resolve public names with the restrictions of our blocklist in place. To do so, we can ask CoreDNS to look into our blocklist and fallthrough to the next lookup method if a name is not on our naughty list. We will rely on /etc/resolv.conf for upstream DNS, but you can adjust the forward plugin configuration we are including to suit your network situation.

. {
    hosts /etc/coredns/blocklist.hosts {
        fallthrough
    }
    forward . /etc/resolv.conf
    log
}

This short configuration snippet instructs CoreDNS in the following ways:

It defines a server block for ., which is the root zone, meaning that any requests should go through this configuration.
It uses the hosts plugin to try resolving a name from /etc/coredns/blocklist.hosts (our blocklist). If a name is not on that list, it will go to the next method of resolving the name.
It forwards any further requests to the DNS servers defined in /etc/resolv.conf.
It enables logging of all incoming requests via the log plugin.

Resolve private DNS

Now that we cover public DNS, let’s look into setting up one or more private DNS domains on this resolver. I am using the hosts plugin again to resolve a templated list of DNS names; you can adjust that to your needs and situation.

First, let’s generate another /etc/hosts-style file for our private zone. The minimal viable solution for this would be this Jinja2 template:

{% for entry in coredns_entries %}
{{ entry.ip }}  {{ entry.host }}
{% endfor %}

This requires passing the host variable coredns_entries to our target host in Ansible - A simple coredns_entries configuration could look like this (let’s throw in a variable for our private DNS zone for good measure):

coredns_internal_domain: privatezone.internal
coredns_entries:
    - ip: 10.0.0.1
      host: gateway.privatezone.internal
    - ip: 10.0.0.2
      host: gitlab.privatezone.internal

You can get smarter with this (e.g. using the coredns_internal_domain variable to eliminate the requirement for adding a fqdn each time), but this is the bare minimum that will get us started. If it’s possible, consider pulling this kind of information out of your inventory. Ansible should template this to the target system, e.g. into /etc/coredns/internal.hosts. Afterwards, adjust the Corefile (that should become a template at this point) to load our internal hosts file:

{{ coredns_internal_domain }} {
    hosts /etc/coredns/internal.hosts
    log
}

. {
    hosts /etc/coredns/blocklist.hosts {
        fallthrough
    }
    forward . /etc/resolv.conf
    log
}

There we go! It’s important to load your private zones before the big catch-all . block, so CoreDNS won’t try to send your private hostnames to public DNS resolvers. Now we have a minimal Corefile on our target system. Let’s look into running CoreDNS as a podman container next.

Running CoreDNS in Podman

The next thing we want to do is figuring out the correct podman command we want to run to start our CoreDNS container. While there is an equivalent to docker-compose available, let’s stay with the (mostly docker-compatible) podman cli. We will later wrap this call in systemd to leverage it as supervisor for our container.

The full podman run call that I came up with is this one:

/usr/bin/podman run --name coredns --read-only -p 10.0.0.1:53:53/tcp -p 10.0.0.1:53:53/udp -v /etc/coredns:/etc/coredns:ro --cap-drop ALL --cap-add NET_BIND_SERVICE coredns/coredns:1.7.0 -conf /etc/coredns/Corefile

Let’s look at the parameters passed here to understand what is going on:

--name coredns will set the container name to “coredns”. This means that running another instance of this command will conflict with any running or stopped one.
--read-only sets the container’s root filesystem to read-only mode. The root filesystem of a container is transient, so applications should not store any important data there anyway. CoreDNS doesn’t need write access to its root filesystem, so let’s disable this to improve isolation.
-p 10.0.0.1:53:53/tcp and -p 10.0.0.1:53:53/udp bind to port 53 on both TCP and UDP for the private IP the system; replace this with your own system’s private IP (or rather, use Jinja templating for it; we will follow up on that). Binding to the IP might be necessary because systemd-resolved is possibly taking the port on localhost, making it impossible to bind to.
-v /etc/coredns:/etc/coredns:ro mounts our configuration folder into the container. Again, this is read-only, to ensure integrity of our configuration files. CoreDNS has no need to write back into this directory.
--cap-drop ALL and --cap-add NET_BIND_SERVICE will drop as much Linux capabilities as possible while maintaining NET_BIND_SERVICE, which is required to bind to the port. Again, this improves isolation and is considered good practice to reduce attack surfaces.

All in all, this tries to limit the application running in the container in what it is capable of doing.

Now, if you try to run this command, it will hopefully fail - CoreDNS will be unable to bind to port 53 (if it does not fail or complain in the logs, you probably disabled SELinux previously - shame on you!). SELinux does not allow it yet. Let’s fix that.

Generating a SELinux policy

Remember when we installed udica in the beginning? Now its time to shine has come! udica is a small but awesome tool that allows generating SELinux policies from container definitions. It takes away a lot of pain with containers and active SELinux. With that being said, I urge you to read up on SELinux and how it works, giving you the opportunity to further slim down policies if possible.

Let’s pull the definition of our “coredns” container, put it into a JSON file and pass it to udica:

$ podman inspect coredns > container.json
$ udica -j container.json coredns

Policy coredns created!

Please load these modules using: 
# semodule -i coredns.cil /usr/share/udica/templates/{base_container.cil,net_container.cil}

Restart the container with: "--security-opt label=type:coredns.process" parameter

This generates a coredns.cil file in the current directory. For me, it looked like this:

(block coredns
    (blockinherit container)
    (blockinherit restricted_net_container)
    (allow process process ( capability ( net_bind_service )))

    (allow process dns_port_t ( tcp_socket (  name_bind )))
    (allow process dns_port_t ( udp_socket (  name_bind )))
    (allow process etc_t ( dir ( getattr search open read lock ioctl )))
    (allow process etc_t ( file ( getattr read ioctl lock open  )))
    (allow process etc_t ( sock_file ( getattr read open  )))
)

It’s relatively small and quite declarative, so the main takeaways from this policy should be:

it will allow using the net_bind_service capability.
it will allow the process to bind to ports labeled dns_port_t for UDP and TCP (which is only port 53).
it will allow the process to read from directories labeled etc_t (which is /etc, where our CoreDNS configuration directory is).

You could further lock down this policy with your own SELinux labels on /etc/coredns, but for now, this should suffice. It will get us past SELinux as a gatekeeper and improve our security posture because we didn’t disable SELinux (yay). To enable this policy, run the command in the udica output.

Interlude: SELinux policies and Ansible

As all configuration steps, this should be included in our Ansible role to make the installation reproducible on fresh systems. Unfortunately, Ansible doesn’t seem to offer a module that will allow applying this, which means we need to fall back to using the shell module. This is how my role handles applying this:

# tasks/main.yml
[...]
- name: create /etc/udica for custom SELinux policies
  file:
    path: /etc/udica
    state: directory
    mode: '0755'

- name: create coredns SELinux policy file
  copy:
    src: files/coredns.cil
    dest: /etc/udica/coredns.cil
  notify:
    - load coredns SELinux module
[...]

# handlers/main.yml
- name: load coredns SELinux module
  shell: 'semodule -i /etc/udica/coredns.cil /usr/share/udica/templates/{base_container.cil,net_container.cil}'

This creates a handler running semodule which is being called upon changes to the SELinux policy file.

systemd service

We’re almost done! At this point, our CoreDNS container should run successfully and you should be able to resolve hosts with dig @10.0.0.1 google.com (replace 10.0.0.1 with your own private IP) as long as you’re running the podman command from earlier.

But as mentioned, podman doesn’t come with a supervising daemon - Which means that no one will restart the container once it’s dead (for whatever reason); Nor will it be restarted upon reboot.

To fix this, let’s use something on our system that is already working as a supervisor to services: systemd! Fortunately, integration with systemd works quite well under podman, as long as you follow a template recommended by Red Hat.

This is the Jinja template that I use to generate /etc/systemd/system/coredns.service, which is based on the service unit template linked above. The main difference to a standard systemd service unit is that podman will write to PID files that systemd will read to be informed about the container status.

[Unit]
Description=CoreDNS private DNS in a container

[Service]
Restart=on-failure
ExecStartPre=/usr/bin/rm -f /%t/%n-pid /%t/%n-cid
ExecStartPre=-/usr/bin/podman rm -f coredns
ExecStart=/usr/bin/podman run --name coredns --read-only --security-opt label=type:coredns.process -p {{ ansible_eth1.ipv4.address }}:53:53/tcp -p {{ ansible_eth1.ipv4.address }}:53:53/udp -v /etc/coredns:/etc/coredns:ro --cap-drop ALL --cap-add NET_BIND_SERVICE --conmon-pidfile /%t/%n-pid --cidfile /%t/%n-cid -d coredns/coredns:1.7.0 -conf /etc/coredns/Corefile
ExecStop=/usr/bin/podman stop coredns
KillMode=none
Type=forking
PIDFile=/%t/%n-pid

[Install]
WantedBy=multi-user.target

Beyond what the template requires, note the additions in comparison to the previous iteration:

--security-opt label=type:coredns.process will allow the container to use our previously created SELinux policy.
-p {{ ansible_eth1.ipv4.address }}:53:53/tcp and -p {{ ansible_eth1.ipv4.address }}:53:53/udp will template in the private IP of my system, which is associated with eth1. Adjust to your situation.

This will deliver a fully functional systemd unit called coredns.service that will utilise podman and a targeted SELinux policy to run CoreDNS on your local network. Do not forget to write your Ansible role in a way that changes to the SELinux policy or your configuration files will trigger a restart.

Wrapping up

That’s it - We now have CoreDNS running, safely wrapped into podman, SELinux and systemd. Using Ansible is a nice bonus, allowing us to deploy our private DNS resolver to additional systems with little overhead. Not all steps have been presented as Ansible snippets, but I hope it will prove useful to those writing better roles than I do.

If everything worked well, we will now see a container running in podman:

$ podman ps
CONTAINER ID  IMAGE                            COMMAND               CREATED       STATUS           PORTS                                         NAMES
16f66c7140ed  docker.io/coredns/coredns:1.7.0  -conf /etc/coredn...  2 hours ago  Up 2 hours ago  10.0.0.1:53->53/tcp, 10.0.0.1:53->53/udp  coredns

To access the logs (which includes all requests the server received and answered), we can run podman logs coredns.

This setup obviously has further room for improvement. Next steps to explore could be (but are not limited to):

Pull the hostnames of our private dns zone out of the Ansible inventory during templating.
Use the cache plugin in CoreDNS to improve performance and reduce the need to reach out to upstream DNS resolvers.
Investigate other CoreDNS plugins for host discovery, exploring the possibilities of Corefile configuration.
Expose metrics via the prometheus plugin. This would require running a Prometheus server somewhere on your internal network, but improving visibility on your services is worthwhile.
Look into implementing a health check via podman to catch any service malfunction that might not result in the container crashing.