Kubernetes Homelab #1: Raspberry Pi Setup

I like to tinker with Kubernetes in my free time and I’ve always wanted to host some services for myself. Vaultwarden is one of those services since I’d like my passwords to be stored away where I can see them. Up to now this was running on an old Intel NUC that I stashed away in my apartment.

Years ago I had been using Raspberry Pis (generation 1 or 2, maybe?) for hosting some fun projects (e.g. a TV antenna receiver attached to a Raspberry Pi to watch TV – when that was still a thing).

This post is part of a blog post series describing my Kubernetes Homelab.

• Part 1: Raspberry Pi Setup (this post)
• Part 2: Cluster Installation with KubeOne
• TBD: Hyperconverged Storage with Longhorn
• TBD: Exposing Services through Tailscale

Now I wanted to try again because cloud is expensive, Kubernetes can be quite fun and I hadn’t tinkered with hardware in a good while. So my goal was to get a setup that

1. had three nodes to make sure Kubernetes made sense. A single node isn’t really a cluster.
2. provided its own storage. A NAS is expensive and a single point of failure, so “hyperconverged” (is that word still in use at all? it was the big hype in on-premise hardware ten years ago) was the way to go.
3. didn’t boot from an SD card. Maybe this is fine now but I had many corrupted SD cards I needed to reformat back in the days. I wanted my machines to boot from a disk.
4. used power over ethernet (PoE) to reduce the cables needed for powering the setup.
5. allowed to expose services to me as its only user with valid TLS certificates and hostnames.

The blog post series that this post kicks off will map out my journey with getting my homelab up and running and document any steps and problems along the way.

Note: All commands in this post are executed as root. You can either prepend them with sudo or open a root shell once with sudo -i.

Hardware #

The core of the setup is comprised of three Raspberry Pi 5 B with a 2.44 GHz ARM Cortex-A76 Quad-Core-CPU and 8GB of RAM. These machines are barely comparable to the original Raspberry Pi. They are quite the workhorses!

To power the Raspberry Pis over ethernet via PoE I bought a TP Link TL-SG1005P (a 5-port gigabit desktop PoE+ switch).

I ended up using the following HATs for the three Raspberry Pi 5 B, bought from Amazon:

• Waveshare PCIe to M.2 HAT+
• Waveshare PoE HAT (f)

Waveshare is a Chinese manufacturer of electronic components for microcontrollers and embedded boards. The choices for Raspberry Pi 5 PoE HATs is quite limited at the moment. An official PoE HAT had been announced, but updates have been rare and search results mostly end on a post half a year old. Jeff Geerling reviewed this PoE HAT (see his blog and the GitHub issue) and overall it looked promising.

M.2 SSD #

I went with Waveshare’s PCIe to M.2 HAT+ to make sure the two HATs would work with each other. However it seems to be hit or miss whether an M.2 SSD is going to be supported by the HAT or not. I started the setup with a Transcend TS256GMTS430S (a 256GB M.2 SSD), but the disk would not get recognized, even after following all troubleshooting steps. After some back and forth with Waveshare support I got the recommendation to use a Western Digital WD BLACK SN770M, which worked like a charm.

As is often the case with these kind of manufacturers, documentation is sparse. It was more or less impossible to figure out why the Transcend SSD wasn’t recognized – some sources were talking about the Raspberry Pi PCIe not working with a specific on-board controller on M.2 SSDs, but as far as my Google research suggested this SSD didn’t seem to be using that controller. In the end things worked out by exchanging hardware, thus it doesn’t seem a good idea to combine this particular HAT with Transcend M.2 SSDs.

Assembly #

Overall assembly worked fine but required some careful strength applied at the right times. I assembled the system with the PoE HAT being attached to the Pi first as the lower layer and then added the PCIe to M.2 HAT on top of it.

The cable provided with the PCIe to M.2 HAT to connect the Pi’s PCIe slot to the HAT was just long enough to work in this stacked setup. Connecting it properly on both ends was a bit fiddly (especially on the Pi side – the HAT side has a very solid mechanism to hold the cable in place), tweezers came in very handy.

System setup #

The best choice of OS on the Raspberry Pi currently seems to be Raspberry Pi OS. It’s based on Debian, which is great because KubeOne (disclaimer: I’m a contributor to it and it’s developed by my employer, Kubermatic) has (limited) support for creating Kubernetes clusters on it.

Before booting from the M.2 SSD a quick detour via a bootable USB stick was needed. I used the Raspberry Pi Imager to write the latest Raspbbery Pi OS “lite” (no desktop environment included) to a spare 32GB USB stick. One of the important customizations here is enabling SSH and adding a SSH public key.

Plugging in the USB stick and powering on the Raspberry Pi by connecting the ethernet cable started Raspberry Pi OS from the USB stick. I used this transient setup to check on the M.2 SSD disk and prepare it to become the boot device. To find its IP address I signed into my home router’s web UI, found the newly connected device and made sure to configure the device to always receive the same IP address from the router’s DHCP server.

After connecting via SSH I had to make sure the M.2 SSD was actually working. That required configuration so that the Raspberry Pi would start with NVMe support which can be added by editing /boot/firmware/config.txt and adding the following line:

dtparam=nvme

On one of the Pis also refused to boot from NVMe (in one of the later steps) before updating its firmware. To do so I ran the following commands (still booting from the USB stick):

$ apt update
$ apt upgrade
$ rpi-update

For this and the dtparam addition to take effect a reboot is required. Afterwards, the SSD showed up as block device:

$ lsblk
NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
sda           8:0    1  28.7G  0 disk
|-sda1        8:1    1   512M  0 part /boot/firmware
`-sda2        8:2    1  28.2G  0 part /
nvme0n1     259:0    0 465.8G  0 disk

At this point the SSD was showing up so I was able to prepare it to become the system’s boot disk. First of all I needed a Raspberry Pi OS (64bit) lite image on the system:

$ curl -O https://downloads.raspberrypi.com/raspios_lite_arm64/images/raspios_lite_arm64-2024-03-15/2024-03-15-raspios-bookworm-arm64-lite.img.xz
$ unxz 2024-03-15-raspios-bookworm-arm64-lite.img.xz

Next step was writing the image to disk:

$ dd if=./2024-03-15-raspios-bookworm-arm64-lite.img of=/dev/nvme0n1

This resulted in a partition table on /dev/nvme0n1 so the output of lsblk changed:

$ lsblk
[...]
nvme0n1     259:0    0 465.8G  0 disk
|-nvme0n1p1 259:1    0   512M  0 part
`-nvme0n1p2 259:2    0   2.1G  0 part

OS modifications #

The OS written to disk did not include modifications to access after booting it. I skipped the Raspberry Pi image writer this time and did the modification in a chroot environment:

$ mount /dev/nvme0n1p2 /mnt
$ mount /dev/nvme0n1p1 /mnt/boot
$ chroot /mnt

The /boot/config.txt file in this environment also required dtparam=nvme since it was separate from the file (on the USB stick environment) previously adjusted.

Next up was changing the hostname:

$ echo "rpi-01" > /etc/hostname

To make sure that DNS resolution for its own name was working properly a slight modification to /etc/hosts and ensuring that the entry for 127.0.1.1 included the new hostname was necessary:

127.0.1.1		rpi-01 raspberrypi

For access to the system after boot I configured the existing user pi to have my SSH key. In addition /boot/ssh tells the OS to start sshd on boot.

$ mkdir -p /home/pi/.ssh
$ echo "ssh-ed25519 AAAA[...] embik" > /home/pi/.ssh/authorized_keys
$ chown -R /home/pi/.ssh pi:pi
$ touch /boot/ssh

I wanted to rename the pi user to embik on first boot by setting /boot/userconf.txt. Unfortunately you always need to pass a hashed password which can be generated by openssl passwd -6. Part of the post-boot steps was going to be removing the set password anyway.

$ echo "embik:$6$..." > /boot/userconf.txt

Everything on the NVMe disk was ready, so the only thing left there was to leave the chroot environment and make sure the disk was cleanly unmounted.

$ exit
$ umount /mnt/boot
$ umount /mnt

Boot from NVMe #

Last step was changing the boot order to make sure the next time the Raspberry Pi started, it would boot from my M.2 SSD.

$ rpi-eeprom-config --edit

I changed BOOT_ORDER to 0xf416 which attempts boot from NVMe and falls back in case the NVMe disk isn’t bootable. In addition this configuration needed PCIE_PROBE=1. Now everything is set and ready to reboot the system into the M.2 SSD.

A curious observation from my first setup was that while 0xf416 should describe the boot order “NVMe -> SD card -> USB stick” (see Raspberry Pi documentation), my Pi would not successfully boot from the still inserted SD card when my NVMe disk wasn’t formatted properly. I had to use an USB stick to recover the system.

Post-boot steps #

Once rebooted, ssh became available after a couple of seconds (booting from the M.2 SSD makes it essentially another system, so ssh-keygen -R was necessary before being able to connect). The system indeed booted from the NVMe disk and mounted its root partition:

$ lsblk
nvme0n1     259:0    0 465.8G  0 disk
|-nvme0n1p1 259:1    0   512M  0 part /boot/firmware
`-nvme0n1p2 259:2    0 465.3G  0 part /

To make sure that no one would be able to use password-based SSH authentication I deleted the user’s password:

$ sudo passwd -d embik

Two last things were necessary to prepare the Pi for Kubernetes: Disabling swap and enabling the memory group. The Raspberry Pi OS seems to have some special swapfile generation (swap isn’t mounted in /etc/fstab as usual), so this seems to be the way to disable swap on next boot:

$ systemctl disable dphys-swapfile.service

By default Raspberry Pi OS doesn’t enable the memory cgroup, which is a hard requirement for Kubernetes. That can be adjusted in /boot/firmware/cmdline.txt by appending two additional boot parameters (requires a reboot):

cgroup_enable=memory cgroup_memory=1

Rinse and repeat two more times to get all three Pis set up correctly.

Next up #

Hardware is all set up now! The next post will discuss setting up Kubernetes on these Raspberry Pis with KubeOne and keepalived. Stay tuned and subscribe to my blog via RSS to not miss any upcoming posts.

Sources #

• https://blog.alexellis.io/booting-the-raspberry-pi-5-from-nvme/
• https://www.jeffgeerling.com/blog/2023/nvme-ssd-boot-raspberry-pi-5
• https://ubuntu.com/tutorials/how-to-kubernetes-cluster-on-raspberry-pi#4-installing-microk8s