Homelab Showcase

This repo contains a sanitized excerpt from my homelab repo as of the date on the commit. It is kept up-to-date automatically using a Forgejo workflow.

My lab/prod setup consists of a series of hosts that all run a Debian variant or Alpine. Some of these hosts are physical machines running in my house, some are appliances that are more or less Raspberry Pi class SBCs, and others are VMs running either on the metal or as a remote VPS.

The purpose of this repo is to give a sort of flavor for what I think is unique and special about my setup, without getting too far into the weeds on specifics that don't matter or sensitive things that don't need to be public.

Don't try to take this thing and actually use it. I'm not going to guarantee that it's even internally coherent. What I will say is that I think it's neat and I hope you do too.

The Setup

This setup has evolved over a number of years, through a kubernetes deploy and back again, and over that time I've evolved a set of core ideas that make this make sense:

Static Service Discovery

Raft is anathema. Etcd? Get outta here.

Instead what we have is what I like to call "static service discovery", by which I mean all service discovery happens at build/deploy time.

Docker Compose

The whole deal here is that we compile a single Docker Compose formation from a set of stacks and deploy the whole shebang at once (see idea 1).

One of the neat things about Compose is that you can add "extensions" to your compose file. Compose will ignore any keys starting with x-, which in the docs they use for silly things like YAML anchors.

Extensions by themselves are kind of underwhelming so I also have a system of hooks that pick them up and do stuff with them. Hooks are Ruby classes that implement one or more lifecycle hook methods and can, among other things, manipulate the Docker Compose formation.

In this repo I use extensions for a variety of things:

• x-web sets up http ingress, including certificate management
• x-security sets sane security defaults
• x-backup specifies directories to back up
• x-gatus allows overriding the monitoring defaults (I use Gatus, it's neat you should check it out)
• x-cron runs scheduled jobs with Ofelia
• x-op-items installs 1Password secrets as files on the host

Templates

I use ERB templates all over the place to implement functionality around hooks. For example, the Caddy config is generated from all of the x-web declarations. Same for Ofelia and the backup cron.

Tailscale

Tailscale is integral to this setup. It's how the compose formation gets deployed, it's how backups get moved around, and it's how machines talk amongst themselves for backend services like authentication. It's also the backhaul between the externally facing VPS and the public applications running in my basement.

Image Baking

Baking docker images is not something I want to have to figure out multiple times, so I taught the system to do it for me. It looks for build: keys in a service and builds whatever is found there. Before building it hashes the contents of the stack directory plus any declared top-level x-source-inputs, then looks for a tag with that hash and the required target platforms, skipping if it finds one. That way baking is quick and I can just include it as part of the regular build process.

DNS and Certificates

I manage DNS with Bunny.net and dnscontrol. The dnscontrol config is pretty much static but there's a dynamically generated part for my homelab domain which builds out records for every service and host.

Along with DNS I also automatically manage certificates with lego. See the certificate-builder stack for details on how that works.

Networking

I take full advantage of Docker Compose's ability to set up wacky networking configurations. Each stack gets its own bridge network that is assigned a random /28 IPv4 network. It also gets an IPv6 network with some interesting properties:

• the lab as a whole has a /48 ULA prefix
• within that, each host gets a /64 based on a hash of its hostname
• within that, each stack gets a /96 based on a hash of its name
• within that, each service gets a /128 based on a hash of its name

<48 bit static>:<16 bit host>:<32 bit stack>:<32 bit container>

I haven't actually set up routes or the network overrides to make it possible to get at them from outside yet, but I have grand plans.

Introspection

Up until fairly recently, to diagnose problems I would ssh into machines and type docker ps and df and other commands with my meat hands. Like a sucker.

It turns out that LLMs are fairly good at driving CLIs, and also fairly good at putting together inferences and debugging stuff given enough information.

Hence: bin/hl, a little CLI that encapsulates a bunch of diagnostic patterns into a discoverable tool. The different "peek"s are presented in ways that make sense to small local LLMs so I don't have to depend on Big AI to figure out why Caddy is crashing this time.

There are a bunch of peeks:

where <name|fqdn>                         # locate a service
triage [host]                             # what's on fire?
route <fqdn> [chain]                      # show me what path an http request takes
host <host> summary                       # how is this host doing?
host <host> gatus                         # what does gatus say about the stuff on this host?
host <host> tailscale                     # how is tailscale doing?
host <host> stack <stack> ps              # are the containers running?
host <host> service <service> inspect     # show a redacted docker inspect
host <host> service <service> env         # show a redcated env
host <host> service <service> logs        # show redacted and limited logs
host <host> service <service> networks    # show the networks a service is attached to
host <host> path stat <absolute-path>     # does this file exist?
host <host> path ls <absolute-path>       # list files at a path
host <host> path read <absolute-path>     # read a file
host <host> rendered <stack>              # list artifacts
host <host> rendered <stack> <file>       # read one

Everything runs through the redaction system and file reads are strictly limited to an allow list.

This is fairly new but even so it's helped me diagnose two fires so far, so I'd say it was a worthwhile thing to do.

Examples

I've included a handful of example stacks, as well as an example config yaml. I suggest poking at the stacks, then the config yaml, and then reading through whatever hooks and peeks strike your fancy.

License

This work is licensed under CC BY-NC-SA 4.0.