WIMH #1 (what's in my homelab)

Welcome to the homegarden

So you want a tour, huh ?

I've been writing about bits and pieces of this lab for a while now. I crashed the honey market, I tried (and partially failed) to evade my ISP, I went rootkit shopping and I played with LVM. But I've never actually sat you down and walked you through the whole thing. What's plugged in, what talks to what, and what's actually running in there.

So here we are. Grab a coffee, this one's gonna be long. Actually, grab the whole pot.

A quick disclaimer before we start, the same one I always give myself : this is a homelab. Or, as I like to call it, the not-so-home-but-more-like-prod-lab. None of this is "best practice" and none of this is a reference architecture. A good chunk of it exists purely because it was fun to build at 2 AM. That's the whole point. A production-ready homelab is about having fun trying things, and then doing them the right way because you got annoyed the third time it broke.

We'll do this in three parts, like a proper open house :

  • The system tour : what's physically in the rack and what each node is made of
  • The network tour : the fabric, the VLANs, and the zone-based firewall holding it all together
  • The VM tour : the 40-ish machines actually doing the work

Let's open the door (which has been removed since a few years).

Part 1 - The system tour

What's in the box

Everything lives in a single 24U rack that, against all my hopes and dreams, is already full. I keep telling myself 24U is plenty. It is not. It is never plenty. If you have a rack, you understand. If you don't, congratulations, you still have money.

Here's the current occupancy :

Going piece by piece :

  • UniFi Dream Machine SE (UDM-SE) : the network brain. It does everything, name it : routing, all the inter-VLAN firewalling, the IDS/IPS, and the geoblocking. Everything that crosses a zone boundary crosses it. We'll come back to this one in the network tour because it deserves its own section.
  • USW-Aggregation : my 10G core. This is what makes the whole "distributed storage over real bandwidth" thing possible instead of being a sad slideshow. The Ceph traffic lives here, you may find some packets playing tag behind the curtains (tag, vlans, you got the joke right ?)
  • TerraMaster U8-450 : an 8-bay NAS. This is the bulk-storage and backup target. My Proxmox Backup Server dumps onto it (more on that in the VM tour), so even if a node decides to spontaneously combust, the backups are sitting somewhere else entirely and it also contains
  • 3x Proxmox nodes : pivoine, petunia and dalia. The actual compute. The flowers. The stars of the show.
  • PowerWalker 1500 IoT UPS : 1500VA standing between my cluster and the kind of dirty power event that turns a healthy Ceph cluster into a very long evening.

And on the roadmap, because a homelab is never done even tho my wallet would like to :

  • a 48-port PoE switch, because I've run out of ports the same way I've run out of U
  • 3x GL.iNet Comet PoE : KVM-over-IP, one per node, powered over PoE. Translation : I'll be able to see the BIOS of a dead node and power-cycle it from my phone, from anywhere. The day I break my proxmox cluster and I need to access the proxmox node physically, I'll be happy to have those.

The nodes - meet the flowers

All three nodes run Proxmox VE (because, and I cannot stress this enough, fuck broadcom). They're named after flowers, which is also why the cluster itself is called homegarden. Yes, I'm aware. No, I won't apologize.

The first one, pivoine (a.k.a. pve01), is the one that started it all : it was my single-node lab before this whole thing metastasized into a cluster even though I stopped at 3 nodes.

  • CPU : AMD Ryzen 7 5700G (8c/16t)
  • RAM : 4x32 DDR4 3200MHz
  • NIC : Mellanox ConnectX-3 10GbE
  • OS disks : 2x Samsung 870 EVO 500GB
  • VM disks : 2x Samsung 980 PRO 2TB
  • Cooler : Noctua NH-D12L
  • PSU : be quiet! Pure Power 12 M 550W
  • Case : Silverstone RM42-502

The two other nodes, petunia (pve02) and dalia (pve03), share the exact same build :

  • CPU : AMD Ryzen 5 8600G (6c/12t)
  • RAM : 2x24 DDR5 5600MHz
  • NIC : Intel X710-DA2
  • OS disks : 2x Intel S3700 200GB (old enterprise SSDs that refuse to die)
  • VM disks : 2x Samsung 990 PRO 2TB
  • Cooler : Noctua NH-L12S
  • Case : RM23-502 Mini

And every node also got a dual 1G NIC bolted on, dedicated entirely to the cluster's nervous system. That's the network we're talking about next.

Add it all up and you've got a cluster with enough cores and RAM that I can be irresponsible with VM counts and still sleep fine. Which, spoiler, is exactly what I do. But I wish I could crank it up to 4x24Gb on petunia and dalia because right now, if pivoine fails, they haven't enough RAM to sustain the load.

Part 2 - The network tour

This is the part I'm proudest of, so lads, buckle the f*** up.

The fabric : from a cursed ring to OpenFabric

Back when I first clustered these three, I built what I called a ring network : three direct cables forming a triangle between the nodes, no switch involved, dedicated to Corosync and cluster chatter. The idea is sound. Keep the latency-sensitive cluster traffic off the switch entirely and give Corosync a clean, private path.

The implementation, however, was held together with hope and static routes. I carved out three /29 subnets, one per link, and manually wrote in /etc/network/interfaces : post-up ip route add ... via ... lines so that, say, pivoine could reach dalia by hairpinning through petunia. I also had to enable IP forwarding on every node to make the middle-man routing work.

It looked roughly like this :

            pivoine
           /        \
    (1G) /            \ (1G)
        /              \
   petunia ─────────── dalia
              (1G)

It worked, I mean, I wrote it and hoped it would work. Turns out it had two problems.

The first was cosmic. For reasons I will take to my grave, the 10.10.1.0/29 subnet simply would not route. I sat there until 3 AM rewriting configs, swapping cables, questioning my career, and then I changed it to 10.10.3.0/29, it worked first try, and I never spoke of it again. Goodbye 10.10.1.x, I loved you (I did not, I hate you, you costed me more sleep that you should have).

The second problem was real : static routing doesn't heal. Pull the petunia <=> dalia cable and the static path between those two is just... gone, until I fix it by hand. On a cluster where Corosync gets very opinionated about losing a link, that's not great.

So I upgraded the whole thing to OpenFabric.

OpenFabric (running on FRR, which Proxmox VE 9 now wires up natively through its SDN Fabrics feature) turns the triangle into a proper dynamic-routing fabric. Instead of brittle static routes between interface IPs, each node gets a loopback address in a 10.10.10.0/28 subet (10.10.10.1/.2/.3) and the fabric figures out the paths itself using a link-state protocol. Pull any single cable and it just reconverges over the remaining links. No more 3 AM. No more hand-editing. It heals.

[pivoine]
root@pivoine:~# vtysh -c "show openfabric neighbor"
Area mesh:
 System Id           Interface   L  State         Holdtime SNPA
 petunia             enp7s0      2  Up            9        2020.2020.2020
 dalia               enp8s0      2  Up            10       2020.2020.2020
root@pivoine:~# vtysh -c "show openfabric route" | grep 10.10
 10.10.10.1/32  IP internal  0                            pivoine(4)
 10.10.10.2/32  IP TE        20      petunia   enp7s0     petunia(4)
 10.10.10.3/32  IP TE        20      dalia     enp8s0     dalia(4)
 10.10.10.1/32  0       -          -           -
 10.10.10.2/32  20      enp7s0     10.10.10.2  -
 10.10.10.3/32  20      enp8s0     10.10.10.3  -

[petunia]
root@petunia:~# vtysh -c "show openfabric neighbor"
Area mesh:
 System Id           Interface   L  State         Holdtime SNPA
 pivoine             enp8s0      2  Up            10       2020.2020.2020
 dalia               enp9s0      2  Up            10       2020.2020.2020
root@petunia:~# vtysh -c "show openfabric route" | grep 10.10
 10.10.10.2/32  IP internal  0                            petunia(4)
 10.10.10.1/32  IP TE        20      pivoine   enp8s0     pivoine(4)
 10.10.10.3/32  IP TE        20      dalia     enp9s0     dalia(4)
 10.10.10.1/32  20      enp8s0     10.10.10.1  -
 10.10.10.2/32  0       -          -           -
 10.10.10.3/32  20      enp9s0     10.10.10.3  -

[dalia]
root@dalia:~# vtysh -c "show openfabric neighbor"
Area mesh:
 System Id           Interface   L  State         Holdtime SNPA
 pivoine             enp8s0      2  Up            10       2020.2020.2020
 petunia             enp9s0      2  Up            10       2020.2020.2020
root@dalia:~# vtysh -c "show openfabric route" | grep 10.10
 10.10.10.3/32  IP internal  0                            dalia(4)
 10.10.10.1/32  IP TE        20      pivoine   enp8s0     pivoine(4)
 10.10.10.2/32  IP TE        20      petunia   enp9s0     petunia(4)
 10.10.10.1/32  20      enp8s0     10.10.10.1  -
 10.10.10.2/32  20      enp9s0     10.10.10.2  -
 10.10.10.3/32  0       -          -           -

And then I layered Corosync on top properly :

  • link0 rides the OpenFabric loopbacks : low latency, self-healing, the primary path
  • link1 rides the management network as an independent switched fallback
[pivoine]
root@pivoine:~# corosync-cfgtool -n
Local node ID 1, transport knet
nodeid: 2 reachable
   LINK: 0 udp (10.10.10.1->10.10.10.2) enabled connected mtu: 1397
   LINK: 1 udp (10.100.2.101->10.100.2.102) enabled connected mtu: 1397

nodeid: 3 reachable
   LINK: 0 udp (10.10.10.1->10.10.10.3) enabled connected mtu: 1397
   LINK: 1 udp (10.100.2.101->10.100.2.103) enabled connected mtu: 1397

[petunia]
root@petunia:~# corosync-cfgtool -n
Local node ID 2, transport knet
nodeid: 1 reachable
   LINK: 0 udp (10.10.10.2->10.10.10.1) enabled connected mtu: 1397
   LINK: 1 udp (10.100.2.102->10.100.2.101) enabled connected mtu: 1397

nodeid: 3 reachable
   LINK: 0 udp (10.10.10.2->10.10.10.3) enabled connected mtu: 1397
   LINK: 1 udp (10.100.2.102->10.100.2.103) enabled connected mtu: 1397

[dalia]
root@dalia:~# corosync-cfgtool -n
Local node ID 3, transport knet
nodeid: 1 reachable
   LINK: 0 udp (10.10.10.3->10.10.10.1) enabled connected mtu: 1397
   LINK: 1 udp (10.100.2.103->10.100.2.101) enabled connected mtu: 1397

nodeid: 2 reachable
   LINK: 0 udp (10.10.10.3->10.10.10.2) enabled connected mtu: 1397
   LINK: 1 udp (10.100.2.103->10.100.2.102) enabled connected mtu: 1397

So now I can yank a single mesh cable and watch quorum not even flinch. Same physical triangle as before, completely different confidence level. I can now sleep deeply and dream about the UDM BEAST and its 25G interfaces...

The VLANs

Everything internal lives in RFC 1918 space (the good old 10.0.0.0/8), carved into one /24 per VLAN. I kinda like to think of myself as a rich guy using a whole /24 for 3 vms... The UDM-SE holds the gateway for each one. I'm not going to hand you the exact map (I've written before about not feeding people free metadata, so it'd be a bit rich to then publish my entire addressing plan), but the structure is the interesting part anyway :

Here's the full map :

VLAN What lives here Zone
1 UniFi gear management admin
2 Proxmox / node management admin
3 Ceph public cluster
4 Ceph private cluster
5 VPN vpn
10 Admin admin
15 Backup backup
20 Production internal
25 DMZ dmz
30 IoT internal
40 Lab / clients internal
45 House internal

Zone-based firewalling

This is where the UDM-SE does most of its work. Rather than writing a thousand VLAN-to-VLAN rules and slowly losing my mind, everything is grouped into firewall zones, and the rules live between zones :

  • Admin : VLANs 1, 2 and 10
  • Backup : VLANs 15
  • Internal : VLANs 20, 30, 40 and 45
  • DMZ : VLAN 25
  • VPN : VLAN 5
  • Cluster : VLANs 3 and 4

And the rules, in plain English :

The Admin zone is not a blanket "see everything" zone. VLANs 1 and 2 are pure management : UniFi gear talks on 1, servers management on 2, and that's it, no internet access and they don't get to roam, at all. It's specifically VLAN 10 that reaches everywhere. So "god mode" is one carefully-watched VLAN rather than a whole zone. If something on VLAN 10 gets popped, that's my bad day, but it's a much smaller attack surface than letting an entire admin block touch anything.

The Backup vlan used to live with the production stuff, until I realised that was exactly backwards. Your backups are the one thing that has to survive everything else getting owned, so the last place they should sit is the same broadcast domain as the services most likely to get popped. So it got its own zone, fully isolated and with no internet access at all. Nothing initiates a connection out, and nothing reaches in except the scoped flows that actually need to push or pull backups. If ransomware ever has a field day in the internal zone, it shouldn't even be able to see the box holding the copies that'll save me.

The Internal zone flows freely between its own VLANs, plus a few surgical exceptions : scoped management hops to specific vpn and cluster IPs (this is how Ansible reaches out to configure things), wazuh clients, nginx hosts in another zones, some prometheus scraping or uptime-kuma icmp and access into the DMZ where it's needed.

The DMZ is the paranoid one. It's walled off from everything, and it runs its own dedicated NTP and DNS so it never has to reach back into my production resolvers. The whole philosophy of a DMZ is "assume this gets compromised", so I don't want a compromised DMZ box doing DNS lookups against the same Pi-hole my house uses. It gets its own everything (I've got opinions about who gets to see my DNS, so no, the sketchy zone doesn't get to use the good resolver)).

The VPN and Cluster zones are also locked down tight, but they get one exception : they use the production DNS and NTP. They're trusted infrastructure rather than exposed services, so they get lucky and share the nice resolvers.

On top of all that, the UDM runs IDS/IPS and geoblocking at the edge. The geoblocking alone cuts an absurd amount of noise. If you've ever pointed a honeypot at the open internet (I have, it's a whole post), you know exactly how much of your inbound traffic is just a handful of countries' worth of bots knocking on every door. Block the regions you have no business talking to and the logs get a lot quieter.

Part 3 - The storage tour

The Hypervisors storage aka cephfs

Storage in this lab has had a journey, and the journey is half the fun.

It started as plain ZFS on a single node. Then, when I went to three nodes and got greedy for High Availability, I went down the LINSTOR + DRBD rabbit hole. I wrote a whole article about building that out, complete with a keepalived VIP and an HA controller. And It worked ! It was also a lot : DRBD reactor, promoter configs, a replicated controller database, etc. It was a great thing to learn and a great thing to eventually retire. I spent wayyy too much time and nights to fix those freaking DRDB errors when an nvme failed and I had to reboot in SOS mode to rebuild the cluster until the sun came out and I had to go to work.

Because I've since moved to Ceph, integrated directly into the Proxmox cluster.

As for why you may ask ? The reason is just fit. Proxmox's Ceph integration is first-class : it's in the GUI, it's in the cluster's bones, and it gives me the shared, self-healing block storage I wanted without me hand-rolling the replication layer. Each node runs 2 OSDs, so the cluster has 6 OSDs total spread across the three flowers, and the whole thing rides the 10G networking we set up earlier :

  • Ceph public lives on VLAN 3, on a dedicated 10G NIC
  • Ceph private / cluster lives on VLAN 4, also on a dedicated 10G NIC
[tree]
root@pivoine:/home/interlope# ceph osd tree
ID  CLASS  WEIGHT    TYPE NAME         STATUS  REWEIGHT  PRI-AFF
-1         10.91574  root default
-3          3.63858      host dalia
 0   nvme   1.81929          osd.0         up   1.00000  1.00000
 1   nvme   1.81929          osd.1         up   1.00000  1.00000
-5          3.63858      host petunia
 2   nvme   1.81929          osd.2         up   1.00000  1.00000
 3   nvme   1.81929          osd.3         up   1.00000  1.00000
-7          3.63858      host pivoine
 4   nvme   1.81929          osd.4         up   1.00000  1.00000
 5   nvme   1.81929          osd.5         up   1.00000  1.00000

[df]
root@pivoine:/home/interlope# ceph osd df
ID  CLASS  WEIGHT   REWEIGHT  SIZE     RAW USE  DATA     OMAP      META     AVAIL    %USE   VAR   PGS  STATUS
 0   nvme  1.81929   1.00000  1.8 TiB  294 GiB  293 GiB   869 KiB  1.2 GiB  1.5 TiB  15.80  0.85   60      up
 1   nvme  1.81929   1.00000  1.8 TiB  401 GiB  399 GiB   1.2 MiB  1.5 GiB  1.4 TiB  21.51  1.15   69      up
 2   nvme  1.81929   1.00000  1.8 TiB  295 GiB  294 GiB   867 KiB  1.3 GiB  1.5 TiB  15.85  0.85   56      up
 3   nvme  1.81929   1.00000  1.8 TiB  402 GiB  401 GiB   1.1 MiB  1.5 GiB  1.4 TiB  21.59  1.16   73      up
 4   nvme  1.81929   1.00000  1.8 TiB  347 GiB  345 GiB  1018 KiB  1.6 GiB  1.5 TiB  18.60  1.00   66      up
 5   nvme  1.81929   1.00000  1.8 TiB  346 GiB  344 GiB  1014 KiB  1.5 GiB  1.5 TiB  18.56  1.00   63      up
                       TOTAL   11 TiB  2.0 TiB  2.0 TiB   6.0 MiB  8.6 GiB  8.9 TiB  18.65
MIN/MAX VAR: 0.85/1.16  STDDEV: 2.34

[stat]
root@pivoine:/home/interlope# ceph osd stat
6 osds: 6 up (since 6d), 6 in (since 6d); epoch: e4248

Each node is also a monitor, a manager and a metadata server for a fully capable HA storage service. I also keep a dedicated Ceph pool for ISOs, so my install media and templates sit on the same resilient, every-node-can-see-it storage as the VMs. No more "oh that ISO only exists on pivoine" nonsense, it's just there, cluster-wide.

[status]
root@pivoine:/home/interlope# ceph -s
  cluster:
    id:     5819dbca-7480-4070-b1cb-aa44b2adfab1
    health: HEALTH_OK

  services:
    mon: 3 daemons, quorum pivoine,dalia,petunia (age 6d)
    mgr: petunia(active, since 6d), standbys: pivoine, dalia
    mds: 1/1 daemons up, 2 standby
    osd: 6 osds: 6 up (since 6d), 6 in (since 6d)

  data:
    volumes: 1/1 healthy
    pools:   4 pools, 129 pgs
    objects: 238.03k objects, 813 GiB
    usage:   2.0 TiB used, 8.9 TiB / 11 TiB avail
    pgs:     129 active+clean

  io:
    client:   773 KiB/s wr, 0 op/s rd, 84 op/s wr

[mon]
root@pivoine:/home/interlope# ceph mon stat
e3: 3 mons at {dalia=[v2:10.100.3.103:3300/0,v1:10.100.3.103:6789/0],petunia=[v2:10.100.3.102:3300/0,v1:10.100.3.102:6789/0],pivoine=[v2:10.100.3.101:3300/0,v1:10.100.3.101:6789/0]} removed_ranks: {} disallowed_leaders: {}, election epoch 402, leader 0 pivoine, quorum 0,1,2 pivoine,dalia,petunia

[mgr]
root@pivoine:/home/interlope# ceph mgr stat
{
    "epoch": 166,
    "available": true,
    "active_name": "petunia",
    "num_standby": 2
}

[fs]
root@pivoine:/home/interlope# ceph fs status
ceph-iso - 3 clients
========
RANK  STATE    MDS      ACTIVITY     DNS    INOS   DIRS   CAPS
 0    active  pve03  Reqs:    0 /s    22     23     16     12
       POOL          TYPE     USED  AVAIL
ceph-iso_metadata  metadata  1414k  2734G
  ceph-iso_data      data    30.2G  2734G
STANDBY MDS
   pve02
   pve01
MDS version: ceph version 19.2.3 (d74d168b1c80fb01e1a30d5e4ca9a45b12bc145b) squid (stable)

Is Ceph heavier than LINSTOR was ? Maybe it was, in some ways. Do I care ? No. The live migrations are clean, there's no IO delay that bothers me in practice, and it's one less bespoke stack for future-me to debug. And I sleep better at night. So I'd say it's a win-win for everyone.

The bulk & backup storage aka the NAS

Now for the storage that isn't trying to be fast, smart and distributed : the TerraMaster U8-450, running TrueNAS. Where Ceph is the fast, shared, self-healing tier the VMs actually live on, the NAS is the slow, big, "just don't lose my stuff" tier.

TrueNAS gives me ZFS with a proper GUI on top, and I've fed it 32GB of RAM for the ARC (because ZFS without RAM is just sadness with extra steps). It's split into two pools :

  • pool-ssd : 4x 1TB SATA SSDs. The landing zone for quick backups, the stuff I want written fast and read back fast. Backups are going there every night with a 7 iteration retention policy
  • pool-hdd : 4x 4TB Toshiba N300 spinning rust. The backups from the SSD pool replicate here weekly with a longer retention (30 iterations), plus it holds some films and the bulk data that doesn't need to be quick.

So the flow is : land fast on SSD, then let TrueNAS age it out onto the HDD pool weekly where I keep more history. My PBS datastores points here, so the whole VM fleet ends up safely off-cluster on storage that doesn't care if a node bursts into flames.

The one cloud on the horizon : free space is getting fierce. Between the bqckups and the films, those 4TB N300s are filling up faster than I'd like, and I've got my eye on some good old 16TB disks to give the HDD pool some breathing room. It's never if you run out of space in a homelab, only when, and apparently my when is now.

Sending it offsite aka the 3-2-1 rule

Everything so far keeps my data safe inside the rack. Ceph survives a dead node, the NAS survives a dead Ceph, PBS survives a dead VM. But all of it sits in the same room, on the same UPS, behind the same door. So if the house floods or burns down, every single copy goes with it. Not great.

The rule everyone throws around is 3-2-1 : 3 copies of your data, on 2 different types of media, with 1 of them offsite. I had the 3 and the 2 sorted, but I was missing the offsite one. (actually they weren't really sorted out since it was the same NAS on different pools and disks...)

That's where PBS 4.0 comes in, since it added S3-compatible object storage as a backup target. Instead of building a second physical PBS at a friend's place and maintaining it forever (and explaining to them why their internet dies every night at 2 AM), I can just push my backups to an S3 bucket in the cloud.

One gotcha : the S3 backend isn't just a dumb dump. PBS keeps a local cache (a dedicated dataset is best) so it's not hammering the S3 API and charging me for every read. So I get the cheap, infinite, somewhere-else storage without my restores crawling.

So the count finally adds up :

  • Copy 1 : the live data on Ceph
  • Copy 2 : the PBS backups on the NAS, different box, different media
  • Copy 3 : those same backups pushed offsite to S3

And since it's landing in someone else's datacenter, it's encrypted before it leaves the house. If you've read any of my other posts you already know I'm not handing a third party my plaintext anything. They get encrypted blobs, nothing else.

Is it perfect ? No. Cloud egress costs money and a full restore from S3 would be slow and annoying. But that copy is there for the day the whole rack is gone, not for restoring a VM I fat-fingered on a Tuesday. For everything else, the local NAS copy is right there and fast.

And then there's the dumbest copy of the lot : an external HDD on my desk that I plug in and sync by hand every now and then. No script, no schedule, I just remember to do it (most of the time I don't). The funny part is that a disk sitting unplugged on a shelf is the one thing ransomware physically can't reach, so my laziest backup somehow ended up being my most ransomware-proof one.

It looks like this :

   [ VMs ] ──► [ PBS ] ──► [ TrueNAS ]   (local, fast)
                       │
                       └──► [ S3 bucket ]   (offsite, encrypted)
                       └──► [ JBOD ]   (on my desk, encrypted)

Part 4 - The VM tour

Right, the part you actually clicked for.

At the time of writing I've got about 40 VMs and 1 golden debian13 template (the famous LVM-having Debian template, yes, the one I had to fix after years of regret). I'll walk them zone by zone, because that's how I think about them.

VPN zone

This is my Tailscale corner, and it's a little more elaborate than "install Tailscale, done".

First, 2 Tailscale signing nodes. The point of these is independence : by running my own signing nodes, node signing happens on infrastructure I own instead of relying fully on Tailscale's control plane for that trust decision. I use the service, but I hold some of my own keys.

Then 4 Tailscale instances, each with a deliberately different job :

  1. Full tunnel : routes all traffic, hands out DNS. The "I want everything to go home" exit node.
  2. Restricted : routes a limited set of subnets, no DNS. For when I want access to some of the lab without my whole device's traffic and name resolution going through it.
  3. Family : for the household, and its egress is deliberately pushed through the DMZ. The family gets connectivity without getting a door into the trusted internals but still got an ad-less network with better security.
  4. VPS entry : this is how my external VPS reaches into the homelab and exposes services to the internet. The bridge from the outside world to the inside, on my terms only.

Four instances sounds like overkill, but each one is a different trust posture, and keeping them as separate nodes means I never have to stop and ask "wait, which policy is this connection under".

Admin zone

  • Win11 VM : the management workstation. Sometimes you just need a Windows box with the right tools and a real browser to administer something that hates being administered from a terminal.
  • SSH bastion : the management workstation but shipped with Linux. It's like the CLI bastion where the win11 box is the click bastion.

Backup zone

  • Proxmox Backup Server (virtualized), with its datastore pointed at the Truenas TerraMaster box. This is the "3 AM peace of mind" machine. ZFS, Deduplicated, incremental backups of the whole VM fleet, living on storage that is not the Ceph cluster, so a catastrophic cluster event doesn't take the backups with it. The backup VLAN is its own segment for exactly this reason : backups should be reachable for backing up and basically nothing else.

Production zone

This is where most of the actual stuff lives. It's a lot, so let me group it.

The data backbone

  • pgs01 / pgs02 : a PostgreSQL master/replica pair. A surprising number of the services below point at this pair instead of running their own throwaway database. One good, replicated Postgres beats a dozen scattered SQLite files hidden inside containers.
[pgs01]
root@pgs01:~# sudo -u postgres psql -p 5432 -c "SELECT slot_name, slot_type, active FROM pg_replication_slots;"
 slot_name | slot_type | active
-----------+-----------+--------
 replica_1 | physical  | t
(1 row)

root@pgs01:~# sudo -u postgres psql -p 5432 -c "SELECT usename, client_addr, state, sent_lsn, replay_lsn FROM pg_stat_replication;"
  usename  | client_addr  |  state  |  sent_lsn  | replay_lsn
-----------+--------------+---------+------------+------------
 replicata | pgs02 | catchup | 0/F3060000 | 0/F3AD34F8
(1 row)

[pgs02]
root@pgs02:~# sudo -u postgres psql -p 5432 -c "SELECT pg_is_in_recovery();"
 pg_is_in_recovery
-------------------
 t
(1 row)

root@pgs02:~# sudo -u postgres psql -p 5432 -c "SELECT status, sender_host, slot_name, written_lsn, flushed_lsn, latest_end_lsn, last_msg_receipt_time FROM pg_stat_wal_receiver;"
  status   | sender_host  | slot_name | written_lsn | flushed_lsn | latest_end_lsn |     last_msg_receipt_time
-----------+--------------+-----------+-------------+-------------+----------------+-------------------------------
 streaming | pgs01 | replica_1 |             | 0/F3AD34F8  | 0/F6EC9EF8     | 2026-05-30 03:49:11.642677+02
(1 row)

Identity & access

  • SSO VM : running Pocket-ID for passkey-based SSO across the entire homelab. One identity, passkeys, everything behind it, less password and attack surface.
  • Guacamole VM : clientless web RDP, used to reach the Windows machines sitting over in the DMZ without exposing RDP directly and easier Windows management.

Reverse proxy & docs

  • Nginx VM : running Nginx Proxy Manager (NPM), pointed at the pgs pair. Combined to my dns, this is what turns my pile of host:port services into clean something.hlb1.lan URLs with Let's Encrypt certs.
  • Gitea VM : installed from binaries, also backed by pgs. It mostly hosts my Ansible playbooks and every docker-compose.yml in the lab, each on its own branch. My infrastructure's source of truth.
  • Wiki.js : also backed by pgs, running on docker, internal documentation, because future-me has the memory of a goldfish and deserves to be written to.

Automation

  • Semaphore VM : installed from binaries, pgs-backed. The web UI and scheduler in front of all those Ansible playbooks from Gitea. This is how config actually gets pushed out across the fleet (and why the firewall has those scoped management exceptions into other zones). I have a few dedicated categories for my homelab such as :

    • apt : this is what allows my entire homelab to stay up to date and modify some apt sources directory. It also runs an apt update playbook every day at 7am.
    • security : this is where I write my CVE patch for like dirtyfrag or copyfail.
    • utilities, ntp, snmpd : this is where the lazy IT guy shows up with its playbooks like deploy the snmpd con ; update a file ; check the linux version or the disk usage
    • ssh : this is where the ssh guidelines are applied with upmost respect and dedication.
[apt]
ansible@bastion:~/semaphore/apt$ ls
apt_add_package.yml  apt_full_upgrade.yml    apt_update.yml   conf               set_no_apt_cache.yml
apt_del_package.yml  apt_update_sources.yml  apt_upgrade.yml  set_apt_cache.yml
[security]
ansible@bastion:~/semaphore/security$ ls
cis_harden.yml  cis_vars.yml  copyfail-and-dirtyfrag.yml  requirements.yml
[utilities]
ansible@bastion:~/semaphore/utilities$ ls
deploy_k8s.yml    disk_space.yml  linstor_pve_cluster.yml  nuke_cluster.yml  rsyslog.yml     update_initramfs.yml
disable_ipv6.yml  enable_brr.yml  linux_version.yml        resolv.conf       update_dns.yml  update_portainer_agent.yml
[ntp]
ansible@bastion:~/semaphore/ntp$ ls
check_ntpsec_status.yml  ntp.conf  ntp-migration.yml  ntpsec_update_conf.yml
[snmpd]
ansible@bastion:~/semaphore/snpmd$ ls
observium_deploy.yml  snmpd.conf
[ssh]
ansible@bastion:~/semaphore/ssh$ ls
check_authorized_keys.yml  conf  remove_ssh_key.yml  ssh.yml

Monitoring & security - the graphs we all secretly love

  • Monitoring VM : Grafana (pointed at pgs01/02), Prometheus, and InfluxDB. The dashboards. The Graphics diseases hit me strong.
![](https://blog.interlope.xyz/bl-content/uploads/pages/8665d7e1f603526a0f1317a94f48035f/pivoine-png.png)
![](https://blog.interlope.xyz/bl-content/uploads/pages/8665d7e1f603526a0f1317a94f48035f/petunia-png.png)
![](https://blog.interlope.xyz/bl-content/uploads/pages/8665d7e1f603526a0f1317a94f48035f/dalia-png.png)
  • Observium : network-side monitoring, keeping an eye on the gear.
  • Wazuh : a full Wazuh deployment with agents across the entire homelab. Log collection, file-integrity monitoring, intrusion detection on the endpoints themselves. After going down the rabbit hole on why you can never truly trust a compromised machine, having centralized eyes on every box is less paranoia and more therapy. Even though I never actually look at all those logs...
  • uptime-kuma : the simple, beautiful "is it up ? is it green ?" board.
  • gotify : push notifications, so the lab can yell at me. I kinda like when it yells at me, gotta admit.

The "useful to actual humans" services

  • Immich : via docker, self-hosted photos. Google Photos, but mine.
  • Nextcloud AIO : also via docker, mostly files, the personal-cloud staple.
  • media stack VM : docker again guys ? Docker-based, Jellyfin and friends. Movie night, self-hosted.
  • Portainer VM : a Docker host managed through Portainer that also runs a grab-bag of smaller tools : BentoPDF, a dev instance of this very blog, Homebox, PairDrop (AirDrop for everyone), and Vaultwarden for secrets and passwords.

Lab-of-the-lab

  • GNS3 (gns03) : a network emulation sandbox for testing topologies and grinding through certifications without touching the real gear.

Time

  • ntp01 / ntp02 : the production time source the trusted zones lean on. Two of them, because time is the one thing you really don't want a single point of failure on (ask anyone who's debugged a cluster with drifting clocks).

DMZ zone

The sealed-off zone, with its own self-sufficient basics :

  • Pi-hole DNS : dedicated DMZ resolver, completely separate from the production one.
  • NTP/DNS box : the DMZ's own time and name services, so it never reaches back into prod.
  • A few Windows VMs : idle-game boxes and similar low-trust throwaways. The kind of thing you specifically don't want on your trusted network, which is what a DMZ is for. (These are the ones Guacamole reaches over to.)

IOT zone

  • Home Assistant OS (HAOS) : the whole smart-home brain, deliberately corralled on the IoT VLAN. IoT devices are not exactly famous for their security, so they get their own segment and HAOS sits in there with them rather than letting them anywhere near the good stuff.

LAB zone

  • A Kubernetes cluster : 1 control-plane node + 3 workers. My IaC and deployment playground. Somewhere to throw manifests, break things, test GitOps flows, and generally make a mess that doesn't touch anything load-bearing and did I already said break things ? Every homelab needs a corner where breaking things is the point.

Putting it all together

So here's the whole organism, zoomed all the way out :

Three flowers, one garden, a 10G spine for storage, a self-healing 1G fabric for the cluster's heartbeat, a zone-based firewall keeping everything in its lane, and a small city of VMs doing everything from serving my photos to catching bots to running my house.

Conclusion

If you've made it this far : thanks, and also, get a hobby (he says, having just written a small novel about his own).

None of this happened in a weekend. It's the result of a lot of late nights, a few cursed subnets, one entire storage stack I built and then retired, and a rack that's been full since roughly the day I bought it. Bits of it are over-engineered. Bits of it are held together with post-up and prayer. The bastion is on the wrong VLAN. And every one of those rough edges is something I learned by breaking first.

That, to me, is what a production-ready homelab really is. Not a perfect, finished, reference-grade datacenter, but a living thing you keep poking at because poking at it is fun. The day it's "done" is the day it gets boring, and then I'll just buy another switch I don't have room for and start again.

Thanks for reading me,

spleenftw

Homelab homelab network linux docker debian blog series-wimh proxmox ceph vlan firewall self-hosting
Share X (Twitter) Reddit LinkedIn