I thought it'd be interesting to document the current state of my personal infrastructure which is used both for my own play and work needs. While it'd be fair to call most of this a homelab I actually have a few servers within a proper datacenter so "personal infrastructure" is perhaps a bit more apt (and a more fun title). Since everything is split between two "sites" we'll talk about them separately.

All of my machines run Linux via Ubuntu 24.04. This is mostly just due to historical reasons as I've worked running Ubuntu (or Debian) based machines for almost 20 years. I still remember getting a Ubuntu 6.06 cdrom shipped out to me and installing it on an old dell laptop. Despite my love for Ubuntu canonical really seems to hate innovations that benefit their users, so I refuse to use snap and hard-pin it to prevent it from ever being installed.

Most services I run are done via Docker and often managed via docker-compose. I've played with a lot of options but honestly this method works reliably and can be easily managed with tooling I'm familiar with. I've had plans for building a proper deployment engine for hosting stuff but just haven't had the time to bite off that work.

For filesystems I use ZFS (except for boot/OS NVMe drives) to back any HDD/SDD/NVMe based storage arrays. Because I run Linux ZFS is actually relatively hit or miss. I haven't had many issues except for one much older array that consistently has kernel task lockups which can be annoying to deal with. It'd be fun to play with using FreeBSD for my largest storage array server (jak, see below) but I just haven't had the time or motivation to open that can of worms yet.

Homelab

The first "site" is my home in the PNW area which benefits from a hydropower based electrical grid (e.g. cheap & stable) and a buried 1Gbps fiber connection via Centurylink. These are both pretty massive benefits for a homelab due to the stability and cost. My power rate until the last year has been 7c per KWH, and it's only gone up by a single cent this year. In the 5 years I've lived here we have had a single 3hr-ish power outage due to a local transformer being hit by a truck. The internet is obviously much less reliable than that, but it's still more than good enough for my needs and I'd estimate the downtime at a few minutes a month (generally around 1-3am). Everything is in a 25U rack which I keep in the garage to limit sound and heat issues.

Network

I run a 40Gbps network at home mostly due to my heavy use of NFSv4. Being able to quickly access files from my workstation for video-editing or development work is a massive time save and provides a huge quality of life boost to my day-to-day computer use.

Because of the AI-slop-push 40Gbps is long obsolete and the hardware for it can be found insanely cheap on ebay (some of it even brand new in the original packaging). I still remember diligently rolling out a brand new Juniper QFX switching layer to bring 40Gbps to braintree many years ago, so I'm also a bit nostalgic of this era (despite my Juniper experience being... bad)

For my needs I only have 4 machines connected via 40Gbps and everything else can either be 10Gbps or 1Gbps so an Arista 7050TX fit perfectly. I just use Mellanox ConnectX-3 based cards for all the hosts and DACs for everything in the rack. For my workstation (located outside the rack) I use a 50-foot OM3 MPO cable which is overkill but relatively cheap and gets the job done.

The primary router is a UDM Pro as it can directly terminate the PPPoE my ISP provides. It's extremely nice to completely bypass the ISPs default router and hop right onto the PON via the ONT. UniFi has come a long way and I'm getting too old to mess around with running my own Linux routers, so this is a relatively low cost and low maintenance solution. I also run my access points and cameras via UniFi for simplicity, but I'm not going to detail that setup as part of this post.

Everything that can runs tailscale which I can confirm is an incredible piece of software and greatly simplifies managing a personal infrastructure that spans multiple "sites". Almost all internal services that need to reach each-other due so over the encrypted tailscale connection. Before tailscale I had run my own wireguard setup, but boy do I not miss that.

Hardware

vel

This is an older host that I use mostly for development work and hosting a few personal-use services. It's setup with a AMD Ryzen 7 3800X and 64Gb RAM. At this point 95% of all code I write is done over vscode SSH on this machine. Splitting the compute load of development across two machines and moving all the power/heat/noise away from my workstation is a godsend for productivity. Another huge benefit is that my actual development workstation can be pretty much anything. I use Windows, Linux and even OSX for development, and the experience is nearly identical on all of them thanks to this setup.

sol

Sol is a relatively new machine I built to mostly run ClickHouse/Postgres and provide a fast NVMe storage array (4x2Tb RAID0) for bulk data processing. I tend to do a lot of work with large datasets, and on top of that I store a few hundred Gbs of my own monitoring and logging data that can be pretty much instantly queried on this machine. It features a AMD EPYC 7F52 and 128Gb RAM built into a Supermicro H11SSL-i which is a great platform that provides IPMI and tons of PCIe slots.

jak

Finally we have an old HP 2U machine containing 2x Intel Xeon E5-2660 v3 and 128Gb RAM. This is used as my primary ZFS machine and pretty much only hosts a large zpool that is served over NFSv4 and samba. The array consists of 4 mirrors of 2x 20Tb HDDs and 1 mirror of 2x 8Tb HDDs (e.g. RAID10, I'm not a huge fan of RAIDZ for a lot of reasons). This sums up to around 67Tb of usable capacity which is split across a bunch of different datasets, many with quotas/separate permissions/etc. I got this machine very cheap on ebay and I'm actually very impressed with how well it handles being a storage workhorse.

FMT2

The second location is a few racks within the Hurricane Electric FMT2 datacenter. Historically this rack has been used for various projects at UPLOL, but it also houses a variety of our own personal devices. Being a proper datacenter we benefit from excellent power distribution, cooling, and proper network peers. Having machines in a datacenter is wonderful for uptime but it definitely can be a hefty cost if you don't have methods to subsidize it. Regardless I can't recommend HE enough; we originally joined when they had a deal giving us a free rack for two years if we brought our own ASN (which is indeed a crazy good deal).

Network

Our primary uplink is 10Gbps peered via HE and our own ASN. Having direct access to a T2 network and running behind your own IP space is amazing for hosting services. The network speed and quality is also obviously very beneficial when dealing with large datasets or say scanning the internet. We have a UniFi EdgeRouter Infinity (ER-8-XG) that runs the whole network mostly just because it was the cheapest option at the time that could easily manage our BGP needs. We also have a 1Gbps link to (and participate within) FCIX.

Hardware

eos

This machine has taken many forms although at this point I think we've passed Ship of Theseus levels as the hostname itself has been around for over 10 years. This has always been my primary machine for hosting shared or fully public services. It contains a AMD Ryzen 9 3900X and 128Gb RAM. While this gen of CPU is certainly starting to show its age compared to the latest insanely powerful AMD lineup, it's still a great chip for hosting general services. Despite it not being "server grade" I've never had any real issues with the core hardware here (except for of course hard-drives).

xd-2/xd-3

Finally we have the remnants of a 3 node kubernetes cluster we used to run for UPLOL development work (don't ask about xd-1...). These machines also feature a AMD Ryzen 9 3900X (there it is) but only contain 64Gb of RAM. Today I just use these to run random ad-hoc services (game servers for friends, etc).

Setup & Services

Of course infrastructure and hardware is cool but how you setup and use it is often many times more interesting. While I run way too many things to list you'd probably be able to guess a lot of the usuals. Some of my favorites are vaultwarden, miniflux and overseerr.

Backups

One always important detail is backups and ensuring important data lives in multiple locations. I accomplish this mostly with restic and zrepl both of which push data from all my hosts onto jak. From there data is further replicated to Backblaze B2 for long-term offsite storage.

Configuration Management

I have my own (currently closed-source) configuration management tool called confck which I use to manage the configuration of all my hosts. All my configuration is described in confck TypeScript files which can be executed and applied to any of my hosts easily.

Monitoring

I also happen to have my own monitoring stack and (again, closed-source) observability tool/UI. This tool also lets me build dashboards and alerts with TypeScript files which has been incredible for reducing the amount of effort required to build some really incredible and useful visualizations.

Summary

All in all the current setup is a culmination of many years running infrastructure both for myself and work. While its certainly overkill in many regards, pretty much everything has a purpose and has been built specifically for its role. I'm very happy with how things run as its extremely low maintenance while providing a high quality of service to me, my family, and friends. On top of that it provides me a great platform for building and developing my own software both for my own personal use and my day job.