Lot’s going on in this one, not a lot fixed though! Enjoy.
Homelab Specs#
In all the the years I’ve been doing this I’m not actually sure I’ve ever put the specs of my machines down. Not because I’m hiding it, I just don’t really know anymore. A lot of the equipment has been gathered over the years and once it’s running I just leave it.
Main server#
.://:` `://:. root@proxmox2
`hMMMMMMd/ /dMMMMMMh` -------------
`sMMMMMMMd: :mMMMMMMMs` OS: Proxmox VE 9.2.3 x86_64
`-/+oo+/:`.yMMMMMMMh- -hMMMMMMMy.`:/+oo+/-` Host: NUC12WSHi5 (M46655-303)
`:oooooooo/`-hMMMMMMMyyMMMMMMMh-`/oooooooo:` Kernel: Linux 7.0.6-2-pve
`/oooooooo:`:mMMMMMMMMMMMMm:`:oooooooo/` Uptime: 6 days, 12 hours, 29 mins
./ooooooo+- +NMMMMMMMMN+ -+ooooooo/. Packages: 968 (dpkg)
.+ooooooo+-`oNMMMMNo`-+ooooooo+. Shell: bash 5.2.37
-+ooooooo/.`sMMs`./ooooooo+- Terminal: termproxy
:oooooooo/`..`/oooooooo: CPU: 12th Gen Intel(R) Core(TM) i5-1240P (16) @ 4.40 GHz
:oooooooo/`..`/oooooooo: GPU: Intel Iris Xe Graphics @ 1.30 GHz [Integrated]
-+ooooooo/.`sMMs`./ooooooo+- Memory: 43.14 GiB / 62.38 GiB (69%)
.+ooooooo+-`oNMMMMNo`-+ooooooo+. Swap: 44.00 KiB / 8.00 GiB (0%)
./ooooooo+- +NMMMMMMMMN+ -+ooooooo/. Disk (/): 22.55 GiB / 93.93 GiB (24%) - ext4
`/oooooooo:`:mMMMMMMMMMMMMm:`:oooooooo/` Local IP (vmbr0): 10.0.20.201/24
`:oooooooo/`-hMMMMMMMyyMMMMMMMh-`/oooooooo:` Locale: en_US.UTF-8
`-/+oo+/:`.yMMMMMMMh- -hMMMMMMMy.`:/+oo+/-`
`sMMMMMMMm: :dMMMMMMMs`
`hMMMMMMd/ /dMMMMMMh`
`://:` `://:`Media server#
_,met$$$$$gg. mbell@mediaserver
,g$$$$$$$$$$$$$$$P. -----------------
,g$$P"" """Y$$.". OS: Debian GNU/Linux 13 (trixie) x86_64
,$$P' `$$$. Host: NucBox G3
',$$P ,ggs. `$$b: Kernel: Linux 6.12.90+deb13.1-amd64
`d$$' ,$P"' . $$$ Uptime: 4 mins
$$P d$' , $$P Packages: 384 (dpkg)
$$: $$. - ,d$$' Shell: bash 5.2.37
$$; Y$b._ _,d$P' Terminal: /dev/pts/0
Y$$. `.`"Y$$$$P"' CPU: Intel(R) N100 (4) @ 3.40 GHz
`$$b "-.__ GPU: Intel UHD Graphics @ 0.75 GHz [Integrated]
`Y$$b Memory: 1.24 GiB / 15.38 GiB (8%)
`Y$$. Swap: 0 B / 976.00 MiB (0%)
`$$b. Disk (/): 72.08 GiB / 226.26 GiB (32%) - ext4
`Y$$b. Local IP (enp3s0): 10.0.20.203/24
`"Y$b._ Locale: en_GB.UTF-8
`""""Note: I used
fastfetchto grab these, it wasn’t available in Debian 12 so I did a quick upgrade to 13 just for this :)
NAS#
Synology something or other, more importantly disk space stats:
10.0.20.202:/volume1/Media 42T 27T 16T 64% /mnt/media2
Hetzner#
- CAX11 - VPN to bypass OSA
- CPX21 - remotelab.uk host
AWS#
Cloudfront/S3/Route53 mostly. All managed in Tofu.
remotelab ssl issues#
First some backstory! I use traefik as my reverse proxy on all my servers (I’m considering moving to Caddy or at least trialing it out at some point.), I also have a lot of internal domains that require SSL:
dig A idontexist.homelab.mikebell.io
; <<>> DiG 9.10.6 <<>> A idontexist.homelab.mikebell.io
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 40271
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;idontexist.homelab.mikebell.io. IN A
;; ANSWER SECTION:
idontexist.homelab.mikebell.io. 300 IN A 10.0.20.45
;; Query time: 30 msec
;; SERVER: 1.1.1.1#53(1.1.1.1)
;; WHEN: Mon Jun 15 08:37:42 BST 2026
;; MSG SIZE rcvd: 75If you want to SSL certs using LetsEncrypt then you can use the DNS resolver, it’s really easy to setup and works really well.
Fast forward to me having a public server (remotelab.uk) and I used the same pattern for setting up SSL certs, I’m not sure why I did this because it’s all public, the default HTTP challenge would have been simpler and cost me less in Route53 fees. A few months ago I was having issues with Traefik and how it picks up routes from docker labels (I still don’t know how I fixed it). I decided it’d be a good opportunity to simplify the setup and move to the HTTP challenge. I got everything moved over and confirmed that all the certs were generated and I was good to go, I didn’t however check which certs were still being used in Traefik. On Friday all the dns created certs expired and I started getting uptime-kuma down notifications on my phone. I finally managed to sit down on Saturday morning and look into it. For some reason Traefik was using the old certs from acme-dns.json and not acme-json. I took a backup of the dns file and then removed it and restarted Traefik which fixed the issue.
One annoying thing I found is that Traefik doesn’t clean up old certs so it’ll try renewing them for domains you no longer use this feels like a waste of Lets Encrypt resources.
ForgeJo renovate issues#
For the past few weeks renovate has been crashing out with:
/usr/local/sbin/renovate: line 13: 171 Killed /usr/local/renovate/node --use-openssl-ca "${RENOVATE_NODE_ARGS[@]}" /usr/local/renovate/dist/renovate.js "$@"
From what I’ve found through searching it’s getting killed because it’s running out of memory. I’m still deciding what to do with this my option are:
- Rescale the server - more cost
- Setup a vpn and run the runners on my local homelab - free but I’m not good with VPNs
- Move the runners to another server on hetzner - way more cost
I’m not super convinced I’ll be sticking with Hetzner in the long term, they’ve upped their prices and the lack of ARM servers make it less attractive as a host.
docker-in-docker config issues#
ForgeJo changed the way it does runner registration at some point (a really good thing! No more custom certs) so I decided to give it a go so that I could update my post on Buliding docker images with ForgeJo Actions.
Turns out I can’t get it working anymore, I’ve no idea why but docker-in-docker just refuses to work with my new setup. It’s a 100% a me problem but I haven’t had the time to properly dig into the old vs new config and figure out what’s going on.