From 264d1c216168cb3632f9582e2b70b404a16fb451 Mon Sep 17 00:00:00 2001 From: JacobPEvans <20714140+JacobPEvans-personal@users.noreply.github.com> Date: Sat, 25 Jul 2026 16:23:02 -0400 Subject: [PATCH 1/2] docs(homelab): note node consolidation and how used gear is vetted Two of the oldest cluster nodes are being folded onto a single newer 2U chassis, so the compute row now says so. The rack-management row was also out of date. It described Java Web Start as the console path generally, when that applies only to the oldest BMCs. Newer generations expose an HTML5 console and a Redfish API, which is what makes both remote console and headless OS install work without a monitor. Adds a short section on auditing second-hand hardware. Most of the rack is bought used, and the checks that actually matter are not the advertised specs - they are remaining SSD write endurance, repeat counts in the event log, the real-time clock as a CMOS battery test, per-supply line input, and prior-owner remnants that outlive a drive-level erase. Kept capability-neutral in line with the rest of the page: no models, serials, addresses, or hardware identifiers. Assisted-by: Claude:claude-opus-5[1m] Claude-Session: https://claude.ai/code/session_01KRNWhpAGfxN1RwxfZshLRE --- about/homelab.mdx | 26 ++++++++++++++++++++++++-- 1 file changed, 24 insertions(+), 2 deletions(-) diff --git a/about/homelab.mdx b/about/homelab.mdx index cccf701..df8c76a 100644 --- a/about/homelab.mdx +++ b/about/homelab.mdx @@ -12,12 +12,34 @@ The homelab is a real production environment, just for one person. Proxmox clust | Layer | What's there | Notes | | --- | --- | --- | -| Compute | A 4-node Proxmox VE cluster | Heterogeneous mix — single-engineer homelab, parts opportunistically combined; 4 nodes give natural majority quorum | +| Compute | A 4-node Proxmox VE cluster | Heterogeneous mix — single-engineer homelab, parts opportunistically combined; 4 nodes give natural majority quorum. Two of the oldest nodes are being consolidated onto a single newer 2U chassis | | Local LLM | Model library shared across cluster nodes and a dedicated Apple Silicon Mac | The GPU-bound former standalone LLM box has since joined the Proxmox cluster as its fourth node | | Storage | ZFS on Proxmox hosts; SAS backplane on one node for cluster bulk storage; NVMe for hot tiers | Mixed-tier by accident, kept by design — bulk on SAS, working sets on NVMe | | Networking | UniFi end-to-end: gateway, switches (with 10G SFP+ uplinks), APs | Single-pane management; 10G fiber backbone where it matters | | Power | Rack UPS for servers; separate UPS for the Home Assistant Pi | Active NUT monitoring planned once the LLM box is built | -| Rack management | Raspberry Pi running Home Assistant; iDRAC vKVM jump VM in cluster for Java Web Start console access | Old BMC firmware needs a Java Web Start client; the jump VM keeps Java off the laptop | +| Rack management | Raspberry Pi running Home Assistant; BMC jump VM in cluster for legacy console access | Only the oldest BMCs still need a Java Web Start client, and the jump VM keeps Java off the laptop. Newer BMC generations expose an HTML5 console and a Redfish API, so both remote console and headless OS install work straight from the browser and the command line | + +### Buying used enterprise gear + +Most of the rack is second-hand, so every arrival is audited against its +listing before it is trusted. The BMC gives that away for free over Redfish +and the vendor CLI, and the useful checks are rarely the advertised specs: + +- **Remaining write endurance on every SSD.** Used enterprise drives are sold + on the life left in them, and a listing almost never quotes it. +- **Repeat counts in the system event log**, not just whether an error is + present. A recurring fault over months is a different machine from one that + logged something once. +- **The real-time clock.** A BMC reporting a date near the epoch means a dead + CMOS battery, whatever the seller claimed about replacing it. +- **Line input per power supply.** A `Critical` health rollup is very often + one uncorded supply rather than a hardware fault. +- **Prior-owner remnants.** Hostnames and retained logs survive a drive-level + secure erase, so their presence shows the erase was narrower than advertised. + +Findings become tickets and follow-up tasks rather than remembered caveats, +and the recorded inventory always reflects what the hardware reported — never +what the listing claimed. ## Network topology From 50e9f73431bf4156f5f68f963aff9c74c563e641 Mon Sep 17 00:00:00 2001 From: JacobPEvans <20714140+JacobPEvans-personal@users.noreply.github.com> Date: Sun, 26 Jul 2026 00:21:30 -0400 Subject: [PATCH 2/2] docs(homelab): extend used-gear lessons with firmware and POST inventory pitfalls Adds sanitized lessons from a used-server bring-up: staged/mandatory firmware upgrade paths, not trusting BMC hardware inventory until POST completes cleanly, a single failed drive halting POST invisibly, CPU thermal ceilings on entry 2U chassis, and where warranty value actually lives on used gear. --- about/homelab.mdx | 42 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 42 insertions(+) diff --git a/about/homelab.mdx b/about/homelab.mdx index df8c76a..45fb350 100644 --- a/about/homelab.mdx +++ b/about/homelab.mdx @@ -36,6 +36,48 @@ and the vendor CLI, and the useful checks are rarely the advertised specs: one uncorded supply rather than a hardware fault. - **Prior-owner remnants.** Hostnames and retained logs survive a drive-level secure erase, so their presence shows the erase was narrower than advertised. +- **Firmware currency, before anything else.** An old BMC doesn't just lack + niceties — it can drop entire management API surfaces: no memory or storage + inventory, no virtual media, no remote power control. Treat a firmware + update as required onboarding, not optional hygiene. +- **The chassis's CPU thermal ceiling, not the socket's.** Entry-tier 2U + chassis often share a cooling design with smaller siblings and cap TDP well + below what the socket supports. A bigger heatsink doesn't raise that + ceiling — it only changes which drive-count and ambient-temperature + combinations are certified. Check the ceiling, and whether a newer CPU + generation needs a BIOS update first, before buying an upgrade chip. +- **Warranty matters most on drives and power supplies, least on CPUs.** + Server CPUs rarely fail once burned in, and datacenters retire + previous-generation chips in bulk — that's where the price-per-performance + is. Put the warranty budget toward drives and PSUs instead, and burn in + every drive before trusting it. + +Firmware age has a second, sharper edge: the upgrade path itself can be long +and mandatory. One arrival needed six sequential firmware updates to reach +current, not one, because the vendor gates later versions behind a minimum +earlier version. Read the release notes for prerequisites before scheduling +the maintenance window — a "quick firmware update" turned into six reboots. + +The most valuable lesson: don't trust a used server's reported hardware +inventory until POST has completed cleanly at least once. One BMC reported +drives — full model and serial numbers included — that had been physically +removed. The stale data was identical across two independent management +APIs and survived both a BMC reset and a full cold power cycle. The cause: +hardware inventory is collected during POST, and POST never completed, so +the BMC kept serving a snapshot over a year old — visible only in its own +collection timestamp. Before trusting any drive or memory listing from a +used machine's BMC, check that the inventory-collection timestamp is +current. If it isn't, everything the BMC reports about installed hardware +is potentially fiction. + +That POST failure had a mundane cause: a single failed drive made the RAID +controller's UEFI driver report unhealthy, which halted POST at an +interactive prompt. With no monitor attached, the machine just looked hung +— and because POST never finished, the BMC never got a window to run its +queued firmware jobs, so they silently stalled instead of failing loudly. +The fix was diagnostic, not electrical: modern BMCs can export a screenshot +of the console over the management API, which turned an unexplained hang +into a one-line diagnosis — another capability the oldest firmware lacks. Findings become tickets and follow-up tasks rather than remembered caveats, and the recorded inventory always reflects what the hardware reported — never