diff --git a/about/homelab.mdx b/about/homelab.mdx index cccf701..45fb350 100644 --- a/about/homelab.mdx +++ b/about/homelab.mdx @@ -12,12 +12,76 @@ The homelab is a real production environment, just for one person. Proxmox clust | Layer | What's there | Notes | | --- | --- | --- | -| Compute | A 4-node Proxmox VE cluster | Heterogeneous mix — single-engineer homelab, parts opportunistically combined; 4 nodes give natural majority quorum | +| Compute | A 4-node Proxmox VE cluster | Heterogeneous mix — single-engineer homelab, parts opportunistically combined; 4 nodes give natural majority quorum. Two of the oldest nodes are being consolidated onto a single newer 2U chassis | | Local LLM | Model library shared across cluster nodes and a dedicated Apple Silicon Mac | The GPU-bound former standalone LLM box has since joined the Proxmox cluster as its fourth node | | Storage | ZFS on Proxmox hosts; SAS backplane on one node for cluster bulk storage; NVMe for hot tiers | Mixed-tier by accident, kept by design — bulk on SAS, working sets on NVMe | | Networking | UniFi end-to-end: gateway, switches (with 10G SFP+ uplinks), APs | Single-pane management; 10G fiber backbone where it matters | | Power | Rack UPS for servers; separate UPS for the Home Assistant Pi | Active NUT monitoring planned once the LLM box is built | -| Rack management | Raspberry Pi running Home Assistant; iDRAC vKVM jump VM in cluster for Java Web Start console access | Old BMC firmware needs a Java Web Start client; the jump VM keeps Java off the laptop | +| Rack management | Raspberry Pi running Home Assistant; BMC jump VM in cluster for legacy console access | Only the oldest BMCs still need a Java Web Start client, and the jump VM keeps Java off the laptop. Newer BMC generations expose an HTML5 console and a Redfish API, so both remote console and headless OS install work straight from the browser and the command line | + +### Buying used enterprise gear + +Most of the rack is second-hand, so every arrival is audited against its +listing before it is trusted. The BMC gives that away for free over Redfish +and the vendor CLI, and the useful checks are rarely the advertised specs: + +- **Remaining write endurance on every SSD.** Used enterprise drives are sold + on the life left in them, and a listing almost never quotes it. +- **Repeat counts in the system event log**, not just whether an error is + present. A recurring fault over months is a different machine from one that + logged something once. +- **The real-time clock.** A BMC reporting a date near the epoch means a dead + CMOS battery, whatever the seller claimed about replacing it. +- **Line input per power supply.** A `Critical` health rollup is very often + one uncorded supply rather than a hardware fault. +- **Prior-owner remnants.** Hostnames and retained logs survive a drive-level + secure erase, so their presence shows the erase was narrower than advertised. +- **Firmware currency, before anything else.** An old BMC doesn't just lack + niceties — it can drop entire management API surfaces: no memory or storage + inventory, no virtual media, no remote power control. Treat a firmware + update as required onboarding, not optional hygiene. +- **The chassis's CPU thermal ceiling, not the socket's.** Entry-tier 2U + chassis often share a cooling design with smaller siblings and cap TDP well + below what the socket supports. A bigger heatsink doesn't raise that + ceiling — it only changes which drive-count and ambient-temperature + combinations are certified. Check the ceiling, and whether a newer CPU + generation needs a BIOS update first, before buying an upgrade chip. +- **Warranty matters most on drives and power supplies, least on CPUs.** + Server CPUs rarely fail once burned in, and datacenters retire + previous-generation chips in bulk — that's where the price-per-performance + is. Put the warranty budget toward drives and PSUs instead, and burn in + every drive before trusting it. + +Firmware age has a second, sharper edge: the upgrade path itself can be long +and mandatory. One arrival needed six sequential firmware updates to reach +current, not one, because the vendor gates later versions behind a minimum +earlier version. Read the release notes for prerequisites before scheduling +the maintenance window — a "quick firmware update" turned into six reboots. + +The most valuable lesson: don't trust a used server's reported hardware +inventory until POST has completed cleanly at least once. One BMC reported +drives — full model and serial numbers included — that had been physically +removed. The stale data was identical across two independent management +APIs and survived both a BMC reset and a full cold power cycle. The cause: +hardware inventory is collected during POST, and POST never completed, so +the BMC kept serving a snapshot over a year old — visible only in its own +collection timestamp. Before trusting any drive or memory listing from a +used machine's BMC, check that the inventory-collection timestamp is +current. If it isn't, everything the BMC reports about installed hardware +is potentially fiction. + +That POST failure had a mundane cause: a single failed drive made the RAID +controller's UEFI driver report unhealthy, which halted POST at an +interactive prompt. With no monitor attached, the machine just looked hung +— and because POST never finished, the BMC never got a window to run its +queued firmware jobs, so they silently stalled instead of failing loudly. +The fix was diagnostic, not electrical: modern BMCs can export a screenshot +of the console over the management API, which turned an unexplained hang +into a one-line diagnosis — another capability the oldest firmware lacks. + +Findings become tickets and follow-up tasks rather than remembered caveats, +and the recorded inventory always reflects what the hardware reported — never +what the listing claimed. ## Network topology