docs(homelab): note node consolidation and how used gear is vetted

about/homelab.mdx

@@ -12,12 +12,76 @@ The homelab is a real production environment, just for one person. Proxmox clust
 
 | Layer | What's there | Notes |
 | --- | --- | --- |
-| Compute | A 4-node Proxmox VE cluster | Heterogeneous mix — single-engineer homelab, parts opportunistically combined; 4 nodes give natural majority quorum |
+| Compute | A 4-node Proxmox VE cluster | Heterogeneous mix — single-engineer homelab, parts opportunistically combined; 4 nodes give natural majority quorum. Two of the oldest nodes are being consolidated onto a single newer 2U chassis |
 | Local LLM | Model library shared across cluster nodes and a dedicated Apple Silicon Mac | The GPU-bound former standalone LLM box has since joined the Proxmox cluster as its fourth node |
 | Storage | ZFS on Proxmox hosts; SAS backplane on one node for cluster bulk storage; NVMe for hot tiers | Mixed-tier by accident, kept by design — bulk on SAS, working sets on NVMe |
 | Networking | UniFi end-to-end: gateway, switches (with 10G SFP+ uplinks), APs | Single-pane management; 10G fiber backbone where it matters |
 | Power | Rack UPS for servers; separate UPS for the Home Assistant Pi | Active NUT monitoring planned once the LLM box is built |
-| Rack management | Raspberry Pi running Home Assistant; iDRAC vKVM jump VM in cluster for Java Web Start console access | Old BMC firmware needs a Java Web Start client; the jump VM keeps Java off the laptop |
+| Rack management | Raspberry Pi running Home Assistant; BMC jump VM in cluster for legacy console access | Only the oldest BMCs still need a Java Web Start client, and the jump VM keeps Java off the laptop. Newer BMC generations expose an HTML5 console and a Redfish API, so both remote console and headless OS install work straight from the browser and the command line |
+
+### Buying used enterprise gear
+
+Most of the rack is second-hand, so every arrival is audited against its
+listing before it is trusted. The BMC gives that away for free over Redfish
+and the vendor CLI, and the useful checks are rarely the advertised specs:
+
+- **Remaining write endurance on every SSD.** Used enterprise drives are sold
+  on the life left in them, and a listing almost never quotes it.
+- **Repeat counts in the system event log**, not just whether an error is
+  present. A recurring fault over months is a different machine from one that
+  logged something once.
+- **The real-time clock.** A BMC reporting a date near the epoch means a dead
+  CMOS battery, whatever the seller claimed about replacing it.
+- **Line input per power supply.** A `Critical` health rollup is very often
+  one uncorded supply rather than a hardware fault.
+- **Prior-owner remnants.** Hostnames and retained logs survive a drive-level
+  secure erase, so their presence shows the erase was narrower than advertised.
+- **Firmware currency, before anything else.** An old BMC doesn't just lack
+  niceties — it can drop entire management API surfaces: no memory or storage
+  inventory, no virtual media, no remote power control. Treat a firmware
+  update as required onboarding, not optional hygiene.
+- **The chassis's CPU thermal ceiling, not the socket's.** Entry-tier 2U
+  chassis often share a cooling design with smaller siblings and cap TDP well
+  below what the socket supports. A bigger heatsink doesn't raise that
+  ceiling — it only changes which drive-count and ambient-temperature
+  combinations are certified. Check the ceiling, and whether a newer CPU
+  generation needs a BIOS update first, before buying an upgrade chip.
+- **Warranty matters most on drives and power supplies, least on CPUs.**
+  Server CPUs rarely fail once burned in, and datacenters retire
+  previous-generation chips in bulk — that's where the price-per-performance
+  is. Put the warranty budget toward drives and PSUs instead, and burn in
+  every drive before trusting it.
+
+Firmware age has a second, sharper edge: the upgrade path itself can be long
+and mandatory. One arrival needed six sequential firmware updates to reach
+current, not one, because the vendor gates later versions behind a minimum
+earlier version. Read the release notes for prerequisites before scheduling
+the maintenance window — a "quick firmware update" turned into six reboots.
+
+The most valuable lesson: don't trust a used server's reported hardware
+inventory until POST has completed cleanly at least once. One BMC reported
+drives — full model and serial numbers included — that had been physically
+removed. The stale data was identical across two independent management
+APIs and survived both a BMC reset and a full cold power cycle. The cause:
+hardware inventory is collected during POST, and POST never completed, so
+the BMC kept serving a snapshot over a year old — visible only in its own
+collection timestamp. Before trusting any drive or memory listing from a
+used machine's BMC, check that the inventory-collection timestamp is
+current. If it isn't, everything the BMC reports about installed hardware
+is potentially fiction.
+
+That POST failure had a mundane cause: a single failed drive made the RAID
+controller's UEFI driver report unhealthy, which halted POST at an
+interactive prompt. With no monitor attached, the machine just looked hung
+— and because POST never finished, the BMC never got a window to run its
+queued firmware jobs, so they silently stalled instead of failing loudly.
+The fix was diagnostic, not electrical: modern BMCs can export a screenshot
+of the console over the management API, which turned an unexplained hang
+into a one-line diagnosis — another capability the oldest firmware lacks.
+
+Findings become tickets and follow-up tasks rather than remembered caveats,
+and the recorded inventory always reflects what the hardware reported — never
+what the listing claimed.
 
 ## Network topology