cpufreq collector hangs forever on arm64 with cppc_cpufreq, leaking a goroutine and an fd per scrape

[pablohoffman]

What did you do?

Ran the cpufreq collector on two arm64 machines (NVIDIA GB10 / DGX Spark) whose scaling driver is cppc_cpufreq, with 20 CPUs and 20 cpufreq policies.

The collector hangs permanently. It does not merely get slow: a goroutine blocked at process start was still blocked 1229 minutes (20.5 h) later, and every subsequent scrape leaks another goroutine and another open file descriptor.

Observed via Grafana Alloy's embedded prometheus.exporter.unix (Alloy v1.18.1, arm64, go1.26.5, github.com/prometheus/procfs v0.21.0). The stack is entirely inside node_exporter/procfs, but to be upfront: I have not tested standalone node_exporter on this hardware, so I cannot claim the standalone binary reproduces it — I would expect it to, since nothing in the path is Alloy-specific.

What did you expect to see?

The scrape completing, or at worst the collector failing and reporting node_scrape_collector_success{collector="cpufreq"} 0.

What did you see instead?

Roughly 40–60% of scrapes never return. From /debug/pprof/goroutine?debug=2:

goroutine 185317 [IO wait, 1229 minutes]:
internal/poll.runtime_pollWait(0xeb287c0f7e00, 0x72)
	/usr/local/go/src/runtime/netpoll.go:351 +0xa0
internal/poll.(*pollDesc).wait(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x28
internal/poll.(*FD).Read(...)
	/usr/local/go/src/internal/poll/fd_unix.go:165 +0x234
os.(*File).Read(...)
	/usr/local/go/src/os/file.go:144 +0x68
os.ReadFile(...)
	/usr/local/go/src/os/file.go:871 +0xb4
github.com/prometheus/procfs/internal/util.ReadUintFromFile(...)
	/go/pkg/mod/github.com/prometheus/procfs@v0.21.0/internal/util/parse.go:85 +0x20
github.com/prometheus/procfs/sysfs.parseCpufreqCpuinfo(...)
	/go/pkg/mod/github.com/prometheus/procfs@v0.21.0/sysfs/system_cpu.go:284 +0x154
github.com/prometheus/procfs/sysfs.FS.SystemCpufreq.func1()
	/go/pkg/mod/github.com/prometheus/procfs@v0.21.0/sysfs/system_cpu.go:249 +0x40
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/go/pkg/mod/golang.org/x/sync@v0.21.0/errgroup/errgroup.go:93 +0x4c

It is stuck in runtime_pollWait — the descriptor was registered with the netpoller and the kernel never marks it readable, so os.ReadFile never returns. Because SystemCpufreq waits on its errgroup, the whole NodeCollector.Collect blocks with it and the HTTP scrape times out.

The concurrency appears to be what triggers it. SystemCpufreq reads every policy in parallel (system_cpu.go:249), deliberately, per the comment about the kernel's intentional 50 ms per-CPU delay. Reading the same files sequentially from a shell is fast and never hangs:

$ for c in /sys/devices/system/cpu/cpu*/cpufreq/scaling_cur_freq; do timeout 3 cat $c; done
# all 20 return in < 200 ms total, no process ever enters D state

So a manual check "proves" the machine is healthy, which made this quite misleading to track down.

Impact

It is not only missing metrics — it is an unbounded resource leak, since each hung scrape leaves a goroutine and an fd behind forever. After ~21 h on a 120 s scrape interval:

	host A	host B	unaffected x86 host
alloy goroutines	2557	2128	313
open fds / pointing at cpufreq	1083 / 740	820 / 543	—
RSS	~800 MB	~800 MB	—
scrapes hanging (of 12)	5	4	0

Restarting the process reclaims it (goroutines 2612 → 273, cpufreq fds 760 → 0, RSS 800 → 386 MB), but the leak starts over immediately.

Workaround

Disabling the collector (--no-collector.cpufreq, or disable_collectors = ["cpufreq"] in Alloy) fixes it completely and instantly — 0 hung scrapes out of 12, max scrape time 30 s → 0.13 s.

Worth noting the trigger only became visible after a same-day upgrade of both the kernel (6.17.0-1021 → 6.17.0-1029-nvidia) and Alloy (1.17.x → 1.18.1) on these two hosts, so I cannot cleanly attribute it to either one; both hosts are the only ones in the fleet with both changes. What is clear is where it blocks. A 50 ms/CPU kernel delay cannot produce a 20-hour block, so something on the kernel/firmware side is failing to ever complete the read, and the parallel access is what exposes it. cppc_cpufreq reads go through the PCC firmware mailbox, which is a plausible place for concurrent access to serialize badly.

Even if the kernel is at fault, node_exporter arguably should not be able to leak unboundedly because of one unreadable sysfs file — a timeout or context on the per-policy reads would contain the damage.

System information

Linux 6.17.0-1029-nvidia aarch64 (Ubuntu 24.04, #29-Ubuntu SMP PREEMPT_DYNAMIC)
NVIDIA GB10 (DGX Spark class), 20 CPUs, 20 cpufreq policies
scaling_driver: cppc_cpufreq

Reproduced on two physically different machines from two different OEMs (Lenovo ThinkStation PGX and ASUS Ascent GX10), so it is not a single-vendor firmware quirk.

node_exporter version

Embedded in Grafana Alloy v1.18.1 (revision 6012ec4, goarch=arm64, goversion=go1.26.5), which vendors github.com/prometheus/procfs v0.21.0.

[waterWang]

/attempt #3791

Investigating the cpufreq collector hang on arm64 cppc_cpufreq. The stack shows os.ReadFile never returning inside procfs SystemCpufreq (system_cpu.go:249/284, running via errgroup). Plan: add a timeout/context bound to the per-policy sysfs reads so a hung read cannot block the whole scrape and leak goroutines/fds.

[waterWang]

Submitted fix: prometheus/procfs#861

Root cause: SystemCpufreq reads every policy in parallel via errgroup, and os.ReadFile on arm64 cppc_cpufreq can block forever (PCC firmware mailbox never responds). One stuck read hangs the whole collector permanently.

Fix: Added readCpufreqFile helper that uses SetReadDeadline (5s) to abort blocked reads. Timed-out attributes are skipped (graceful degradation), so the remaining CPUs still produce metrics and node_scrape_collector_success stays 1.

Verification:

• go build ./sysfs/ (GOOS=linux) ✓
• go vet ./sysfs/ ✓
• Deadline logic verified with os.Pipe + SetReadDeadline — blocked read returns i/o timeout within 300ms.
• TestReadCpufreqFileTimeout exercises the deadline-abort path with a FIFO.

Once merged, downstream projects (node_exporter, Grafana Alloy) need to bump github.com/prometheus/procfs to pick up the fix.