Add /proc + PSI resource sampling timeline (observability Phase 1) - #11
Add /proc + PSI resource sampling timeline (observability Phase 1)#11Agent-Hellboy wants to merge 1 commit into
Conversation
The per-event activity log shows what ran but not machine state: during an incident (e.g. a one-minute hang) there is no CPU/memory/pressure/disk-latency context to explain the stall. mmap events are not memory usage; block_rq_issue alone is not disk latency. This adds a low-frequency sampling timeline — aggregate state, not per-scheduler-event tracing — per issue #1 section 8. New userspace sampler (internal/sample) reads /proc + PSI, Linux only (no eBPF, no-op stub elsewhere), and writes two tables queryable via `ltm query sql`: - system_samples (~1s): CPU%, load, runnable/blocked, memory/swap, PSI cpu/mem/io avg10, aggregate disk and network throughput. - process_samples (~5s): per-process CPU%, RSS, state, threads, cumulative I/O, cgroup — one row per process per tick. Rates (CPU%, disk, net) are deltas over the interval since the previous sample; process CPU% uses measured wall-elapsed time, not the assumed tick spacing. - Schema: abi.yaml + gen tool now emit DDL/SchemaDoc/column-lists for multiple tables (events keeps its historical constant names; sample tables get <goName>Columns/InsertColumns/InsertPlaceholders). Pure /proc parsers are split from Linux I/O so the text formats are unit-tested on any OS. - Daemon: sampleLoop runs on its own 1s/5s cadence (daemon.Config SystemSampleEvery/ProcessSampleEvery; negative disables), joined before the store closes. Writes share the single-writer connection with the flush loop. - storage: SystemSample/ProcessSample types, batch inserts, LatestSystemSample; Prune now trims the sample tables alongside events (process_samples grows with process count). - ltm status prints the latest system sample as a one-liner (+ JSON "system"). Verified on Linux 6.8: full test suite, integration.sh, and a live run — 12 system samples in ~13s, 800 process samples, CPU% correct (busy loop 99.9%, dd disk writes captured), PSI parsed, status one-liner rendered, 0 drops. Docs: recording.md resource-sampling section. CHANGELOG updated.
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 311c6fdac3
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| out.DiskReadKB = sectorsToKB(dsk.sectorsRead - s.prevDsk.sectorsRead) | ||
| out.DiskWriteKB = sectorsToKB(dsk.sectorsWritten - s.prevDsk.sectorsWritten) | ||
| out.NetRxKB = int64((net.rxBytes - s.prevNet.rxBytes) / 1024) | ||
| out.NetTxKB = int64((net.txBytes - s.prevNet.txBytes) / 1024) |
There was a problem hiding this comment.
When a network interface (for example a short-lived veth) or block device disappears/reappears, the aggregate current counter can be lower than the previous sample; subtracting these uint64 values wraps before conversion, so the next row can record enormous disk/net KB values and corrupt the resource timeline. Please detect current < previous for these cumulative counters and treat it as a reset/baseline update instead of emitting a wrapped delta.
Useful? React with 👍 / 👎.
| if prev, ok := s.prevProc[pid]; ok && st.jiffies >= prev { | ||
| ps.CPUPct = procCPUPct(st.jiffies-prev, elapsed) |
There was a problem hiding this comment.
If a PID exits and another process reuses the same PID between the 5s sweeps, prevProc[pid] is still treated as the previous sample for the new task; whenever the new task's jiffies are greater than or equal to the old value, st.jiffies-prev combines two unrelated process lifetimes and reports bogus CPU%. Please key the previous sample by a stable process identity such as PID plus /proc/<pid>/stat start time, or reset the delta when that identity changes.
Useful? React with 👍 / 👎.
1 participant
Stacked on #10 (base is
fix/self-capture-feedback-loop); GitHub will retarget this tomainonce #10 merges. Implements issue #1 section 8, Phase 1.Why
The per-event activity log shows what ran but not machine state: during an incident (e.g. a one-minute hang) there is no CPU/memory/pressure/disk context to explain the stall.
mmapevents are not memory usage;block_rq_issuealone is not disk latency. This adds a low-frequency sampling timeline — aggregate state, not per-scheduler-event tracing.What
New userspace sampler (
internal/sample, reads/proc+ PSI, Linux-only, no eBPF, no-op stub elsewhere) writing two tables queryable vialtm query sql:system_samples(~1s): CPU %, load, runnable/blocked, memory/swap, PSI cpu/mem/io avg10, aggregate disk & network throughput.process_samples(~5s): per-process CPU %, RSS, state, threads, cumulative I/O, cgroup — one row per process per tick.Rates (CPU %, disk, net) are deltas over the interval since the previous sample; process CPU% uses measured wall-elapsed time, not the assumed tick spacing.
abi.yaml+ gen tool now emit DDL/SchemaDoc/column-lists for multiple tables. Pure/procparsers split from Linux I/O so formats are unit-tested on any OS.sampleLoopon its own 1s/5s cadence (daemon.ConfigSystemSampleEvery/ProcessSampleEvery; negative disables), joined before the store closes.LatestSystemSample;Prunenow trims the sample tables (relevant becauseprocess_samplesgrows with process count).ltm status: latest system sample as a one-liner (+ JSONsystem).Verification (Linux 6.8)
Full test suite +
integration.shgreen, plus a live run: 12 system samples in ~13s, 800 process samples, CPU% correct (busy loop 99.9%,dddisk writes captured), PSI parsed,statusone-liner rendered, 0 drops.Docs:
recording.mdresource-sampling section. CHANGELOG updated.🤖 Generated with Claude Code