schedwhy

schedwhy explains one thread's runnable waiting time without claiming more than the kernel observations support. Version 0.4 follows one TID from sched_wakeup until it becomes next in sched_switch, and assigns every nanosecond to exactly one cause bucket.

Accounting model

Intervals are settled whenever an observed state changes. Overlap is resolved with this fixed priority:

1. cgroup CFS bandwidth throttling
2. post-migration wait
3. hard IRQ or softirq execution
4. a SCHED_FIFO, SCHED_RR, or SCHED_DEADLINE occupant
5. EEVDF target entity was ineligible
6. EEVDF selected entity had an earlier virtual deadline
7. fair-scheduler competition or unknown

The final bucket is deliberately conservative. Merely observing another fair task on a CPU does not prove why the target lost a scheduling decision.

For every schedulable entity from the target thread to the root cfs_rq, v0.3 correlates three scheduler functions:

1. pick_eevdf() starts a selection context.
2. entity_eligible() supplies direct eligibility evidence when the search evaluates the target explicitly.
3. set_next_entity() identifies the entity that actually won.

At wakeup, the collector snapshots up to eight (sched_entity, cfs_rq) levels. Each pick_eevdf() is matched to the target entity competing at that specific level. The first target entity that loses identifies whether the leaf thread or one of its parent cgroups prevented the target from participating further down the hierarchy.

The augmented EEVDF tree can skip the target without calling entity_eligible() on it. In that case, v0.3 reconstructs the same eligibility comparison from sum_w_vruntime, sum_weight, zero_vruntime, and the current entity. Reports mark this evidence as reconstructed. If the required hooks or fields are unavailable, the decision remains EEVDF_UNRESOLVED. The model follows the kernel's EEVDF documentation and fair.c.

What “why” means

Every direct explanation is structured as an auditable chain:

observed kernel state → scheduler rule → decision → runnable-delay effect

For EEVDF, the collector records the adjusted cfs_rq virtual-runtime sum, weight, zero point, and target key used by the eligibility comparison. User space derives the runqueue average, the target's virtual-time lead, and lag:

rq_average = zero_vruntime + adjusted_sum / adjusted_weight
service_lead = target_vruntime - rq_average
lag = rq_average - target_vruntime

A positive service lead produces negative lag and explains an ineligible decision. If the target is eligible, target_deadline - selected_deadline shows exactly how much earlier the winner's virtual deadline was. The report keeps direct and reconstructed verdicts distinct.

Throttle explanations include runtime_remaining, quota, and period when the kernel exposes them. Interrupt explanations include the IRQ handler name or softirq vector. Higher-class explanations state the competing policy and the class-precedence rule.

This scope ends at the kernel decision. A negative lag can prove that an entity arrived at the decision ahead in virtual time; it cannot identify which application function created the preceding CPU demand. That requires later stack or application-event correlation.

Each completed report is checked in user space:

sum(cause_ns) == run_timestamp - wakeup_timestamp

The BPF side emits only state boundaries for the selected task and its current CPU. A user-space reducer owns the timeline and accounting policy. If a boundary event is dropped, the episode is marked tainted and no explanation is printed.

Build and run

Requirements: Linux with BTF and eBPF enabled, clang, bpftool, libbpf, libelf, zlib, make, and root privileges for loading programs.

make
make test
sudo ./build/schedwhy --tid 8172
sudo ./build/schedwhy --tid 8172 --explain
sudo ./build/schedwhy --tid 8172 --json
sudo ./build/schedwhy --tid 8172 --html schedwhy-report.html
sudo ./build/schedwhy --tid 8172 --record capture.swhy
./build/schedwhy --replay capture.swhy --html replay.html

Output views

The default terminal view is intentionally compact: it prints the total delay and only non-zero cause buckets. --explain adds wakeup context, direct or reconstructed evidence, EEVDF samples, a four-step “why this episode waited” proof, event counts, and the accounting check.

--json emits newline-delimited JSON, one object per completed runnable episode. The top-level schema_version is 1. In addition to aggregate causes_ns, each object contains wake_ns, run_ns, and a timeline of mutually exclusive intervals expressed as offsets from wakeup. This makes the stream suitable for downstream analysis without changing the accounting model. eevdf_hierarchy.layers lists each competition level from leaf to root, its cfs_rq cgroup, target entity, per-reason decision counts, and the latest losing layer. The why object contains the numeric EEVDF proof, bandwidth evidence, interrupt identity, higher-class sample, and an explicit evidence boundary. Cgroup IDs are resolved to cgroup v2 paths when they still exist locally.

--record FILE stores the raw boundary-event stream in a versioned binary format. --replay FILE runs the same reducer, JSON writer, and HTML renderer without root access or eBPF attachment. Recordings are intentionally tied to their declared event ABI version; incompatible files fail closed rather than being partially decoded.

--html FILE writes the same reports as a self-contained HTML document. It starts with capture-level latency percentiles and aggregate cause allocation, then renders the selected episode as a horizontal runnable path from waker, through the attributed wait, to sched_switch. Mutually exclusive causes appear as colored phases; the largest causes are labeled with exact duration and share, and every individual interval remains available on hover. A selector switches between episodes. A scheduler-hierarchy path below the timeline marks the exact cgroup or leaf level where the latest EEVDF decision was lost. A second causal path renders observed state → kernel rule → decision → effect, including the measured lag or deadline gap. The slowest episodes remain available with expandable raw timelines and EEVDF decision context, with zero-valued causes omitted. The document is finalized when tracing stops; it can be combined with either terminal mode or --json.

Timeline detail is capped at 1024 merged intervals per episode. If that limit is exceeded, JSON sets timeline_truncated and HTML shows a warning. Aggregate cause accounting remains complete and must still sum exactly to total_ns.

The CFS throttle probes are optional at attach time. If the kernel does not expose throttle_cfs_rq and unthrottle_cfs_rq, the tool warns and keeps running; that time remains in fair scheduler or unknown.

Evidence and limitations

• Wake, switch, migration, IRQ, softirq, and exit use typed BTF tracepoints with CO-RE task-field reads.
• Cgroup bandwidth uses the target task group's cfs_rq on the episode's current CPU. This identifies leaf task-group quota throttling, not yet the full ancestor hierarchy.
• Migration attribution means the observed interval from a target migration until its next relevant switch. It is not a counterfactual claim that another CPU would have run the task sooner.
• The sched_entity.vlag union member is captured as a raw diagnostic snapshot, but is not used as the eligibility verdict for an on-rq entity.
• Scheduler hierarchy capture is capped at eight entities. A deeper chain is marked EEVDF_HIERARCHY_TRUNCATED; unavailable hooks or fields remain explicitly unresolved.
• Cgroup paths are presentation metadata resolved from /sys/fs/cgroup inode IDs. A replay on another host can still explain the numeric hierarchy even when the original paths no longer exist.
• CPU affinity remains an observed migration/placement condition, not a counterfactual claim that a different allowed CPU would have run sooner.
• Repeated wakeups while an episode is already active are coalesced into the original episode.
• TID matching checks every level of task_struct.thread_pid->numbers[], so a TID supplied from a container or WSL PID view can still match scheduler events.

make test runs deterministic accounting cases for the mutually exclusive cause priority, including the two EEVDF buckets and exact time conservation. It also replays synthetic hierarchy, throttle, softirq, and higher-class event streams and checks the numeric why evidence and final accounting without root.