EPIC: Real-telemetry causal observable grounding — make OTel evidence usable by the structural engine

[leo-aa88]

Why this epic exists

The #177 epic (Causal diagnostic inference under partial observability) is implemented: Observable → Hypothesis → ObservationModel → structural partition → outcome → class scoring → presentation are all merged, and the structural result is exposed opt-in in the CLI/API (--structural, PR #207). Its empirical result is clear, and it moved the bottleneck:

• On trace-loc the structural bridge retains the truth extremely well — ~100% candidate recall, ~2.4 candidates/case — but 0% unique identification.
• On the real OTel corpus the structural adapter often cannot generate candidates at all.

So the limiting factor is no longer partitioning, scoring, or another heuristic. It is grounding trustworthy observables from real telemetry. This epic makes real OTel evidence legible to the structural engine.

This is inside the existing freeze carve-out (AGENTS.md: "metrics/traces ingestion for causal localization" is unfrozen). It is analysis-core grounding, not net-new platform surface.

The goal (the strategic shift)

Optimize for "turn a huge telemetry mess into a compact, trustworthy diagnostic packet without throwing away the true cause" — not "improve top-1." "Truth retained among 2–3 evidence-backed hypotheses" is already a useful result. Headline metrics are candidate recall + selectivity + outcome distribution, never top-1.

What the data actually shows (grounding the design)

Checked the real otel-fresh spans directly:

• caller→callee is already available — spans carry parent_span_id; call_edges / build_service_graph derive the graph from it. The graph is incomplete (not every path is traced — this is why Redesign trigger detection: rare-event correlation and service linkage, not 12 regexes + a 30-minute window #82's linkage dropped ~44% of positives), not absent.
• Observables are metric-only today — the structural adapter derives sig:{service} from error_rate / latency_ms metrics, so a service with spans but no metrics is UNKNOWN → "can't generate candidates."
• Spans are latency-rich, error-sparse — an incident case had 52k spans, 96% with duration_ms, only 24 with error-status. Latency is a strong per-span signal we largely ignore for observables.
• We drop the richest evidence at ingest — raw OTLP carries span kind / peer.service / per-edge status; our ingestion keeps only service/operation/duration/status and leaves TraceSpan.attributes null. That per-edge evidence is exactly what Phase F needed for sound absence attribution.

Milestones

M1 (first, deliberately narrow): span-derived observables — get a selective candidate set on real OTel without inventing evidence

Derive per-service observables from real OTLP spans — latency-first, plus sparse error-status where actually present — without converting missing telemetry into ABSENT (preserve UNKNOWN). Run the existing structural bridge against the already-spent OTel dev corpus and report candidate recall + selectivity + outcome distribution. No ranking changes, no new causal heuristics, no default promotion.

Success criteria (frozen before looking):

• truth retained ≥ 90%
• candidate set materially smaller than service enumeration
• UNKNOWN discipline preserved (missing measurement never becomes ABSENT)
• invented evidence: 0

M2 (later): preserve OTLP semantic attributes through ingestion

Carry span kind / peer.service / per-edge status into TraceSpan.attributes so per-edge causal evidence exists — the signal Phase F's sound absence attribution and #82's complete linkage both need. Measured against the same frozen gate.

M3 (later): re-run the structural experiment on a fresh frozen real OTel corpus

Freeze a new real corpus before looking. Report candidate recall, median candidate-set size / fraction of services, healthy abstention, outcome distribution, causal-kind coverage. Mostly-UNCERTAIN is acceptable; the question is whether the packet narrowed the search space without deleting the truth.

Explicitly NOT in scope

No Bayesian layer, no new reranker, no internal agent loop, no MCP server, no more synthetic fault families. The evidence says the highest-value missing component is the boring, load-bearing one: make real telemetry legible to the causal model.

Relationship to other issues

• Continues EPIC: Causal diagnostic inference under partial observability #177 (which closes as implemented-with-empirical-limitations, pointing here).
• Unblocks the deferred debt: sound causal absence attribution (Phase F), promoting --structural from opt-in to default, and Redesign trigger detection: rare-event correlation and service linkage, not 12 regexes + a 30-minute window #82's regex-detector replacement — all were deferred pending exactly this real per-edge grounding.

[leo-aa88]

M1 — frozen measurement protocol (recorded before implementing / looking)

Freezing the protocol before writing the span-derived observable, per the epic's discipline.

Corpus: otel-fresh (the existing, already-spent OTel dev corpus) — 9 labeled-positive incident cases + 12 healthy negatives. (M3 will freeze a new, unseen corpus; M1 deliberately uses the spent one.)

Arms (same structural bridge, only the observable source differs):

• baseline — current metric-only sig:{service} (summarize_metrics): error_rate / latency_ms metrics.
• treatment — sig:{service} derived from spans (latency-first via duration_ms, plus sparse error-status via status_code), combined with the metric sig (PRESENT > ABSENT > UNKNOWN priority; a branch is only "measured" when the data is actually present).

Metrics (frozen definitions):

• generation rate (positives): fraction of positive cases where the adapter returns ≥1 candidate (not abstain) — attacks H2's "can't generate candidates".
• candidate recall / truth retained (positives): fraction of positives whose labeled root-cause service is in the structural candidate set. Reported two ways: over all positives, and over positives whose cause service actually has telemetry (spans or metrics) in-window.
• selectivity: median candidate-set size, and median candidate fraction = |candidates| / |services seen in the case|. Want ≪ 1.0 (not enumeration).
• healthy behavior (negatives): outcome distribution + fraction producing no candidate (abstention).
• outcome distribution across all cases (IDENTIFIED / NON_IDENTIFIABLE / IRREDUCIBLE / UNCERTAIN / NO_COMPATIBLE).
• invented evidence: 0 — enforced as a code property + unit test: a service with no measured branch stays UNKNOWN, never fabricated to ABSENT/PRESENT.

Frozen pass gate (from the epic body):

1. truth retained ≥ 90% (on positives whose cause is telemetry-observable).
2. median candidate fraction materially below 1.0 (a real narrowing, not enumeration).
3. UNKNOWN discipline preserved (invented evidence = 0).
4. no ranking changes, no new causal heuristics, no default promotion.

Report baseline vs treatment side by side against this gate. Landing is conditional on the gate; "treatment doesn't help" is an acceptable, publishable outcome.

[leo-aa88]

M1 result (PR #210) — frozen gate NOT met on recall; landing as capability

Ran the frozen protocol above. Generation 0→6/9, selectivity 0.17 (median candidate fraction), UNKNOWN discipline clean, invented evidence 0 — the primary M1 goal (a selective candidate set on real OTel without inventing evidence) is achieved. But truth retained is 3/8 (38%), below the ≥90% gate.

Miss taxonomy (all 6 misses): 5 = locally-silent cause (the cause's own spans are measured-NORMAL — no service-local anomaly), 1 = no telemetry for the cause (kafka). Zero M1 / generation / availability bugs. A service-local observable cannot retain a locally-silent cause; that needs per-edge evidence (M2).

Per the epic's bar (a primitive is worth landing when it is correct about the evidence it represents), M1 lands as a sound capability with the recall gate explicitly unmet — it proves service-level grounding works where service-level evidence exists, and pins down exactly why M2 (preserve per-edge OTLP attributes) is the next lever.

[leo-aa88]

M1 result — corrected after #210 review

The M1 numbers posted above were inflated by two observable bugs that the #210 review caught. Unset span status was counted as a measured success, which fabricated OBSERVED-ABSENT for five causes. And baseline-only call edges leaked into incident candidate generation. Both are fixed (explicit-status error branch, incident-only edges), along with the merge record and a median latency reducer. The shadow eval and the product path now share one builder.

Re-measured on otel-fresh with the same frozen protocol:

• generation 0/9 → 5/9 (was reported 6/9)
• truth retained 3/8 (38%), unchanged; the ≥90% gate is still not met
• selectivity: median candidate fraction 0.33 (was reported 0.17); still ≪1.0
• healthy abstention 11/12 (was reported 10/12)
• 0 causes read as ABSENT (was 5) — they are now correctly UNKNOWN

Miss taxonomy is unchanged in kind: 5 locally silent causes plus 1 with no telemetry, and no bugs in the observable, availability or generation categories. paymentUnreachable was previously retained only through the baseline-edge leak. An unreachable service is absent from the incident call graph, so recovering it soundly is an M2 (per-edge evidence) problem.

[leo-aa88]

M2a — feasibility finding + frozen protocol (recorded before implementing)

Feasibility: per-edge evidence already exists in the spans we ingest

The original M2 assumed per-edge evidence needed the converter to keep span kind / peer.service, plus a fresh capture. A survey of the otel-fresh case files says it doesn't. A caller's client span for a call carries its own explicit status, and the callee it targets can be learned from observed parent→child relations rather than guessed from names. paymentUnreachable is the clearest case: payment emits no incident spans, yet 10/10 of checkout's oteldemo.PaymentService/Charge spans are ERROR with no payment child, while in the baseline the same operation had payment children.

Trace-only ceiling (survey, spent corpus — feasibility only, not tuning): edge evidence toward the true cause exists for 4/9 positives: cart, payment ×2, recommendation. Combined with M1, truth retention can reach at most 4/8 (50%); the only newly recoverable case is the unreachable callee. The other misses leave no trace-side signal: ad (CPU saturation), frontend ×2 (entry service, no instrumented caller showing an anomaly), product-catalog (no failing calls toward it), and kafka (no spans). They need resource, queue or traffic metrics. So ≥90% is not reachable from traces on this corpus, and M2a is measured as a ceiling, not a pass/fail gate. Specificity risk, pre-registered: 5/12 healthy windows have at least one edge tripping the error-rate or ≥2× latency cutoff.

Frozen semantics

Observable edge:{caller}->{callee} — a separate coordinate, never folded into sig:{callee}.

• Callee identity: a (caller_service, operation) pair maps to callee S only if ≥90% of that operation's observed cross-service children are in S. Otherwise it's ambiguous: no mapping, no evidence.
• Error branch: over the caller's incident spans of mapped operations, ERROR / (OK + ERROR), with explicit statuses only (UNSET excluded). Measured iff at least one explicit status exists.
• Latency branch: median incident duration over median baseline duration of those spans. Measured iff both exist and the baseline median is > 0.
• Three-valued: PRESENT if a measured branch is anomalous (rate ≥ 5%, ratio ≥ 2×); ABSENT only if both branches are measured-and-normal; else UNKNOWN (no observable). An edge with no incident calls produces no observable — absence of calls is not evidence.

Generation and model:

• The callee of a PRESENT edge joins the candidate set.
• process:{S} keeps expecting sig:{S} = PRESENT, and additionally expects edge:{C}->{S} = PRESENT for each edge toward S that has an edge signal.
• Soft expectations only: no hard rules, no ranking change, no default promotion.

Measurement (otel-fresh, same corpus/protocol as M1)

Arms: M1 (structural_signals without edge observables) vs M2a (with). Report generation, truth retained (all / cause-has-telemetry), selectivity (median candidate set and fraction), healthy abstention, outcome distribution, and a per-case recovery table for M1's six misses.

Landing criteria (frozen):

1. Semantics are sound and tested (invented evidence 0; UNKNOWN discipline).
2. Truth retention does not decrease vs M1.
3. Any change in healthy abstention or selectivity is reported, not hidden.

The result is the measured ceiling. If it plateaus near 4/8, that is evidence that the remaining gap belongs to metric/resource semantics (M2b), not the trace model.

[leo-aa88]

M2a result (PR #211) — trace-only ceiling measured

Frozen protocol above, on otel-fresh:

• generation 5/9 → 8/9
• truth retained 3/8 → 4/8, exactly the ceiling the survey predicted; paymentUnreachable recovered
• selectivity 0.33 → 0.14
• healthy abstention 11/12 → 7/12 (a specificity cost)

IDENTIFIED precision is 0/8. Every IDENTIFIED outcome is wrong: 5 on healthy windows and 3 naming the wrong service. All 4 retained truths are UNCERTAIN outcomes.

The healthy false edges are all latency-only: every status is UNSET, and median client-span latency swings 2.1–4.3× on normal traffic. So the ≥2× cutoff inherited from the metric model isn't specific per edge. It is pre-registered for evaluation on M3's fresh corpus, not tuned here.

Ceiling conclusion: trace evidence plateaus at 4/8 on this corpus. The remaining misses — ad (CPU), frontend ×2 (entry traffic), product-catalog, kafka (queue) — leave no trace-side signal and need metric/resource semantics (M2b).

[leo-aa88]

M2b — frozen protocol: utilization observables (recorded before implementing)

Why (from the M2a ceiling survey)

Trace evidence plateaus at 4/8. For 2 of the 4 remaining misses, attributable metric evidence exists and is cleanly separated from healthy windows:

signal	healthy max	incident
ad jvm.cpu.recent_utilization	1.31×	adHighCpu ~1100×
frontend nodejs.eventloop.utilization	1.06×	loadGeneratorFlood 2.47×

imageSlowLoad, product-catalog and kafka have no usable signal. For product-catalog, only otel.sdk.* self-telemetry moves (~1.8×): a false-positive trap, not evidence. Pre-registered expectation: ~6/8 truth retained on otel-fresh.

Frozen semantics

Observable family util:{service}:{resource} — its own coordinates, never folded into sig:{service}. Metrics are classified by OpenTelemetry semantic-convention name shape, not by names picked from this corpus:

• Utilization gauges — the last dotted segment ends in utilization. Resource = the segment before it: jvm.cpu.recent_utilization/system.cpu.utilization/container.cpu.utilization → cpu; nodejs.eventloop.utilization → eventloop; system.memory.utilization → memory. Statistic: incident mean / baseline mean. Values must be finite and ≥ 0; measured iff both windows have samples and the baseline mean is > 0.
• CPU-time counters — names ending .cpu.time (semconv cumulative seconds). Resource = cpu. Statistic: incident rate / baseline rate, where rate = (last − first) / Δt. Only for a verifiably single-series, monotonic stream: at most one sample per timestamp, never decreasing. Otherwise it's UNKNOWN. We checked this before freezing: process.cpu.time for recommendation has 2 samples/timestamp and 7 decreases (interleaved state= series flattened by the converter), while jvm.cpu.time for ad is a single monotonic series.
• Excluded as self-telemetry: otel.sdk.*, otelcol*.
• Unattributed samples (service = None) contribute nothing, and are counted in the attribution report.

Three-valued, per metric then per class:

• A metric is PRESENT if its ratio ≥ 2× (discretize_ratio HIGH), ABSENT if measured and < 2×, else UNKNOWN.
• The (service, resource) observable takes the witness: PRESENT if any metric in the class is PRESENT, else ABSENT if any is measured-normal, else UNKNOWN. UNKNOWN produces no observable.
• A utilization drop is ABSENT for a saturation observable, not an anomaly.

Generation and model:

• A service with a PRESENT util: observable joins the candidate set.
• process:{S} additionally expects each util:{S}:{r} that has a signal to be PRESENT.
• Soft only: no hard rules, no ranking change, no default promotion. Without util signals, the M2a model is byte-identical.

Measurement (otel-fresh, same protocol)

Arms M2a vs M2a+M2b. Report generation, truth retained, selectivity, healthy abstention, outcome distribution, IDENTIFIED precision (0/8 at M2a), a per-case recovery table, and an attribution-quality report: the share of utilization/CPU samples that are attributable, and which classes are lost to service=None or to interleaved series.

Landing criteria (frozen):

1. Semantics sound and tested.
2. Truth retention not decreased vs M2a.
3. Abstention, selectivity and IDENTIFIED-precision changes reported, not hidden.

M3 then validates the whole model (M1+M2a+M2b) out of sample.

[leo-aa88]

M2b result (PR #212) — utilization observables

Frozen protocol above, on otel-fresh:

• truth retained 4/8 → 6/8 (75%), matching the pre-registered ~6/8. adHighCpu recovered via util:ad:cpu, loadGeneratorFlood via util:frontend:eventloop.
• healthy abstention unchanged at 7/12 (util adds no healthy false positives)
• selectivity 0.14 → 0.19
• IDENTIFIED precision still 0/7; the healthy ones come from M2a's latency-only edges, pre-registered for M3

Attribution quality: only 9% (3,514 / 37,282) of utilization and CPU-time samples are attributable to a service. The two recoveries survived because JVM and Node runtime metrics carry service.name; container.* and process.cpu.time mostly arrive with service=None. So attribution materially caps this primitive. The converter fix (resource attributes → service, plus metric_type) should come with M3's fresh capture.

Model decomposition so far: traces recover topology and failure propagation; utilization metrics recover locally silent resource faults; imageSlowLoad, product-catalog and kafka are unobservable in the captured telemetry. Next is M3: validate M1 + M2a + M2b out of sample.

[leo-aa88]

M2b result — corrected after #212 review (6/8 withdrawn)

The #212 review found that no metric sample in the pipeline carried series identity:

• per-state gauges (system.cpu.utilization by cpu.mode) were averaged into a false measured-normal;
• on the product path, metric_pk silently dropped a second series at the same timestamp.

Fixed end to end. The converter records datapoint attributes plus service.instance.id, the row key includes them, and jsonl round-trips them. Utilization is measured only on one verified series with a matching declared metric_type; everything else is UNKNOWN.

Consequence: otel-fresh has no series identity (0 of 37,282 utilization/CPU samples), so M2b measures nothing on it. Its numbers are identical to M2a (4/8, abstention 7/12, IDENTIFIED 0/8). The earlier 6/8 is withdrawn: it rested on an unverified single-series assumption.

M2b becomes testable on M3, which now has a hard requirement: capture with the identity-carrying converter.

[leo-aa88]

M3 — frozen protocol: out-of-sample generalization test (recorded before any capture exists)

M3 is the genuine generalization test for the structural model built in M1/M2a/M2b. Everything below is fixed before the corpus is captured. Nothing in the model is changed after results are seen.

Sequence

1. This protocol (frozen now).
2. Capture-tooling safety guard. The capture must never run kubectl against an implicit context. An explicit --kube-context is required, and prod/live-looking contexts are refused.
3. The M3 evaluator is committed and reviewed before the capture runs. It generalizes the M1/M2 measurement scripts, so the measurement can't be shaped by the data.
4. Fresh capture on a dedicated non-prod Kubernetes cluster (maintainer-run), using the identity-carrying converter (post-Utilization observables for the structural model (#209 M2b) #212).
5. The evaluator runs once, on the frozen model, over the captured case directories. Results are posted here as-is.

Frozen model

M1 + M2a + M2b exactly as merged, via structural_signals: span sig, per-edge edge: observables and util: observables with series identity. The evaluation records the commit SHA. No semantic change after the data is seen. That includes the ≥2× edge-latency cutoff, which stays as-is even though it's known to be weak (below).

Corpus

OTel Demo on k8s, captured with scripts/eval/otel_corpus.py:

• all built-in flag scenarios (productCatalogFailure, paymentServiceFailure, cartServiceFailure, recommendationServiceCacheFailure, kafkaQueueProblems, adHighCpu, imageSlowLoad, loadGeneratorFloodHomepage);
• the committed Chaos Mesh scenarios (cartRedisPartition, checkoutPodKill, productCatalogIoFault, recommendationCpuStress);
• ≥12 healthy negatives.

Baseline and post windows use the tooling defaults. I don't inspect case contents before the one-shot evaluator run.

Reported (all of it, regardless of outcome)

• Per arm (M1 / M2a / M2a+M2b): generation, truth retained (all positives and cause-has-telemetry), selectivity (median candidate set and fraction), healthy abstention, outcome distribution, IDENTIFIED precision.
• A per-case table, including which observable family retained each truth: sig / edge / util.
• Series-identity coverage: the share of utilization/CPU samples that name their instance. This is a prerequisite for M2b, not a model result.
• Default-path eval (raglogs eval with the learned ranker + the rare_event trigger path): trigger accuracy, root-cause accuracy, negative precision. This is the pre-registered ranker/relative-magnitude check on identity-carrying data.
• The healthy-window flagged-edge rate at the unchanged ≥2× cutoff: the M2a specificity follow-up.

Pre-registered decision rules (thresholds from otel-fresh, fixed now)

• The structural model generalizes if, on M3:

  • truth retained (cause has telemetry) is ≥ 50% — the trace-only ceiling M2a reached on otel-fresh;
  • the median candidate fraction is ≤ 0.33;
  • healthy abstention is ≥ 58% (7/12 on otel-fresh).

  Otherwise it does not generalize, and that is reported plainly.

• M2b is validated if at least one resource-fault positive (adHighCpu, recommendationCpuStress) is retained via a util: observable on a named-instance series, and no healthy window gets a PRESENT util: observable.

  • If series-identity coverage is ~0, M2b is untestable on M3, not failed.
  • If util fires on a healthy window, M2b is not validated.

• The edge-latency cutoff is reported as-is. Any recalibration is future work designed on a separate DEV split and tested on a new held-out capture (sealed-split tooling), never on M3.

• IDENTIFIED precision is reported as-is (0/8 on otel-fresh). No outcome-label change is made on M3 data.

• Unobservable cases (no telemetry signal for the cause) are listed as such. They are not model failures to be tuned away.

[leo-aa88]

M3 protocol amendment: the selectivity bound is ≤ 1/3 (recorded before any capture)

Amended criterion: median candidate fraction ≤ 1/3. It replaces "≤ 0.33". It is evaluated in exact rational arithmetic: each fraction is |localization| / |services| as a ratio of integers, and the median is taken over those.

Why: the bound is meant to be the structural bar, "at least as selective as M1 on otel-fresh". The committed evaluator (#214) reproduces M1's otel-fresh median fraction as exactly 1/3. "0.33" was that value rounded to two decimals. Taken literally, it fails M1's own reference value and draws an arbitrary line (0.331 passes, 0.334 fails). Exact arithmetic also rules out a float-rounding failure at 1/3 itself.

When: this is recorded before the M3 corpus is captured. No M3 data exists, and none has been looked at. Nothing else in the protocol changes. The other bars stay as frozen:

• truth retained (cause has telemetry) ≥ 0.50;
• healthy abstention (no hypothesis generated) ≥ 0.58;
• the M2b rules;
• the one-shot run.

After capture, no criterion changes.

Procedure, unchanged in substance:

1. This amendment, plus the evaluator constant change (PR to follow).
2. Non-prod k8s capture with --negatives 12.
3. A separate --chaos capture for the Chaos Mesh cases.
4. The corpus directory is frozen, and its sha256 is recorded by the driver.
5. The evaluator runs once, on clean main.
6. Its output is posted here verbatim, whatever it shows.