Epic: Explanation quality — measure it before extending it

[leo-aa88]

Context

A full-codebase review (2026-08-26) found that raglogs has built a substantial production surface — auth, rate limiting, retention, webhooks, OTel, 5 adapters, scope isolation, 39 env vars — around an explanation engine that has never been measured on anything but a single 404-line synthetic incident (sample_data/sample_incident/).

The three structural problems, in order:

1. No way to know if the output is correct. 40 unit test files, ~11k lines of tests, and none of them assert that an explanation is right. There is no eval corpus, no accuracy metric, and no baseline to compare against.
2. The flagship output is fitted to the demo fixture. The words checkout, Webhook queue, and the Stripe event prefix evt_ are literals inside the analysis engine. On anyone else's logs the narrative degrades to a generic template while the README advertises the polished demo output.
3. CI enforces none of the project's own rules. The workflow is the unmodified GitHub Python starter template. make lint fails on main right now (make lint fails on main with 24 ruff errors (contract says it must pass) #68).

Ordering rationale

Measurement comes before change. Every P2 item modifies the core analysis engine, and there is currently no way to tell whether such a change improves or regresses output quality — so P1 must land first, or P2 is guesswork.

Tracked issues

P0 — unblocks everything else

• make lint fails on main with 24 ruff errors (contract says it must pass) #68 — make lint fails on main with 24 ruff errors (already filed; blocks CI enforces nothing: replace the starter workflow with real gates (ruff, deps, Postgres integration tests) #75)
• CI enforces nothing: replace the starter workflow with real gates (ruff, deps, Postgres integration tests) #75 — CI enforces nothing: replace the starter workflow with real gates
• Trigger search reads 5000 arbitrary rows (LIMIT with no ORDER BY) — non-deterministic explanations #76 — Trigger search reads 5000 arbitrary rows (LIMIT with no ORDER BY) — non-deterministic explanations (standalone, no blockers)

P1 — measurement foundation

• Build the eval harness: case format, accuracy metrics, and a trivial baseline arm #77 — Build the eval harness: case format, accuracy metrics, trivial baseline arm (blocked by CI enforces nothing: replace the starter workflow with real gates (ruff, deps, Postgres integration tests) #75)
• Add RCAEval RE2/RE3 as a labeled eval corpus (MIT, 360 cases with root-cause + injection-time labels) #78 — Add RCAEval RE2/RE3 as a labeled corpus — MIT, 360 cases with root-cause + injection-time labels (blocked by Build the eval harness: case format, accuracy metrics, and a trivial baseline arm #77)
• Generate a deploy-class incident corpus from the OpenTelemetry Demo + Chaos Mesh #79 — Generate a deploy-class incident corpus from the OpenTelemetry Demo + Chaos Mesh (blocked by Build the eval harness: case format, accuracy metrics, and a trivial baseline arm #77)
• Benchmark normalization/fingerprinting against Loghub-2.0 template ground truth #80 — Benchmark normalization/fingerprinting against Loghub-2.0 template ground truth (blocked by Build the eval harness: case format, accuracy metrics, and a trivial baseline arm #77)

P2 — core quality

• Remove demo-fitted string literals from the analysis engine ('checkout', 'Webhook queue', 'evt_') #81 — Remove demo-fitted string literals from the analysis engine (blocked by Build the eval harness: case format, accuracy metrics, and a trivial baseline arm #77, Add RCAEval RE2/RE3 as a labeled eval corpus (MIT, 360 cases with root-cause + injection-time labels) #78)
• Redesign trigger detection: rare-event correlation and service linkage, not 12 regexes + a 30-minute window #82 — Redesign trigger detection: rare-event correlation and service linkage (blocked by Build the eval harness: case format, accuracy metrics, and a trivial baseline arm #77, Add RCAEval RE2/RE3 as a labeled eval corpus (MIT, 360 cases with root-cause + injection-time labels) #78)
• Calibrate confidence scoring against measured accuracy (or stop publishing a fake probability) #83 — Calibrate confidence scoring against measured accuracy (blocked by Build the eval harness: case format, accuracy metrics, and a trivial baseline arm #77, Add RCAEval RE2/RE3 as a labeled eval corpus (MIT, 360 cases with root-cause + injection-time labels) #78, Redesign trigger detection: rare-event correlation and service linkage, not 12 regexes + a 30-minute window #82)

P3 — correctness and debt

• EMBEDDINGS_PROVIDER=local is non-functional: Vector(1536) column vs 384/768-dim local models #84 — EMBEDDINGS_PROVIDER=local is non-functional: Vector(1536) vs 384/768-dim models
• No performance baseline: pool sizing, per-row ClusterMember inserts, no partitioning, no load test #85 — No performance baseline: pool sizing, per-row ClusterMember inserts, no partitioning, no load test
• Split the 1,391-line README into docs/ and make the examples match real output #86 — Split the 1,391-line README into docs/ (best done after Remove demo-fitted string literals from the analysis engine ('checkout', 'Webhook queue', 'evt_') #81)
• Repo hygiene: delete other/, stray exported_logs.json, sample_data drift, verify .env history #87 — Repo hygiene: other/, stray exported_logs.json, sample_data drift, verify .env history
• Policy: freeze net-new platform surface until eval shows lift over the baseline #88 — Policy: freeze net-new platform surface until eval shows lift over the baseline

Related bugs already filed

#64 (numeric ID normalization fragments clusters — relevant to #80) · #65 (timeline effect ordering) · #67 (SEVERITY_WEIGHT_* dead config — same pattern as #84) · #70 (compare hides dropped_triggers)

Dependency graph

#68 ──> #75 ──> #77 ──┬──> #78 ──┬──> #81
                      │          ├──> #82 ──> #83
                      ├──> #79 ──┘
                      └──> #80

#76, #84, #85, #87   (independent, any time)
#86                  (after #81)
#88                  (policy decision, any time)

Guiding principle

Every P2 change should be accompanied by a delta in the eval numbers, reported as lift over a trivial baseline ("most frequent error cluster in window"), not as an absolute score.

There is a live critique in the RCA literature (arXiv:2510.04711) that simple rule-based methods match or beat state-of-the-art on four widely used public benchmarks — so absolute scores on these corpora mean little. If raglogs' lift over GROUP BY fingerprint ORDER BY count is near zero, that is the finding, and it is far better to learn it in week one than after another six subsystems.

[leo-aa88]

Status reconciliation (2026-09-22)

This umbrella has largely delivered, and the project has moved past the state it describes. Current standing of the tracked work:

• P0 (unblockers): done — CI gates, lint on main, deterministic trigger search all landed.
• P1 (measurement foundation): done — the eval harness (Build the eval harness: case format, accuracy metrics, and a trivial baseline arm #77), RCAEval RE2/RE3 (Add RCAEval RE2/RE3 as a labeled eval corpus (MIT, 360 cases with root-cause + injection-time labels) #78), the OTel Demo corpus (Generate a deploy-class incident corpus from the OpenTelemetry Demo + Chaos Mesh #79), and Loghub normalization (Benchmark normalization/fingerprinting against Loghub-2.0 template ground truth #80) all exist; "measure before extend" is now the enforced discipline (every src/core/ change states an eval delta).
• P2 (core quality): Remove demo-fitted string literals from the analysis engine ('checkout', 'Webhook queue', 'evt_') #81 (demo literals removed) and Calibrate confidence scoring against measured accuracy (or stop publishing a fake probability) #83 (calibrated confidence) closed. Redesign trigger detection: rare-event correlation and service linkage, not 12 regexes + a 30-minute window #82 remains the one substantial live core-quality item — its rare-event + linkage mechanism is implemented and measured, but default promotion is deferred pending real per-edge telemetry grounding (see below).
• P3+ debt: resolved/closed as encountered.

Since this epic was filed, the analysis engine grew a full deterministic causal-inference layer (#177, phases A–I, now closed as implemented with empirical limitations). The measured result relocated the bottleneck to grounding trustworthy observables from real telemetry, which is now its own epic:

➡️ #209 — EPIC: Real-telemetry causal observable grounding.

Reconciled live children: #82 (trigger detector, deferred → unblocked by #209) and #88 (the standing freeze policy — a gate, not a task). Everything else this umbrella tracked is closed. Leaving #74 open only as the index for #82/#88; the active quality frontier is #209.