Skip to content

runbooks: make issue severity a first-class impact axis, require self-contained citations#317

Merged
Savid merged 1 commit into
masterfrom
runbooks/issue-severity-axis-self-contained-citations
Jul 21, 2026
Merged

runbooks: make issue severity a first-class impact axis, require self-contained citations#317
Savid merged 1 commit into
masterfrom
runbooks/issue-severity-axis-self-contained-citations

Conversation

@Savid

@Savid Savid commented Jul 21, 2026

Copy link
Copy Markdown
Member

Summary

Formalizes classification.severity (critical|major|minor) on the shared devnet issue contract as an impact rank — a distinct axis from confidence (evidence strength) — and threads it consistently through the investigation pipeline. Also tightens evidence-citation discipline so every ref is self-contained and re-runnable.

Changes

  • devnet_issue_contract.md — owns the severity scale with per-level criteria; adds severity to the classification example and Field rules; clarifies first_bad value/at must be a concrete coordinate (never prose), with guidance for gradual/monotonic onset; requires ref to be executable and self-contained.
  • devnet_issue_collation.md — adds a "rank impact" step that re-judges an incoming severity onto the scale rather than copying it verbatim, and emits within-expected issues as minor instead of dropping them; self-check enforces it.
  • devnet_bug_report.md — distinguishes the board's severity thresholds from the contract-owned scale, and documents that the top-level severity is a copy of issue.classification.severity that must stay in sync (the renderer ranks/filters on the top-level field).
  • evidence_discipline.md — requires each citation to be self-contained (no "same query as above"), allowing a placeholder like {dora} only when the same citation says how to resolve it.

Docs-only change (runbooks); no code paths affected.

🤖 Generated with Claude Code

…-contained citations

Formalize classification.severity (critical|major|minor) on the shared issue
contract as an impact rank distinct from confidence (evidence strength), and
thread it through the pipeline:

- devnet_issue_contract: own the severity scale with per-level criteria; clarify
  first_bad must be a concrete coordinate (not prose) and require self-contained,
  re-runnable refs.
- devnet_issue_collation: add a "rank impact" step that re-judges an incoming
  severity onto the scale rather than copying it, and emits within-expected
  issues as minor instead of dropping them; self-check enforces it.
- devnet_bug_report: distinguish the board's severity thresholds from the
  contract-owned scale, and keep the top-level severity copy in sync with
  issue.classification.severity (the renderer ranks on the top-level field).
- evidence_discipline: require each citation to be self-contained (no
  "same query as above"), with a placeholder allowed only when the ref also
  says how to resolve it.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown
Contributor

🐼 Smoke eval — f377b18: ✅ 8/8 pass

📊 Interactive report — tokens p50 16,536 · tokens/solve 21,349.

Reference points: master@2042fdc 100% · v0.38.6 96%.

question result tokens tools
forky_node_coverage 16,027 5
tracoor_node_coverage 19,036 11
mainnet_block_arrival_p50 16,819 10
list_datasources 14,257 4
block_count_24h 16,252 8
missed_slots_24h 15,723 6
chartkit_default_arrival_distribution 47,508 17
storage_upload_session_scoped 25,167 14
🔭 Langfuse traces (8 runs; ⚠️ = failed)

The report walks this branch's commits against the master baseline and the most recent release. A self-contained copy is in the run's eval-smoke-* artifact.

@Savid
Savid merged commit 1f61a26 into master Jul 21, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants