runbooks: make issue severity a first-class impact axis, require self-contained citations#317
Merged
Savid merged 1 commit intoJul 21, 2026
Conversation
…-contained citations Formalize classification.severity (critical|major|minor) on the shared issue contract as an impact rank distinct from confidence (evidence strength), and thread it through the pipeline: - devnet_issue_contract: own the severity scale with per-level criteria; clarify first_bad must be a concrete coordinate (not prose) and require self-contained, re-runnable refs. - devnet_issue_collation: add a "rank impact" step that re-judges an incoming severity onto the scale rather than copying it, and emits within-expected issues as minor instead of dropping them; self-check enforces it. - devnet_bug_report: distinguish the board's severity thresholds from the contract-owned scale, and keep the top-level severity copy in sync with issue.classification.severity (the renderer ranks on the top-level field). - evidence_discipline: require each citation to be self-contained (no "same query as above"), with a placeholder allowed only when the ref also says how to resolve it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
samcm
approved these changes
Jul 21, 2026
Contributor
🐼 Smoke eval —
|
| question | result | tokens | tools |
|---|---|---|---|
forky_node_coverage |
✅ | 16,027 | 5 |
tracoor_node_coverage |
✅ | 19,036 | 11 |
mainnet_block_arrival_p50 |
✅ | 16,819 | 10 |
list_datasources |
✅ | 14,257 | 4 |
block_count_24h |
✅ | 16,252 | 8 |
missed_slots_24h |
✅ | 15,723 | 6 |
chartkit_default_arrival_distribution |
✅ | 47,508 | 17 |
storage_upload_session_scoped |
✅ | 25,167 | 14 |
🔭 Langfuse traces (8 runs; ⚠️ = failed)
The report walks this branch's commits against the master baseline and the most recent release. A self-contained copy is in the run's eval-smoke-* artifact.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Formalizes
classification.severity(critical|major|minor) on the shared devnet issue contract as an impact rank — a distinct axis fromconfidence(evidence strength) — and threads it consistently through the investigation pipeline. Also tightens evidence-citation discipline so everyrefis self-contained and re-runnable.Changes
devnet_issue_contract.md— owns the severity scale with per-level criteria; addsseverityto the classification example and Field rules; clarifiesfirst_badvalue/atmust be a concrete coordinate (never prose), with guidance for gradual/monotonic onset; requiresrefto be executable and self-contained.devnet_issue_collation.md— adds a "rank impact" step that re-judges an incoming severity onto the scale rather than copying it verbatim, and emits within-expected issues asminorinstead of dropping them; self-check enforces it.devnet_bug_report.md— distinguishes the board's severity thresholds from the contract-owned scale, and documents that the top-levelseverityis a copy ofissue.classification.severitythat must stay in sync (the renderer ranks/filters on the top-level field).evidence_discipline.md— requires each citation to be self-contained (no "same query as above"), allowing a placeholder like{dora}only when the same citation says how to resolve it.Docs-only change (runbooks); no code paths affected.
🤖 Generated with Claude Code