Skip to content

fix(cli): distinguish degraded judge/synthesis runs from a clean pass (DSE-901) - #60

Merged
ernestprovo23 merged 2 commits into
mainfrom
ernest/dse-901-wave4
Aug 1, 2026
Merged

fix(cli): distinguish degraded judge/synthesis runs from a clean pass (DSE-901)#60
ernestprovo23 merged 2 commits into
mainfrom
ernest/dse-901-wave4

Conversation

@ernestprovo23

Copy link
Copy Markdown
Member

Summary

  • Bug (DSE-901, High): adversarial (and synthesize/debate) mode exited 0 even when the judge/synthesizer call failed while members still answered. A verification gate that only checks the exit code (e.g. /conclave-verify) reads a partial run as a full pass. Reproduced live on 2026-07-25: an Anthropic credit failure took out the claude judge, 4/5 members still answered, adversarial.verdict came back null, and the CLI exited 0.
  • Fix (ticket's suggested option 1, the conservative/backward-compatible one): CouncilResult gains a computed degraded field — True when synthesis_error or adversarial.verdict_error is non-None — serialized as a top-level key in every --json payload. The CLI now exits a new, distinct code 3 for a degraded run. Existing exit codes are unchanged: 0 = clean pass, 1 = hard failure (zero usable member answers, or Elite not ready), 2 = usage/config error. 3 was chosen (not the ticket's example of 2) specifically to avoid colliding with the existing usage-error meaning of 2.
  • Explicitly out of scope (per ticket): judge/synthesizer failover to another provider. Noted as a follow-up in the CHANGELOG.
  • Docs updated: the CLI's own exit-code docstring (the primary source of truth), DOCUMENTATION_INDEX.md's CLI-tests row, tests/test_cli.py's module docstring, and a CHANGELOG.md entry. No full contract rewrite — DSE-752 owns that separately.

Files changed

  • src/conclave/models.pyCouncilResult.degraded (Pydantic computed_field)
  • src/conclave/cli.py_DEGRADED_EXIT_CODE = 3; wired into the --json, human-render, and --stream exit paths (in that priority order: hard failure > degraded > clean)
  • tests/test_cli.py, tests/test_synthesizer.py — new tests (see below)
  • CHANGELOG.md, DOCUMENTATION_INDEX.md — exit-code contract docs

Release classification

~/.claude/scripts/release_control.py classify returned:

{"classification": "security-specific", "reasons": ["security-control behavior changed"], "classifier_version": "dse-release-control-v1"}

Do not merge without a human receipt (per rules/release-control.md).

Manual read of the reasons: the deterministic keyword scanner matched generic English words in this diff's docstrings/tests"role" (from if m.get("role") == "system", a message-dict key, not an authz role), "scope" (from "out of scope for this fix"), and "verification" (from "verification-gate consumer"). No auth/session/credential/IAM/tenant-isolation/CORS/rate-limit code path is touched. That said, per the binding rule the classifier's output is authoritative over agent judgment, so this is flagged as-is rather than self-cleared — a human should confirm before merge.

Test plan

  • pytest — full suite: 733 passed, 91.17% coverage (CI gate is 75%)
  • ruff check . — all checks passed
  • ruff format --check — the 4 touched Python files are formatted (README.md has a pre-existing, unrelated formatting drift not touched by this PR)
  • New tests added:
    • tests/test_cli.py: test_adversarial_judge_failure_json_is_degraded_not_clean, test_adversarial_judge_failure_human_exits_degraded_code (mock the judge call to raise, mirroring the live incident), test_adversarial_clean_pass_still_exits_zero (regression: full success still exits 0), test_all_members_failed_synthesize_mode_exits_one_not_degraded (regression: hard failure still exits 1 even though degraded also reads True there — priority order verified)
    • tests/test_synthesizer.py: unit pins for the degraded computed field (test_degraded_false_by_default, test_degraded_true_when_synthesis_error_set, test_degraded_true_when_only_adversarial_verdict_error_set, test_degraded_false_on_clean_synthesis, test_degraded_is_a_top_level_json_key), plus assert result.degraded is True added to the existing judge-failure/unkeyed-judge/synthesizer-failure tests

🤖 Generated with Claude Code

… (DSE-901)

Adversarial (and synthesize/debate) mode exited 0 even when the judge/
synthesizer call failed while members still answered -- a verification
gate reading only the exit code (e.g. /conclave-verify) could not tell
a partial run from a clean pass. This reproduced live on 2026-07-25: an
Anthropic credit failure took out the claude judge while 4/5 members
answered, adversarial.verdict came back null, and the CLI exited 0.

CouncilResult gains a computed `degraded` field (True when synthesis_error
or adversarial.verdict_error is set), serialized as a top-level JSON key.
The CLI now exits a distinct code 3 for a degraded run, keeping the
existing 0 (clean)/1 (hard failure: zero usable answers or Elite not
ready)/2 (usage error) meanings unchanged. Judge/synthesizer failover to
another provider is explicitly out of scope for this fix (follow-up).
ernestprovo23 added a commit that referenced this pull request Aug 1, 2026
Pre-existing formatting drift (inline-comment spacing) unrelated to
any specific PR's diff — confirmed byte-identical to origin/main
before this fix, so every PR touching this repo inherits a failing
ruff-format check regardless of what it actually changes. Surfaced
while merging PR #60 (DSE-901).
@ernestprovo23
ernestprovo23 merged commit facb36c into main Aug 1, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant