Skip to content

Commit 43a072b

Browse files
docs(receipt): Stage 2 council dispositions, full-population census transitions
The single seat returned REJECT with four MAJORs; every finding was checked at source. Two were real planner defects (fixed in c8c105c); the other four were evidence-class findings, answered here with artefacts rather than prose: the golden gate is marked NOT MET AS WRITTEN and amended to projected equality checked between named commits; the census attribution now rests on all 5,434 questions through both arms (identical 1,844 / crash 2,349 / whole-question-phrase 1,227 / different 14; zero regressions), with the mechanism read from the frozen escape function itself (1,217 of 1,227) and the instrument difference from run_census stated; the gold parity and rescued-crasher claims are counted per item; the studyloop suite attribution rests on a matched control at 80ee57b in the same sandbox (299 of 305 failing ids in common, identical exception profiles). The four census texts affected by the F2 fix keep their recorded outcomes, so the Stage 2 receipts stand exactly.
1 parent 70ddf2b commit 43a072b

3 files changed

Lines changed: 101 additions & 33 deletions

File tree

.secrets.baseline

Lines changed: 31 additions & 1 deletion
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

docs/architecture/session-memory/receipts/semantic-layer/stage2-census-transitions.json

Lines changed: 1 addition & 0 deletions
Large diffs are not rendered by default.

docs/architecture/session-memory/receipts/semantic-layer/stage2-record-2026-09-11.md

Lines changed: 69 additions & 32 deletions
Original file line numberDiff line numberDiff line change
@@ -24,47 +24,63 @@ An empty result is never silent: `plan="none"` plus a note says nothing was sear
2424
|---|---|---|
2525
| crashes through the real MCP tool | 0/91 | **0/91** (frozen control still 42/91) |
2626
| crashes through the CLI | 0/91 | **0/91** (was 42/91 + zero rows on the other 49) |
27-
| CLI ≡ MCP, ordered session lists | 91/91 | **91/91**, paired delta exactly 0 |
27+
| CLI ≡ MCP, ordered session lists | 91/91 | **91/91** — counted directly over the per-item `ranked` arrays (`stage2-census-transitions.json``gold_counters`), not inferred from equal aggregates |
2828
| K-stratum paired delta vs frozen | not negative | K **0.182 → 0.303**; P 0.034 → 0.034; R 0.103 → 0.138 |
29-
| planner golden | unchanged | **cases byte-identical**; row shape extended additively — see deviation |
29+
| planner golden | unchanged | **NOT MET AS WRITTEN — gate amended** (see below): equal after projecting `message_id` away, checked between `80ee57bb` and `d060d3f2` by `scripts/eval/stage2_council_evidence.py` |
3030
| macro recall@5 (reported, not gated) || **0.1066 → 0.1585**, paired +0.052, CI95 [+0.011, +0.100] (57 clusters) |
3131
| latency || mcp p50 20.8 ms / p95 92 ms (was 18.5 / 94.5) |
3232

33-
Honest detail: of the 42 former crashers only **5** become hits at K=5. The crash class is dead;
33+
Honest detail: of the 42 former crashers only **5** become hits at K=5 (`frozen_crash_and_live_hit_at_5: 5`, counted per item; `moved_outside_crash_class: []`). The crash class is dead;
3434
those questions now have the same ranking problem as everything else. That is Stage 4's problem,
3535
and the K/P/R shape says so: K and R moved, P did not.
3636

37-
## Deviation from the freeze wording — recorded, not hidden
37+
## Gate amendment — the golden gate was changed, not met
3838

39-
The freeze said "golden file unchanged". The golden's `row_keys` gained `"message_id"` and each
40-
case result gained its id; every case's query, ordering and preview is byte-identical (verified
41-
by diff: five added lines, nothing removed). The gate's purpose — the explicit-syntax and simple
42-
cases must not move while the planner changes — held. Real message ids were a stated Stage 2
43-
requirement (the agent's interface had no citation handle and the census could not gate the tool
44-
without them), so the additive change is the requirement, not drift.
39+
The freeze said "golden file unchanged". That literal gate is **not met**: `row_keys` gained
40+
`"message_id"` and each case result gained its id (15 changed lines in the diff stat: the
41+
trailing-comma lines count as changed). The seat was right that labelling this "unchanged" was
42+
wrong. The gate is hereby **amended** to: *the golden is equal to its Stage 1 form after
43+
projecting `message_id` away* — checked between named commits `80ee57bb` and `d060d3f2` by
44+
`scripts/eval/stage2_council_evidence.py` (`golden_projection.equal_after_projecting_message_id_away:
45+
true`, `added_row_key: ["message_id"]`). Why the amendment rather than a waiver: real message ids
46+
were a Stage 2 requirement (the agent's interface had no citation handle; the census could not
47+
gate the tool without them), and the invariant the gate protected — explicit-syntax and simple
48+
cases do not move while the planner changes — is exactly what the projected equality checks.
4549

4650
## Census through the real tool — first time possible
4751

4852
`stage2-census-mcp.json`: 5,434 eligible · hits **3,345** · hit@5 **0.6156** (Stage 1 frozen
4953
0.1305) · crash **0** (was 2,349) · vocabulary 257 · ranking 1,832 (untied 1,546) · tied hits 0.
5054

51-
A jump that size was not accepted on sight. A 400-question paired probe (seed 11, both arms,
52-
same self-exclusion, session lists compared) attributes all of it to two shipped defects and
53-
finds **zero regressions**:
55+
A jump that size was not accepted on sight. The **full population** was re-run through both arms
56+
with identical self-exclusion and the ranked *session lists* compared per question
57+
(`stage2-census-transitions.json`, `per_question` carries every row; 1,834 s):
5458

55-
| class | n | frozen own-session hits | live own-session hits |
59+
| class (frozen → live) | n | frozen own hit | live own hit |
5660
|---|---|---|---|
57-
| identical ranked lists (planner ran in both) | 140 | 54 | 54 |
58-
| frozen **crash** (the and/or/not class) | 189 | 0 | 129 |
59-
| frozen **empty**, live rows — **70 of 71 texts contain a `"`** | 71 | 0 | 57 |
60-
61-
The second defect the gold set could not see: the shipped code treated any question containing
62-
a double-quote character as "explicitly quoted" and searched its *entire text* as one exact
63-
phrase — which matches nothing but the question itself. Learner turns quote error messages,
64-
file names and code constantly (71 of 400 = 18%). The planner now lifts quoted spans out as
65-
phrases and plans the rest.
66-
67-
Sample hit@5 frozen 0.135 / live 0.600 — both within 0.02 of their full-census receipts.
61+
| identical ranked lists (planner ran in both) | 1,844 | 845 | 845 |
62+
| frozen **crash** (the and/or/not class) | 2,349 | 0 | 1,577 |
63+
| frozen **empty**, live rows | 1,227 | 0 | 1,016 |
64+
| different lists | 14 | 4 | 4 |
65+
| **regressions** (frozen hit, live miss) | **0** of 5,434 | | |
66+
67+
Mechanism, read from the shipped code rather than inferred: for each frozen-empty question the
68+
Stage 1 replica's own `_frozen_escape_fts_query` was applied to the text — **1,217 of 1,227**
69+
produce a single double-quoted phrase spanning the whole question (`whole_question_as_one_phrase`;
70+
10 `other`). The shipped code treated any text containing a `"` as "explicitly quoted" and
71+
searched the *entire question* as one exact phrase, which matches nothing but the question
72+
itself. Learner turns quote error messages, file names and code constantly (22.6% of the
73+
population). The planner now lifts quoted spans out as phrases and plans the rest.
74+
75+
Bound on the attribution: the transition table is a paired classification, not an ablation.
76+
The crash class and the whole-phrase class are disjoint by construction (a question is
77+
classified by what the frozen code did to it), so the counts above are exact for this corpus;
78+
they are not a claim about other corpora. One instrument difference, stated so the numbers are
79+
not mistaken for each other: the transition script excludes the question's **single** message
80+
id, while `run_census` also excludes its byte-identical re-asks within the same session; so the
81+
script's absolute rates (frozen 0.1562 / live 0.6334) run slightly above the receipts (0.1305 /
82+
0.6156). Both arms receive identical exclusions inside each instrument, so the classes, the
83+
mechanism and the zero-regression count stand; the receipts remain the official numbers.
6884

6985
Per source (Stage 1 → Stage 2): claude_code 0.016 → 0.806 (its turns are the quote-heavy,
7086
punctuation-heavy ones), kiro_cli 0.202 → 0.520, codex 0.168 → 0.475, grok 0.221 → 0.588.
@@ -76,11 +92,13 @@ punctuation-heavy ones), kiro_cli 0.202 → 0.520, codex 0.168 → 0.475, grok 0
7692
because the harness still imported the deleted planner — the two were then landed together,
7793
which is the correct unit).
7894
- `agent-session-tools`: **1,833 passed** on the final tree (1,805 before Stage 2, +28).
79-
- `studyloop` package: 3,912 passed; 166 failed / 147 errors, none caused by Stage 2 — 157 are
80-
one leaked event loop (`asyncio.run() cannot be called from a running event loop`) that also
81-
hits `memory_search`/`get_active_topics`/`log_struggle` tests; the rest are sandbox conditions
82-
(no PyPI, no `tmux`, PTY spawn timeouts, journey-root guard). The only file whose failures name
83-
`session_search` passes 10/11 in isolation, the one red being `tmux`.
95+
- `studyloop` package, **matched control**: the pre-Stage-2 tree (`80ee57bb`, worktree, same
96+
sandbox) fails 305 node ids; the Stage 2 tree fails 313; **299 in common**, identical exception
97+
profiles (158 `RuntimeError` event-loop leaks, 15 `TimeoutError`, 14 journey-root guards, 11
98+
`MultiplexerError` = no `tmux`, 11 `CalledProcessError` = no PyPI, in both). The 14 only-in-Stage-2
99+
ids are wheel-build smoke tests (network), one live-provider test, and two lifecycle ids that
100+
differ only by stderr noise concatenated into the node id; none touch search. Suspected
101+
environmental; not demonstrated clean beyond this comparison.
84102
- Toy corpus (CI validation tier): 7/7 on measured values.
85103

86104
## Deferred, with reason
@@ -102,6 +120,25 @@ punctuation-heavy ones), kiro_cli 0.202 → 0.520, codex 0.168 → 0.475, grok 0
102120
- Ranking, not vocabulary and not crashes, is the dominant miss on the corpus the product serves:
103121
1,832 ranking misses vs 257 vocabulary misses.
104122

105-
## Council addendum
123+
## Council addendum — single seat `openai.gpt-6-astra`, verdict **REJECT** (6 findings)
106124

107-
_(appended when the single-seat review returns)_
125+
Brief 25.5 KB; 58 s; 844 words. Every finding verified at source before acting.
126+
127+
| id | sev. | finding | disposition |
128+
|---|---|---|---|
129+
| F1 | MAJOR | fallback after a rejected explicit query re-entered explicit detection; `fts:fts:alpha?` runs `alpha?` unguarded | **CONFIRMED at runtime** (reproducer crashed with `fts5: syntax error near "?"`). Fixed: the fallback calls `plan_natural_language()` only, strips every prefix; regression tests through MCP and CLI (`fts:fts:alpha?`, `fts:"alpha" OR ? AND`) |
130+
| F2 | MAJOR | an uppercase operator inside a quoted span classified as explicit, so `"error OR warning" recovery` never widened | **CONFIRMED.** Operators now count only outside double-quoted spans (`_has_operator_outside_quotes`); tests pin the seat's discriminating case (upper == lower inside quotes; operator outside quotes still explicit) |
131+
| F3 | MAJOR | census attribution overclaimed from a 400-sample; mechanism unverified | **ACCEPTED** → full-population per-question transition receipt; mechanism read from the frozen escape function (1,217/1,227); zero regressions across 5,434; bound and instrument difference stated |
132+
| F4 | MAJOR | golden gate labelled met when it was changed; "five added lines" vs 15-line stat | **ACCEPTED** → gate marked NOT MET AS WRITTEN and amended to projected equality, checked between named commits by script; the 15 lines are trailing-comma changes on lines that also gained a comma |
133+
| F5 | MINOR | 91/91 and "5 rescued" inferred, not counted | **ACCEPTED** → counted directly (`gold_counters`): 91/91 ordered lists, 5 of 42, `moved_outside_crash_class: []` |
134+
| F6 | MINOR | studyloop attribution lacked a matched control | **ACCEPTED** → pre-Stage-2 worktree run in the same sandbox; 299/305 common ids, identical exception profiles |
135+
136+
After the corrections the code fix and tests are in `c8c105c4`; the gold and census receipts
137+
were **not** regenerated for F1/F2 because neither reproducer pattern occurs in the gold set or
138+
in the census population in a way that changes an outcome (checked over all 5,434 texts: zero start
139+
with `fts:`; **4** carry an uppercase operator inside quotes — all four re-run through the fixed tool
140+
keep their recorded own-session outcome, 3 hits and 1 miss, now via the `or` plan). The receipts
141+
therefore stand exactly.
142+
Escalation rule: a REJECT with MAJORs escalates to three seats — **pending Andy's call** whether
143+
to run the three-seat council now or fold Stage 2 into the Stage 3 three-seat review already
144+
scheduled.

0 commit comments

Comments
 (0)