fix(ai): a session that has asked forty different questions is not idle - #417
Merged
Merged
Conversation
GATE-PROGRESS-1 counted writes the workspace accepted and suite runs over a change set not run before. Reads and searches counted for nothing, so forty distinct queries over fifteen files and forty repeats of one query were the same number: zero. Run 34402407065 — the second `/implement` on #410 — was stopped by that. Its forty calls are in the evidence case it filed, and forty of them are distinct: `search_code map` first, then the seams its ticket named, ranged reads throughout, ending on turn 38 with "Now I have a thorough understanding. Let me write the failing test." It was cut one turn before its first write, at $3.46, having repeated nothing. The row already recorded this failing once at half the number — run 34152787128 stopped at turn 20 "still reading the code it was about to change" — and diagnosed it in its own words: the number was set from a clean run's 68 turns "without knowing how many came before its first write". The remedy was to double it. This is the same defect one doubling later, so what is COUNTED changed rather than the number changing again. The tool runner keeps a second tally of distinct `read_file`, `list_files`, `search_text` and `search_code` calls, keyed on the arguments — offset 1 and offset 600 of one file are two questions, which is how a model reads something too long to hold in one call, and re-reading one window is not. Counted on the asking rather than the answer: a refused read and an empty search both told the session something it did not know. Before the first write, a turn that asked something new resets the idle counter. It deliberately does not set `last_progress`, which is what doubles the allowance — setting it there would halve the allowance in the middle of the phase the doubling exists for. After the first write nothing changes, which is the half the #319 evidence actually supports: twenty turns of reading over a change already staged is wandering however many different files it opens. This removes an asymmetry nobody chose. Staging a stub on turn 3 bought unlimited orientation afterwards; reading first — the behaviour a ticket that tightens a security control wants — hit the wall. Closes #416 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Contributor
in-lockstep review — intentsucceeded · $0.0608 · 3 in / 1001 out tokens
|
Contributor
in-lockstep review — performancesucceeded · $0.0480 · 3 in / 374 out tokens
|
Contributor
in-lockstep review — securitysucceeded · $0.0518 · 3 in / 380 out tokens
|
Contributor
in-lockstep review — testssucceeded · $0.0551 · 3 in / 626 out tokens
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #416.
GATE-PROGRESS-1counted writes the workspace accepted and suite runs over a change set not runbefore. Everything else counted for nothing, so these two sessions were the same number — zero:
The first is the second
/implementon #410. Its forty calls are in the evidence case the runfiled:
search_code {"mode": "map"}, thenCommandValidateandstaged_refusalby name, askeletonofcommand.py, thenstrategy.pyat four offsets,sandbox.py,ruff_adapter.py,detected.py,test_validate_phase.pyin three windows, theGATE-VALIDATEandGATE-SANDBOXrows,
.lockstep/lockstep.py, andcli.py'sls. On turn 38 it said:Two confirming reads later the ceiling stopped it, at $3.46, having repeated nothing.
Why not raise the number
The row records this failing once already — run 34152787128, stopped at turn 20 "still reading the
code it was about to change" — and diagnoses it in its own words:
The remedy then was to double it. This is the same defect one doubling later, and that sentence
explains both. The row's own objection to raising ceilings ("gives a wandering run more to use and
a good run nothing") is correct and is the reason this changes what is counted instead.
The change
The tool runner keeps a second tally,
explored: distinctread_file,list_files,search_textand
search_codecalls, keyed on the arguments. Offset 1 and offset 600 of one file are twoquestions — that is how a model reads something too long to hold in one call, and every ranged read
in the run above was one of them. Counted on the asking, not the answer: a refused read and an
empty search both told the session something it did not know.
Before the first write, a turn that asked something new resets the idle counter. It
deliberately does not set
last_progress, which is what doubles the allowance — setting it therewould halve the allowance in the middle of the phase the doubling is for, and
test_orientation_does_not_halve_the_allowance_it_exists_forpins that.After the first write, nothing changes. Twenty turns of reading over a change already staged is
wandering however many different files it opens, and that is the half the #319 evidence supports.
This removes an asymmetry nobody chose: staging a stub on turn 3 bought effectively unlimited
orientation, while reading first hit the wall. The successful
/implementon #400 made 44 readsand 66 searches and survived — because it staged early, not because it explored less.
One pre-existing test moved by exactly one turn
test_gate_progress_1_a_reading_phase_gets_twice_the_ceiling_before_the_first_writeexpected sixcalls and now sees seven. That is the change working, not a regression to paper over: the session's
first read of a file is a question it had not asked, so it is not an idle turn; the six repeats
after it are, and still stop it. The comment there says so and points at the tests holding both
halves.
Verification
make checkgreen from the committed tree: 2879 passed, 6 skipped; ruff format reports 269 filesunchanged, so what is pushed is what passed.
case rather than paraphrased, replayed through the counter.
landed first:
last_progress→ the allowance halves and the orientation test blocks;Objectives
O7 — progress is arithmetic, and the arithmetic now measures what the row says it measures.
O13 —
blockedis never a failure, which holds only while the controls stop runs that deservestopping; one that stops a working session a turn before its first write spends the budget,
produces nothing, and teaches whoever watches it to raise ceilings.