Skip to content

fix(ai): a session that has asked forty different questions is not idle - #417

Merged
tpouyer merged 1 commit into
mainfrom
fix/orientation-is-progress
Sep 9, 2026
Merged

tpouyer merged 1 commit into
mainfrom
fix/orientation-is-progress

Conversation

@tpouyer

@tpouyer tpouyer commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

Closes #416.

GATE-PROGRESS-1 counted writes the workspace accepted and suite runs over a change set not run
before. Everything else counted for nothing, so these two sessions were the same number — zero:

calls: 40   distinct: 40   repeats: []   files touched: 15     <- run 34402407065
calls: 40   distinct:  1   repeats: [the same read x40]

The first is the second /implement on #410. Its forty calls are in the evidence case the run
filed: search_code {"mode": "map"}, then CommandValidate and staged_refusal by name, a
skeleton of command.py, then strategy.py at four offsets, sandbox.py, ruff_adapter.py,
detected.py, test_validate_phase.py in three windows, the GATE-VALIDATE and GATE-SANDBOX
rows, .lockstep/lockstep.py, and cli.py's ls. On turn 38 it said:

"Now I have a thorough understanding. Let me write the failing test. Here's what I need to test:

  1. CommandValidate declares EXECUTES_CODE (currently…"

Two confirming reads later the ceiling stopped it, at $3.46, having repeated nothing.

Why not raise the number

The row records this failing once already — run 34152787128, stopped at turn 20 "still reading the
code it was about to change" — and diagnoses it in its own words:

because the number was set from the clean run's 68 turns without knowing how many came before its
first write

The remedy then was to double it. This is the same defect one doubling later, and that sentence
explains both. The row's own objection to raising ceilings ("gives a wandering run more to use and
a good run nothing") is correct and is the reason this changes what is counted instead.

The change

The tool runner keeps a second tally, explored: distinct read_file, list_files, search_text
and search_code calls, keyed on the arguments. Offset 1 and offset 600 of one file are two
questions — that is how a model reads something too long to hold in one call, and every ranged read
in the run above was one of them. Counted on the asking, not the answer: a refused read and an
empty search both told the session something it did not know.

Before the first write, a turn that asked something new resets the idle counter. It
deliberately does not set last_progress, which is what doubles the allowance — setting it there
would halve the allowance in the middle of the phase the doubling is for, and
test_orientation_does_not_halve_the_allowance_it_exists_for pins that.

After the first write, nothing changes. Twenty turns of reading over a change already staged is
wandering however many different files it opens, and that is the half the #319 evidence supports.

This removes an asymmetry nobody chose: staging a stub on turn 3 bought effectively unlimited
orientation, while reading first hit the wall. The successful /implement on #400 made 44 reads
and 66 searches and survived — because it staged early, not because it explored less.

One pre-existing test moved by exactly one turn

test_gate_progress_1_a_reading_phase_gets_twice_the_ceiling_before_the_first_write expected six
calls and now sees seven. That is the change working, not a regression to paper over: the session's
first read of a file is a question it had not asked, so it is not an idle turn; the six repeats
after it are, and still stop it. The comment there says so and points at the tests holding both
halves.

Verification

  • make check green from the committed tree: 2879 passed, 6 skipped; ruff format reports 269 files
    unchanged, so what is pushed is what passed.
  • The regression case is run 34402407065's own forty calls, kept verbatim from its evidence
    case rather than paraphrased, replayed through the counter.
  • Four falsification controls fired and restored from the commit, each asserting its own edit
    landed first:
    • novelty ignored before the first stage → the regression case blocks again;
    • novelty counted after the first stage too → the post-write wandering test stops stopping;
    • exploring sets last_progress → the allowance halves and the orientation test blocks;
    • the question key forgets its arguments → two windows of one file read as one question.

Objectives

O7 — progress is arithmetic, and the arithmetic now measures what the row says it measures.
O13blocked is never a failure, which holds only while the controls stop runs that deserve
stopping; one that stops a working session a turn before its first write spends the budget,
produces nothing, and teaches whoever watches it to raise ceilings.

GATE-PROGRESS-1 counted writes the workspace accepted and suite runs over a change set not run
before. Reads and searches counted for nothing, so forty distinct queries over fifteen files and
forty repeats of one query were the same number: zero.

Run 34402407065 — the second `/implement` on #410 — was stopped by that. Its forty calls are in
the evidence case it filed, and forty of them are distinct: `search_code map` first, then the seams
its ticket named, ranged reads throughout, ending on turn 38 with "Now I have a thorough
understanding. Let me write the failing test." It was cut one turn before its first write, at
$3.46, having repeated nothing.

The row already recorded this failing once at half the number — run 34152787128 stopped at turn 20
"still reading the code it was about to change" — and diagnosed it in its own words: the number was
set from a clean run's 68 turns "without knowing how many came before its first write". The remedy
was to double it. This is the same defect one doubling later, so what is COUNTED changed rather
than the number changing again.

The tool runner keeps a second tally of distinct `read_file`, `list_files`, `search_text` and
`search_code` calls, keyed on the arguments — offset 1 and offset 600 of one file are two
questions, which is how a model reads something too long to hold in one call, and re-reading one
window is not. Counted on the asking rather than the answer: a refused read and an empty search
both told the session something it did not know.

Before the first write, a turn that asked something new resets the idle counter. It deliberately
does not set `last_progress`, which is what doubles the allowance — setting it there would halve
the allowance in the middle of the phase the doubling exists for. After the first write nothing
changes, which is the half the #319 evidence actually supports: twenty turns of reading over a
change already staged is wandering however many different files it opens.

This removes an asymmetry nobody chose. Staging a stub on turn 3 bought unlimited orientation
afterwards; reading first — the behaviour a ticket that tightens a security control wants — hit the
wall.

Closes #416

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

in-lockstep review — intent

succeeded · $0.0608 · 3 in / 1001 out tokens

location finding
⚠️ src/in_lockstep/ai/invoker.py:571 The asked_something_new branch guards on not last_progress, but last_progress is set to a non-empty string only after a productive write turn; the guard is correct as stated. However, explored is a snapshot taken before dispatch and compared to the value after, yet explores (the baseli
⚠️ tests/in_lockstep/test_orientation.py:163 test_gate_progress_1_a_session_repeating_one_question_still_stops expects idle_turns == 40 (the doubled pre-staging allowance), but given the bug above, idle would reset every turn once explored has risen above zero from the first call — the session would never stop, making this test a false
⚠️ src/in_lockstep/ai/builtins.py:793 The key for _asked serialises argument values with repr(v) and keys with str(k). If an argument value is a string, repr adds quotes, so {'path': 'foo'} hashes as ('read_file', (('path', "'foo'"),)). That is consistent within a run, but it means the key format is not the natural identity
ℹ️ the lens was given this change's blast radius (2325 chars)

@github-actions

github-actions Bot commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

in-lockstep review — performance

succeeded · $0.0480 · 3 in / 374 out tokens

location finding
⚠️ src/in_lockstep/ai/builtins.py:793 _asked set grows unboundedly for the lifetime of a session with no eviction.

@github-actions

github-actions Bot commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

in-lockstep review — security

succeeded · $0.0518 · 3 in / 380 out tokens

location finding
⚠️ src/in_lockstep/ai/builtins.py:793 Tool argument values are repr()d into the deduplication key, not normalised.
ℹ️ the lens was given this change's blast radius (2325 chars)

@github-actions

github-actions Bot commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

in-lockstep review — tests

succeeded · $0.0551 · 3 in / 626 out tokens

location finding
⚠️ tests/in_lockstep/test_orientation.py:178 test_orientation_does_not_halve_the_allowance_it_exists_for does not assert the session completed all thirty reads.
⚠️ tests/in_lockstep/test_orientation.py:155 test_gate_progress_1_a_session_repeating_one_question_still_stops asserts idle_turns == 40 but the attribute name on the outcome is not visible in the diff.
ℹ️ the lens was given this change's blast radius (2325 chars)

@tpouyer
tpouyer merged commit f86baae into main Sep 9, 2026
4 checks passed
@tpouyer
tpouyer deleted the fix/orientation-is-progress branch September 9, 2026 21:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(ai): a session that has asked forty different questions is not idle

1 participant