Skip to content

fix(ptc): teach the session scope every PTC program shares - #869

Open
WeiLinCool wants to merge 1 commit into
T-Auto:masterfrom
WeiLinCool:fix/ptc-scope-guidance
Open

WeiLinCool wants to merge 1 commit into
T-Auto:masterfrom
WeiLinCool:fix/ptc-scope-guidance

Conversation

@WeiLinCool

@WeiLinCool WeiLinCool commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Closes #868

What was wrong

PTC can evaluate a program in a runtime that retains state, so a top-level const/let/class/function outlives the program that declared it. A second declaration of that name is a SyntaxError at compile time and the whole program is discarded before its first line runs, tool calls included. A program that throws keeps its names as well — bindings are instantiated before evaluation — so a failed program cannot be retried verbatim: the retry dies on a name, not on its arguments. The name is never reclaimable: there is no way to delete a lexical binding, and const is not a context property, so Object.keys cannot even enumerate it.

In the session that filed #868 (127 run_code programs) this was 11 of 17 failures (65% of failures, 8.7% of programs). Two were retries of a program that had already failed. Names declared on turn 1 (q1, e1) still collided on turn 6, and failed programs left ~70 permanently unusable names.

The shipped guidance taught the pattern: its Step 3 example is const [load, head] = await Promise.all([...]), and the model copied load into the next program, dying on program two.

The change

release/0.5.5 rewrote the same file. This branch is rebased onto the current master (which contains 0.5.5), so the diff against the 0.5.5 text is 15 lines:

  • ## One session, one scope — a top-level name can outlive its program, a second one is a compile-time error that discards the whole program, a failed program keeps its names, so each program owns its names and a retry renames.
  • Two mistake-table rows — the same top-level name in a second program; re-sending a failed program with new arguments.
  • BATCHING_GUIDANCE_REVISION → '5' (0.5.5 already used 4).

What this deliberately does not do: wrap the examples in { }. That was the first shape of this PR and it is incompatible with 0.5.5: test/batching-examples.test.mjs compiles and executes every shipped example through an AsyncFunction and asserts the returned summary, so an example has to return, and return cannot live inside a bare block. Switching the examples to console.log silently removed them from that test's coverage — I reverted it for that reason.

The stronger alternative, if maintainers prefer 0% over 5%: wrap each example body in return await (async () => { ... })(). It keeps the returned value, keeps batching-examples green, and makes the names local, at the cost of turning every example into an IIFE. Say the word and I will send it as a follow-up.

Validation

A. Model-driven A/B over one audit task in a retained REPL — same model, same task, same run_code contract, only --guidance differs, 4 runs per arm, 14-program budget:

guidance programs discarded by a redeclaration rate
0.5.5 text 56 7 12.5%
this branch 56 3 5.4%
$ node bench/probes/probe-scope-run.mjs --guidance /tmp/rev055.md --steps 14 --json
$ node bench/probes/probe-scope-run.mjs --guidance prompts/batching-guidance.md --steps 14 --json

B. Why the examples are not the vehicle — bench/probes/probe-scope-guidance.mjs runs every fenced example twice in one retained REPL. Both texts score 6 of 7 redeclared, because the examples are unchanged; that is the measurement that says this fix has to be the rule, not the shape.

C. Suite — npm run verify on this branch (darwin-arm64): 247 of 248 suites pass, 7 skipped. The one failure is preset-fragment, which compares the shipped presets/ops-ptc.patch.yml checksum against the locally installed host checkout; that checkout is at a newer commit (838665e0… vs the recorded ce02148b…), and this branch does not touch presets/. It passes in CI, where the host is installed from the pinned range.

D. CI required — green on the rebased head: https://github.com/T-Auto/dsh-ops/actions/runs/43458558329

node-floor              success  2m50s
source-size             success
rust-lint               success
workflow-lint           success
tools-trial             success
validate (linux-x64)    success  3m48s
validate (win32-x64)    success  7m40s
validate (darwin-x64)   success  6m55s
validate (darwin-arm64) success  3m29s
CI required             success

An earlier run of the same content was red twice for infrastructure reasons, not for this change: validate (darwin-arm64) was cancelled with zero steps and no runner assigned, and validate (darwin-x64) failed once on test/native-guidance.test.mjs:59, which waits for the Native reader to be published and then asserts the official read is already hidden — publication and hiding are two steps, so a loaded runner can lose that race. Both legs pass on the current head.

A state-retaining PTC runtime keeps a program's top-level `const`, `let`,
`class` and `function`, so a second declaration of that name is a SyntaxError
at compile time and the whole program is discarded before its first line runs,
tool calls included. A program that throws keeps its names too, which is why
re-sending a failed program verbatim dies on a name rather than on its
arguments, and why the name is never reclaimable.

Measured on one 127-program ops-ptc session: 11 of 17 failed programs were this
exact error, two of them retries of a program that had already failed, with
names declared on turn 1 still colliding on turn 6.

The guidance now states one scope per session, makes each program own its
top-level names, separates a retained name from a re-declared one, and lists
both traps in the mistake table. The examples keep their `return`:
batching-examples executes them and asserts the returned summary, so a printed
example would silently drop out of that coverage.

A model-driven A/B over one audit task in a retained REPL discards 7 of 56
programs on the 0.5.5 text and 3 of 56 after (4 runs per arm, 14-program
budget, only the guidance file differs).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant