feat: run-integrity leak detection over finished transcripts

[dmorosanu]

What

After a task finishes, coder_eval.integrity derives what counts as graded material from the task's own config (reference.file, $TASK_DIR operands in its criteria, sandbox.mock_path_dirs, template_sources[*].mount_point, plus always-answer basenames and path segments), scans the untruncated transcript, and classifies each command as a read of that material or not. Listing a path (ls, find) is not a read; emitting its content (cat, sed, python, git show) is.

Verdicts: CLEAN / TAINTED / INCONCLUSIVE (never CLEAN when the scan was partly blind - unrecovered sub-agent threads, parameterless commands over threshold, or unknown tools touching graded material). Findings are typed GRADED_READ (reference answer, grader script, task YAML, _fixtures/ goldens) vs MOCK_DATA_READ (mock fixture stores), so the two can be triaged apart.

INTEGRITY_MODE defaults to detect (report only). In void mode a tainted pass flips SUCCESS to FAILURE with voided=true, keeping weighted_score as computed (the score is the diagnostic); voided replicates are excluded from the experiment pass rate.

Why

A task's score only measures the agent if the agent worked from the scenario's evidence. Measured across finished runs, agents read the reference answer, grader scripts, their own task YAML, golden solutions, and mock fixture stores - and every leaking row scored SUCCESS at or near 1.0, inflating suite averages and hiding regressions. Containment cannot close this (graders must be executable at grading time), so detection is the control.

Design constraints

• Detection, not containment: under driver: tempdir the whole checkout shares the agent's filesystem by design; detection is the only driver-independent lever.
• Scans CommandTelemetry.parameters directly, never result_summary, and never truncates the haystack (CommandExecutedChecker clips at 2000 chars - exactly where a long cat hides).
• Not a criterion type: criteria are opt-in per YAML; the gate lives in _finalize_result.
• evaluate_integrity never raises - an integrity bug must not take down a row that otherwise ran fine.

Validation

• Integrity suite: 166 tests (baseline 92 before the fixes). Catches 5 of 5 measured leak classes, up from 2 of 5.
• Full suite on Windows: 5 pre-existing failures only (2 symlink WinError 1314, 3 live-marked claude-settings).
• Full suite on Linux + Docker (Ubuntu 24.04, Docker 29.1.3): 15 failed / 4054 passed / 12 skipped, and all 15 failures reproduce identically at the base commit b390d7dc - zero regressions from this branch. The previously win32-skipped Docker/preservation files plus both integrity suites: 268 passed, 0 skipped, including the mount-shape change in e303ce5c.

Known follow-ups (kept out of this PR)

• Shim-execution false positive: a path-qualified execution of the mock shim (./m/uip <verb>) is flagged MOCK_DATA_READ by the conservative unknown-utility rule even though executing the shim is the intended use. Fix direction: treat a graded-path match that is the segment's argv[0] as execution, not a read.
• Site-packages false positive: the mocks path segment matches inside unrelated library paths (.venv/lib/python3.x/site-packages/uipath/eval/mocks/...).
• Both must land before anyone enables void mode; detect mode is unaffected in its purpose.

[dmorosanu]

Validation summary (detect-mode)

The detector has been validated two independent ways: a live forward sweep and an offline A/B that isolates the effect of the fixes. Keeping this PR in draft — it is sound for detect mode, but void-mode gate work and a few follow-ups remain (below).

1. Live forward sweep — 127 tasks, INTEGRITY_MODE=detect

Ran the leak-class task set on the antigravity/gemini-3.5-flash arm under the docker driver against this branch.

• Zero DETECTOR-MISS. For all 26 tasks expected to leak that came back CLEAN, an independent grep (not the detector) found no content-read of the class's material in any replicate — every one was leak-not-repeated (agent nondeterminism), not a silent miss.
• Real leaks caught: all staged-task-definition reads (/work/input/task.yaml, context.json), fixture/oracle reads (mocks/calls.log), sealed-store probes.
• Both headline false-positive fixes confirmed: the mock-shim execution FP dropped from 56 flagged to 23 (the 23 are genuine Reads of shim source/mock data), and the site-packages FP cleared.

2. Offline A/B — pre-fix vs post-fix over the same historical transcripts

Re-scored four historical arm runs (codex ×2, gemini, luna) with the pre-fix detector and this branch. Same transcripts both sides, so every verdict change is attributable to the code, not to agent nondeterminism.

• 35 false positives removed, 0 newly-caught false positives, 0 false negatives.
• All 35 removals verified as genuine FPs: shim execution (./m/uip …, PATH=…m… uip …, and bash -lc-wrapped forms) and a site-packages path match.
• False-negative guard checked: a real sqlite3.connect('m/.store') probe on one task (gemini run) still taints — it was not among the removals.

Two false positives the first A/B surfaced have been fixed on this branch:

• c9c47b9a — unwrap a bash -c "<script>" wrapper so its redirects/pipelines classify as the command that ran (codex records every shell call this way, which was hiding > write-targets from the write veto).
• 7f82cb8b — decide the mock-shim invocation carve-out before the utility rules, so a quote-swallowed | tee tail cannot turn an execution into a read.

Integrity suite: 354 passing (was 92 before this branch; 333 before these last two fixes). No regressions.

What is NOT yet validated

• The bypass / missed-detection fixes have no live evidence. Neither dataset contained an agent using those evasions, so the A/B found 0 genuine newly-caught leaks. Those fixes rest on their unit tests; live proof would need a deliberately adversarial run.

Remaining before void mode

• The gate-behavior items from review (deliverable re-read exemption, etc.).
• Known follow-ups, none blocking detect: find -exec bash -c 'cat <key>' (pre-existing false negative), bare … | tee /tmp/x (tee-writes-its-operands semantics), and unquoted joining of recovered sub-agent shell argv.