Skip to content

Commit 7ad438e

Browse files
authored
ledger: diagnosable executor attribution via TDD_EXECUTOR_MODEL and Executor.reason (#92)
* test: TDD_EXECUTOR_MODEL yields source declared TDD-Run: 15 TDD-Cycle: 1 TDD-Phase: red * feat: harness-declared executor identity via TDD_EXECUTOR_MODEL TDD-Run: 15 TDD-Cycle: 1 TDD-Phase: green * refactor: TDD_EXECUTOR_MODEL resolves with source declared TDD-Run: 15 TDD-Cycle: 1 TDD-Phase: refactor * test: declared identity overrides transcript detection TDD-Run: 15 TDD-Cycle: 2 TDD-Phase: red * feat: declared executor identity takes precedence TDD-Run: 15 TDD-Cycle: 2 TDD-Phase: green * refactor: the declared override wins over a readable transcript TDD-Run: 15 TDD-Cycle: 2 TDD-Phase: refactor * test: unknown executor carries the missing-env reason TDD-Run: 15 TDD-Cycle: 3 TDD-Phase: red * feat: Executor.reason — CLAUDE_CODE_SESSION_ID not set TDD-Run: 15 TDD-Cycle: 3 TDD-Phase: green * test: unknown executor carries the no-transcript reason TDD-Run: 15 TDD-Cycle: 4 TDD-Phase: red * feat: reason names the session whose transcript was not found TDD-Run: 15 TDD-Cycle: 4 TDD-Phase: green * test: unknown executor carries the no-model-record reason TDD-Run: 15 TDD-Cycle: 5 TDD-Phase: red * feat: reason distinguishes a model-less transcript from a missing one TDD-Run: 15 TDD-Cycle: 5 TDD-Phase: green * test: an unattributed run leaves an executor_unknown event TDD-Run: 15 TDD-Cycle: 6 TDD-Phase: red * feat: run start logs executor_unknown with the detection reason TDD-Run: 15 TDD-Cycle: 6 TDD-Phase: green * test: run start result warns when the executor is unknown TDD-Run: 15 TDD-Cycle: 7 TDD-Phase: red * feat: executor_warning in the run start envelope TDD-Run: 15 TDD-Cycle: 7 TDD-Phase: green * test: doctor names the executor-identity diagnosis TDD-Run: 15 TDD-Cycle: 8 TDD-Phase: red * feat: informational executor identity check in doctor TDD-Run: 15 TDD-Cycle: 8 TDD-Phase: green * docs: document TDD_EXECUTOR_MODEL, Executor.reason, executor_unknown in §5.1 and harness-integration * docs: friction log for issue-74-executor-attribution * test: pin a declared executor identity so the suite runs the same in CI
1 parent 871a048 commit 7ad438e

8 files changed

Lines changed: 303 additions & 21 deletions

File tree

docs/PRD.md

Lines changed: 20 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -109,16 +109,30 @@ Many runs may reference one contract. This is what makes A/B comparison across m
109109

110110
The harness exposes a session identifier but **not** the model. Resolution order:
111111

112-
1. `CLAUDE_CODE_SESSION_ID` from the environment → locate
112+
1. `TDD_EXECUTOR_MODEL` environment variable — set by the launching harness when it knows
113+
the answer (e.g. a subagent harness that inherits its parent's `CLAUDE_CODE_SESSION_ID`
114+
and would be mis-attributed). Recorded with `source: declared`. Wins over transcript.
115+
2. `CLAUDE_CODE_SESSION_ID` from the environment → locate
113116
`~/.claude/projects/<slug>/<session-id>.jsonl` → read the `model` field.
114-
2. Failing that, a `--executor` label supplied by a **human** at `run start`.
115-
3. Failing that, `unknown`, and the run is excluded from model-comparison metrics.
117+
Recorded with `source: transcript`.
118+
3. Failing that, a `--executor` label supplied by a **human** at `run start`.
119+
Recorded with `source: human`.
120+
4. Failing that, `unknown` (`source: unknown`), and the run is excluded from model-comparison
121+
metrics. `Executor.reason` records why: `CLAUDE_CODE_SESSION_ID` not set; no transcript
122+
found for the session; the transcript contains no model records.
123+
124+
When resolution yields `unknown`, `run start` emits an `executor_unknown` integrity event
125+
(detail = the reason) and includes `executor_warning` in the success envelope so the gap is
126+
visible at the moment it can still be fixed.
127+
128+
`tdd doctor` reports an informational `executor identity` check (always `ok: true`) showing
129+
`<source>: <model>`, plus the reason when the source is `unknown`.
116130

117131
- **R5.1** The transcript lookup is isolated behind a single resolver so an undocumented format
118132
change breaks one function, not the tool.
119-
- **R5.2** Agents never supply executor identity by any path. Step 2 is a human affordance.
133+
- **R5.2** Agents never supply executor identity by any path. Step 3 is a human affordance.
120134
- **R5.3** Resolution requires the tool to run on the same host as the agent. Remote or CI
121-
execution falls through to step 2.
135+
execution can set `TDD_EXECUTOR_MODEL` to declare the answer explicitly.
122136

123137
### Cycle
124138
| Field | Notes |
@@ -161,7 +175,7 @@ Reserved per-run keys: `plan_quality_score` (per plan phase, with rationale), `c
161175

162176
### IntegrityEvent
163177
Typed: `test_removed`, `test_weakened`, `undeclared_file_touched`, `restore_mismatch`,
164-
`off_protocol_invocation`, `stale_artifact`, `plan_blob_changed`.
178+
`off_protocol_invocation`, `stale_artifact`, `plan_blob_changed`, `executor_unknown`.
165179

166180
### Blocker
167181
Typed: `regression`, `target_unfixable`, `bad_red`, `plan_defect`, `tooling`, `context_exhausted`,

docs/harness-integration.md

Lines changed: 18 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -104,16 +104,27 @@ and auditors should read the two accordingly.
104104

105105
## Executor identity and subagents
106106

107-
`tdd run start` records which model is executing by reading the harness session id
108-
(`CLAUDE_CODE_SESSION_ID`) and resolving the model from that session's transcript.
109-
This is trustworthy only when the executor is a **top-level session** — its own
110-
terminal, worktree, or cloud session.
107+
`tdd run start` records which model is executing. Resolution order (first match wins):
108+
109+
1. **`TDD_EXECUTOR_MODEL`** — set this env var when the launching harness knows the answer
110+
(e.g. a CI pipeline or a subagent harness). Recorded with `source: declared`. Takes
111+
precedence over transcript detection.
112+
2. **Transcript** — reads `CLAUDE_CODE_SESSION_ID` and resolves the model from
113+
`~/.claude/projects/<slug>/<session-id>.jsonl`. Recorded with `source: transcript`.
114+
Trustworthy only for top-level sessions (see below).
115+
3. **`--executor`** — human-supplied label at `run start`. Recorded with `source: human`.
116+
4. **`unknown`**`Executor.reason` records why: env var not set; no transcript for the
117+
session; transcript contains no model records. An `executor_unknown` integrity event is
118+
emitted and `result.executor_warning` is set in the `run start` envelope.
119+
120+
`tdd doctor` always reports an informational `executor identity` check (`ok: true`) showing
121+
the resolved `<source>: <model>`, plus the reason when the source is `unknown`.
111122

112123
An in-process subagent (Claude Code's Agent/Task tool) inherits the parent's session
113124
id and has no transcript of its own in the location the resolver reads, so a run
114-
started by a subagent is attributed to the **parent's** model. The run itself is
115-
unaffected — but if you are comparing models across runs, dispatch executors as
116-
separate top-level sessions, not as subagents, or the comparison is silently wrong.
125+
started by a subagent would be attributed to the **parent's** model. Set
126+
`TDD_EXECUTOR_MODEL` in the subagent's environment to declare the correct model
127+
explicitly — the declared path was added precisely to fix this case.
117128

118129
## Concurrent-command refusals
119130

src/tddcli/cli.py

Lines changed: 15 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -304,6 +304,12 @@ def cmd_doctor(args) -> Envelope:
304304
"ledger outside worktree", not str(ledger.path).startswith(str(worktree)), str(ledger.path)
305305
)
306306

307+
ex = identity.resolve(worktree)
308+
ex_detail = f"{ex.source}: {ex.model}"
309+
if ex.source == "unknown":
310+
ex_detail += f" — {ex.reason}"
311+
check("executor identity", True, ex_detail)
312+
307313
projects: dict[str, dict] = {}
308314
for name, project in cfg.projects.items():
309315
before = len(checks)
@@ -798,6 +804,8 @@ def cmd_run_start(args) -> Envelope:
798804
ledger.event(run_id, None, "baseline_reused", json.dumps(sorted(reused)))
799805
if implausible:
800806
ledger.event(run_id, None, "baseline_accepted", json.dumps(implausible))
807+
if executor.source == "unknown":
808+
ledger.event(run_id, None, "executor_unknown", executor.reason or "")
801809

802810
# Baselines and the collection snapshot, per project (R9.5, R8.9) — from the
803811
# probe above, so the suite is not run twice.
@@ -850,12 +858,15 @@ def cmd_run_start(args) -> Envelope:
850858

851859
verb, opening = engine.opening_action(cycle)
852860
detail = f"Run {run_id} started ({executor.model}, via {executor.source}). {opening}"
861+
result: dict = {
862+
"baselines": {n: len(v) for n, v in ledger.baselines(run_id).items()},
863+
"executor_source": executor.source,
864+
}
865+
if executor.source == "unknown":
866+
result["executor_warning"] = executor.reason or ""
853867
return Envelope(
854868
run=engine.run_state(cycle),
855-
result={
856-
"baselines": {n: len(v) for n, v in ledger.baselines(run_id).items()},
857-
"executor_source": executor.source,
858-
},
869+
result=result,
859870
next_action=NextAction(verb, detail),
860871
)
861872
finally:

src/tddcli/identity.py

Lines changed: 16 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -26,7 +26,8 @@
2626
class Executor:
2727
model: str
2828
session: str | None
29-
source: str # transcript | human | unknown
29+
source: str # transcript | human | declared | unknown
30+
reason: str | None = None
3031

3132

3233
def _slug(path: Path) -> str:
@@ -69,14 +70,25 @@ def _model_from_transcript(path: Path) -> str | None:
6970

7071
def resolve(project_path: Path | None = None, human_label: str | None = None) -> Executor:
7172
session = os.environ.get("CLAUDE_CODE_SESSION_ID")
72-
if session:
73+
74+
declared = os.environ.get("TDD_EXECUTOR_MODEL")
75+
if declared:
76+
return Executor(model=declared, session=session, source="declared")
77+
78+
reason: str | None = None
79+
if not session:
80+
reason = "CLAUDE_CODE_SESSION_ID is not set"
81+
else:
7382
transcript = _find_transcript(session, project_path)
74-
if transcript is not None:
83+
if transcript is None:
84+
reason = f"no transcript for session {session} under {TRANSCRIPT_ROOT}"
85+
else:
7586
model = _model_from_transcript(transcript)
7687
if model:
7788
return Executor(model=model, session=session, source="transcript")
89+
reason = f"no model records in transcript {transcript}"
7890

7991
if human_label:
8092
return Executor(model=human_label, session=session, source="human")
8193

82-
return Executor(model="unknown", session=session, source="unknown")
94+
return Executor(model="unknown", session=session, source="unknown", reason=reason)
Lines changed: 89 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,89 @@
1+
# Implementation Friction Log: tasks/issue-74-executor-attribution.md
2+
3+
- Run: 15
4+
- Executor: claude-sonnet-4-6 (source: transcript)
5+
- Plan blob: `a66891f7adb8b6ea80f680ac92727388eff9ed8a` (declared)
6+
- Started: 2026-08-29T07:15:49.931037+00:00 Ended: 2026-08-29T08:03:42.177931+00:00 Outcome: complete
7+
- Baseline failures at start: tddcli=0
8+
9+
## Plan fidelity
10+
11+
- Declared cycles: 8
12+
- Delivered: 8 Skipped: 0
13+
- Never reached: none
14+
- Human interventions: 0
15+
16+
### Cycle 8: doctor reports executor identity and the failure reason informationally _(standard)_
17+
- **Target:** `tddcli::tests/test_executor_attribution.py::test_doctor_reports_executor_identity`
18+
- **Projects:** `tddcli`
19+
- **Suite runs by phase:** {'AWAITING_TEST': 1, 'AWAITING_IMPL': 1, 'CLOSE_SWEEP': 1}
20+
- **First run outcome:** failed (as expected)
21+
- **Commits:**
22+
- `241591a22` [red] test: doctor names the executor-identity diagnosis (1 files)
23+
- `c658611f1` [green] feat: informational executor identity check in doctor (1 files)
24+
25+
### Cycle 7: the run start envelope surfaces the attribution warning _(standard)_
26+
- **Target:** `tddcli::tests/test_executor_attribution.py::test_run_start_envelope_carries_executor_warning`
27+
- **Projects:** `tddcli`
28+
- **Suite runs by phase:** {'AWAITING_TEST': 1, 'AWAITING_IMPL': 1, 'CLOSE_SWEEP': 1}
29+
- **First run outcome:** failed (as expected)
30+
- **Commits:**
31+
- `86541dc3d` [red] test: run start result warns when the executor is unknown (1 files)
32+
- `2d33e0d0e` [green] feat: executor_warning in the run start envelope (1 files)
33+
34+
### Cycle 6: run start records an executor_unknown event with the reason _(standard)_
35+
- **Target:** `tddcli::tests/test_executor_attribution.py::test_run_start_records_executor_unknown_event`
36+
- **Projects:** `tddcli`
37+
- **Suite runs by phase:** {'AWAITING_TEST': 1, 'AWAITING_IMPL': 1, 'CLOSE_SWEEP': 1}
38+
- **First run outcome:** failed (as expected)
39+
- **Commits:**
40+
- `726ea02f2` [red] test: an unattributed run leaves an executor_unknown event (1 files)
41+
- `daf7f7f97` [green] feat: run start logs executor_unknown with the detection reason (1 files)
42+
43+
### Cycle 5: resolve records why detection failed: transcript has no model line _(standard)_
44+
- **Target:** `tddcli::tests/test_executor_attribution.py::test_reason_names_the_model_less_transcript`
45+
- **Projects:** `tddcli`
46+
- **Suite runs by phase:** {'AWAITING_TEST': 1, 'AWAITING_IMPL': 1, 'CLOSE_SWEEP': 1}
47+
- **First run outcome:** failed (as expected)
48+
- **Commits:**
49+
- `6764fa319` [red] test: unknown executor carries the no-model-record reason (1 files)
50+
- `34029594a` [green] feat: reason distinguishes a model-less transcript from a missing one (1 files)
51+
52+
### Cycle 4: resolve records why detection failed: transcript not found _(standard)_
53+
- **Target:** `tddcli::tests/test_executor_attribution.py::test_reason_names_the_missing_transcript`
54+
- **Projects:** `tddcli`
55+
- **Suite runs by phase:** {'AWAITING_TEST': 1, 'AWAITING_IMPL': 1, 'CLOSE_SWEEP': 1}
56+
- **First run outcome:** failed (as expected)
57+
- **Commits:**
58+
- `cd3685c78` [red] test: unknown executor carries the no-transcript reason (1 files)
59+
- `8e0a82601` [green] feat: reason names the session whose transcript was not found (1 files)
60+
61+
### Cycle 3: resolve records why detection failed: session env missing _(standard)_
62+
- **Target:** `tddcli::tests/test_executor_attribution.py::test_reason_names_the_missing_session_env`
63+
- **Projects:** `tddcli`
64+
- **Suite runs by phase:** {'AWAITING_TEST': 1, 'AWAITING_IMPL': 1, 'CLOSE_SWEEP': 1}
65+
- **First run outcome:** failed (as expected)
66+
- **Commits:**
67+
- `603318a7e` [red] test: unknown executor carries the missing-env reason (1 files)
68+
- `f648879ae` [green] feat: Executor.reason — CLAUDE_CODE_SESSION_ID not set (1 files)
69+
70+
### Cycle 2: the declared override wins over a readable transcript _(standard)_
71+
- **Target:** `tddcli::tests/test_executor_attribution.py::test_declared_override_beats_transcript`
72+
- **Projects:** `tddcli`
73+
- **Suite runs by phase:** {'AWAITING_TEST': 1, 'AWAITING_IMPL': 1, 'CLOSE_SWEEP': 2}
74+
- **First run outcome:** failed (as expected)
75+
- **Commits:**
76+
- `325b820e4` [red] test: declared identity overrides transcript detection (1 files)
77+
- `1a2d64046` [green] feat: declared executor identity takes precedence (1 files)
78+
- `369b18d35` [refactor] refactor: the declared override wins over a readable transcript (1 files)
79+
80+
### Cycle 1: TDD_EXECUTOR_MODEL resolves with source declared _(standard)_
81+
- **Target:** `tddcli::tests/test_executor_attribution.py::test_env_override_resolves_as_declared`
82+
- **Projects:** `tddcli`
83+
- **Suite runs by phase:** {'AWAITING_TEST': 1, 'AWAITING_IMPL': 1, 'CLOSE_SWEEP': 2}
84+
- **First run outcome:** failed (as expected)
85+
- **Commits:**
86+
- `dffa231ed` [red] test: TDD_EXECUTOR_MODEL yields source declared (1 files)
87+
- `63a3fa815` [green] feat: harness-declared executor identity via TDD_EXECUTOR_MODEL (1 files)
88+
- `6c606917c` [refactor] refactor: TDD_EXECUTOR_MODEL resolves with source declared (1 files)
89+

tests/conftest.py

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,15 @@ def _isolated_lease_dir(tmp_path, monkeypatch):
2020
monkeypatch.setenv("TDD_LEASE_DIR", str(tmp_path / "worker-leases"))
2121

2222

23+
@pytest.fixture(autouse=True)
24+
def _pinned_executor_identity(monkeypatch):
25+
"""A developer's shell resolves identity from its live Claude session; CI
26+
resolves nothing and every run logs executor_unknown. Pin a declared
27+
identity so both behave the same; the attribution tests delenv this to
28+
exercise the unknown paths."""
29+
monkeypatch.setenv("TDD_EXECUTOR_MODEL", "pytest-executor")
30+
31+
2332
@pytest.fixture
2433
def ledger_home(tmp_path, monkeypatch):
2534
home = tmp_path / "ledger-home"

0 commit comments

Comments
 (0)