fix(agent): append system_prompt to the Claude Code preset instead of replacing

[Mihaiii]

Why append instead of replace

ClaudeAgentOptions.system_prompt accepts either a plain string or the claude_code preset dict. A plain string replaces Claude Code's entire default system prompt — the only way to keep the default is {"type": "preset", "preset": "claude_code", "append": ...}. coder_eval passes the experiment's system_prompt straight through as a string, so any experiment that sets even a one-line prompt silently strips every behavioral instruction the harness ships with.

That is exactly what the skills-repo experiments do. The nightly config sets an innocuous sandbox guard:

https://github.com/UiPath/skills/blob/main/tests/experiments/nightly.yaml#L39-L40

system_prompt: |
  You are a coding agent. Do not access files in sibling runs/* directories. Everywhere else is permitted.

(same pattern in tests/experiments/default.yaml#L21, smoke.yaml#L39, smoke-windows.yaml#L23, and the skill-comparison templates)

One sentence of sandbox policy costs the whole Claude Code system prompt.

Observed impact (skills nightly, skill-rpa-execution-map-greenfield)

The task is a turn-budget gate (max_turns: 10, expected 6) that assumes the agent batches tool calls per turn. Every claude-sonnet-5 run exhausted the cap; pass/fail depended on where the cap happened to land. Transcript analysis across four runs (31172161551, 31174117722 ×3 attempts, 31178128981, 31179344004 ×2 attempts):

1. Zero parallel tool calls, ever. Example distribution (run 31179344004, attempt 2): 30 assistant messages — 0 multi-tool, 19 single-tool, 11 with no tool call at all. The instruction to emit independent tool calls together in one message lives in the default system prompt; with it gone, Sonnet paid one turn per call and blew the 10-turn budget in 7/7 attempts. Skill-doc prose telling the agent to batch (added and strengthened twice in UiPath/skills) was read in-transcript and changed nothing — a reference file cannot substitute for the missing system-level contract.
2. Narration bloat. 10–11 assistant messages per run contained no tool call, just interim commentary — the default prompt's conciseness/minimal-output rules were gone.
3. Tool-choice drift. Runs used Bash cat/sed/find where the default prompt directs the dedicated Read/Grep/Glob tools (e.g. cd TextReport && cat project.json && cat Main.xaml), losing the harness's file-tracking and permission integration.

Beyond the observed items, replacing the prompt also drops the default guidance on code-reference formatting, task management, professional tone, and the security guardrails — none of which an experiment author intends to disable when adding a sandbox-scoping sentence.

The change

• claude_code_agent.py: when system_prompt is configured, wrap it as SystemPromptPreset(type="preset", preset="claude_code", append=...) so the default prompt survives and the experiment text is appended. None still means the untouched SDK default.
• agent_config.py: system_prompt field description updated ("appended to the agent's default system prompt" — previously "Replaces").
• tests/test_agent.py: two tests via the existing _capture_sdk_options pattern (append wrapping; None passthrough).

Behavioral note for existing consumers

Every experiment that sets system_prompt switches from replace to append semantics with this release. For the known consumers (sandbox-scoping one-liners) this is the intended repair. An experiment that deliberately relied on full replacement to suppress default Claude Code behavior would need a different mechanism.

Judge (agent_judge.py) and user-simulator paths construct their own options and are unaffected.

🤖 Generated with Claude Code

[Mihaiii]

@akshaylive The review is addressed now, please have a second look. Antigravity doesn't need any change because it already is on append mode by default, not replace.

[Mihaiii]

@uipreliga @akshaylive Addressed. Please have another look.

[akshaylive]

Re-review: fix(agent): append system_prompt to the Claude Code preset instead of replacing

PR #92 by @Mihaiii · fix/sys-prompt-append → main · OPEN · reviewed against 0e444f0 (previous review was against f066834)

This is a strong response to the last round — every blocker is genuinely closed, and several were verified on the wire rather than by reading the diff. The judge seam in particular is now defended at three independent layers (forced assignment after model_copy, a fail-loud SubAgentRunner guard, and system_prompt already being in _FRAMEWORK_OWNED_SDK_FIELDS), and a hostile YAML override was confirmed unable to win. The unset path is fixed properly — the preset now goes out unconditionally, so --system-prompt "" is gone — and _transport_command() pins the argv contract rather than the TypedDict shape, which is the right regression surface for this bug class. exclude_dynamic_sections=True is a real reproducibility win (verified honored through the SDK's initialize control request, not inert), and Codex no longer silently drops the field. Two things still block: a documented config combination (system_prompt_mode: replace + system_prompt_file) hard-crashes at task load, and the migration surface for the semantics flip is missing. Overall 9.2 / 10 (up from 8.9), weakest axis Evaluation Harness Quality at 8.0 / 10 (up from 5.5).

Summary

Axis	Score	🔴	🟠	🟡	🔵	Top Issue
1. Code Quality & Style	8.9 / 10	0	0	2	1	_effective_prompt_mode returns a mode string, forcing an -O-stripped assert re-check
2. Type Safety	9.9 / 10	0	0	0	1	system_prompt_mode passed through an untyped **kwargs seam in the simulator
3. Test Health	9.3 / 10	0	0	1	2	Antigravity's changed "" forwarding is untested and has no CI extra
4. Security	10.0 / 10	0	0	0	0	No findings — prior judge trust-boundary finding verified closed on the wire
5. Architecture & Design	9.4 / 10	0	0	1	1	system_prompt_semantics emitted by only 3 agents, but documented as a global contract
6. Error Handling & Resilience	8.9 / 10	0	1	0	1	replace + system_prompt_file hard-fails at task load
7. API Surface & Maintainability	9.5 / 10	0	0	1	0	CE030 doc-parity still doesn't cover agent configs
8. Evaluation Harness Quality	8.0 / 10	0	1	2	0	Affected task not re-baselined; no migration note or BREAKING CHANGE: footer

Overall Score: 9.2 / 10 · Weakest Axis: Evaluation Harness Quality at 8.0 / 10
Totals: 🔴 0 · 🟠 2 · 🟡 7 · 🔵 6 across 8 axes reviewed. (Previous: 🔴 1 · 🟠 4 · 🟡 2 · 🔵 5.)

Prior blockers — all closed

Prior finding	Status
agent_judge ran with the coding-agent preset prepended (🔴)	RESOLVED — verified on the wire against a hostile YAML override
test_system_prompt_none_leaves_sdk_default pinned a false contract (🟠)	RESOLVED — preset sent unconditionally; argv-level tests added
Cross-run comparability broke with no marker (🟠)	PARTIALLY — marker added; task not re-baselined, no migration note
Shared base field had three contradictory semantics (🟠)	RESOLVED — codex implemented, antigravity verified against its SDK
docs/agents/CLAUDE_CODE.md:102 contradicted the code (🟠)	RESOLVED — all three agent pages now correct and consistent
No transport-level test (🟡) · exclude_dynamic_sections unset (🟡)	RESOLVED — both
Empty-string case (🔵) · non-root import comment (🔵)	RESOLVED — both

Blockers

1. system_prompt_mode: replace + system_prompt_file: hard-fails at task load — the exact combination the new docs and the validator docstring advertise as supported. check_replace_mode_has_prompt accepts the pair at construction, but resolve_agent_system_prompt (src/coder_eval/orchestration/task_loader.py:239-241) clears system_prompt_file = None before assigning system_prompt = content, deliberately, to dodge the mutual-exclusivity validator. Because BaseAgentConfig sets validate_assignment=True (models/agent_config.py:123), that first assignment re-runs the new after-validator against the intermediate state (mode="replace", system_prompt=None, system_prompt_file=None) and raises — with an error naming the very field the user did set, so it reads as self-contradictory. The experiment path hits it too (experiment.py:529). Reproduced end-to-end:

   constructed OK: p.md replace
   RESOLVE FAILED: ValidationError — system_prompt_mode='replace' requires
   system_prompt (or system_prompt_file) to be set
   

   Suggested fix: set both fields in one batched update (model_copy(update={"system_prompt": content, "system_prompt_file": None})) so no half-updated state is ever validated — note a naive reorder doesn't work either, it just trips the mutual-exclusivity validator instead. (src/coder_eval/models/agent_config.py:258-273)

2. The one test covering that combination asserts the wrong half of the flow. test_system_prompt_mode_replace_requires_prompt (tests/test_agent.py:487-496) only checks that parse_agent_config(system_prompt_file=..., system_prompt_mode="replace") constructs; its comment claims "task_loader inlines it into system_prompt at resolution time" but it never calls the loader. That's structurally the same shape as last round's test_system_prompt_none_leaves_sdk_default — a green test standing in front of a broken path. A test that goes through load_task (or resolve_agent_system_prompt directly) would have caught blocker 1.

3. The semantics flip ships with no migration surface. tasks/python_cli_simulated_judged/echo_simulated_judged.yaml:19 is worded specifically for replace semantics ("reply with that exact string verbatim and nothing else — no preamble, no commentary… Ignore any project context") and is llm_judge-graded, so its baseline moves — but it wasn't re-baselined, reworded, or pinned to system_prompt_mode: replace. Codex users are affected more sharply: the field went from silently dropped to injected as developer_instructions, and system_prompt_mode doesn't exist on CodexAgentConfig (confirmed extra_forbidden), so there's no way back short of deleting the field — yet the only user-facing trace is one cell in a comparison table 200 lines into CODEX.md, and its "Known Limitations" section (where a reader would have learned the field was ignored) is untouched. No commit carries a BREAKING CHANGE: footer, so semantic-release won't put this in CHANGELOG.md. The system_prompt_semantics marker lets a dashboard segment the discontinuity, but does nothing for anyone comparing against a stored baseline. (src/coder_eval/agents/claude_code_agent.py:1180-1199, docs/agents/CODEX.md:216)

Non-blocking, but please consider before merge

Correctness gaps

• system_prompt: "" with system_prompt_mode: replace passes validation and sends an empty string as the entire system prompt — precisely the degraded regime this PR exists to eliminate. The validator checks is None, and _effective_prompt_mode() tests is not None, so blank slips through both. An empty or whitespace-only system_prompt_file reaches the same state (task_loader does content = ….strip()). Verified: parse_agent_config(type=CLAUDE_CODE, system_prompt="", system_prompt_mode="replace") constructs cleanly. Normalizing a blank resolved prompt to None in resolve_agent_system_prompt would fix this and the Antigravity "" case at one seam. (src/coder_eval/models/agent_config.py:258-272)
• The SubAgentRunner guard only fires when system_prompt is not None, so a sub-agent that omits the prompt still gets the full coding-agent preset — the same failure, reached via the omission branch. Not reachable in-tree today, but the guard is written as a contract for callers who "own their config" and doesn't enforce what it states. The invariant is "a sub-agent must have a system_prompt and be in replace mode". (src/coder_eval/evaluation/sub_agent.py:90-100)
• _effective_prompt_mode's fail-open (replace + no prompt → append) degrades silently. Fail-open is the right posture here, but a log.warning would make a silently-downgraded judge visible in task.log rather than only inferable from run.json. (src/coder_eval/agents/claude_code_agent.py:1243-1255)

Contract & schema

• system_prompt_semantics is emitted by exactly the three agents this PR touched. agents/noop_agent.py and every out-of-tree SPI agent (e.g. the coder_eval_uipath Delegate agent) emit nothing, while docs/REPORT_SCHEMA.md:140-144 documents absence as "pre-append regime" — so a dashboard following the documented rule permanently mis-buckets brand-new plugin runs into the pre-migration cohort. Giving Agent.get_environment_info() a base default (or stamping it in the orchestrator) makes the documented rule true and removes the two hardcoded literals at the same time. Also worth noting the pre-marker semantics differ per agent: codex dropped the field, antigravity already appended — so those runs are actually comparable and the doc over-warns for them.
• docs/AB_EXPERIMENTS.md:133-137 lists system_prompt_mode as an A/B lever ("append-vs-replace arms") alongside model and allowed_tools, with no caveat — while CLAUDE_CODE.md says it's "rarely needed in tasks". A replace arm sends no default prompt at all, so anyone A/B-testing append vs replace will attribute the delta to their prompt when it's dominated by the missing behavioral guidance. Either drop it from the list or annotate what the arm actually measures.

Testing

• Antigravity is the one agent whose behavior actually changed (or None dropped, so "" is now forwarded as system_instructions) and it's the one with no coverage: the new test only asserts the environment_info marker, nothing references system_instructions, and CI never installs the antigravity extra (grep -c antigravity .github/workflows/pr-checks.yml = 0). Extracting a pure _build_local_agent_kwargs() (mirroring Codex's _build_thread_options) would make the forwarding assertable without the SDK. Good news on the Codex side: CI does run uv sync --extra codex, so those tests genuinely execute.
• Codex's developer_instructions tests compare a literal against the dict the line under test just built, so a misspelled kwarg wouldn't be caught statically (_build_thread_options returns dict[str, Any] splatted into thread_start). The name is right today (openai_codex/api.py:139), so this is drift exposure — an inspect.signature assertion under the existing openai_codex skip would close it.
• _effective_prompt_mode's fail-open branch and _unwrap_system_prompt's (None, None) fallthrough are both untested.

Design / readability

• _effective_prompt_mode() returns a mode string, which forces its only caller to re-establish the invariant the helper just checked, via a bare assert that python -O strips. There are now three layers guarding one state. Having it return the resolved value (str | SystemPromptPreset) and deriving the marker from that removes the assert and the re-derivation while keeping the SSOT property. Worth noting sub_agent.py:82-84 in this same PR states the opposite convention explicitly ("Not assert because the check must survive python -O") — verified the assert isn't a live hole, but the inconsistency invites a future reader to treat it as a real guard. (src/coder_eval/agents/claude_code_agent.py:1191-1198)
• reports._unwrap_system_prompt re-derives the regime by sniffing the wire value's Python type rather than reading the system_prompt_semantics marker this PR adds, and the "System Prompt Mode" row therefore appears only in replace mode — so an append run is visually indistinguishable from a pre-PR run. Reading the marker instead (and emitting the row unconditionally) would be both more correct and simpler. While there, -> tuple[str | None, Literal["replace"] | None] would tighten the return type. (src/coder_eval/reports.py:60-74)
• CE030 doc-parity still doesn't cover agent configs (tests/lint/doc_schema_parity.py:45-48 tracks only TaskDefinition/RunLimits/Dataset/SimulationConfig, and its header explicitly excludes nested models). Last round's doc drift happened on exactly this untracked surface, and this PR adds another user-facing field plus a hand-written doc row to it.

Nits

• The _build_options comment grew from 4 to 10 lines; the first 8 carry genuinely non-obvious SDK behavior, but the last two restate what _effective_prompt_mode's docstring and the Field(description=…) already say. (src/coder_eval/agents/claude_code_agent.py:1180-1190)
• The "a sub-agent's prompt is its entire identity" rationale is now written out in six places; only agent_judge.py:271 cross-references instead of restating. Keeping the full version on SubAgentRunner's guard (the enforcement point) and pointing at it elsewhere would be tighter.
• system_prompt_mode="replace" reaches the simulator through parse_agent_config(**kwargs: Any), so pyright checks nothing — a typo would only surface at construction via extra="forbid". The code already asserts isinstance(agent_config, ClaudeCodeAgentConfig) two lines later, so constructing it directly would be statically checked. (src/coder_eval/simulation/user_simulator.py:205-220)
• The system_prompt_semantics value is a bare literal in three places with no shared type; a typo like "appended" type-checks and silently splits dashboard buckets. A SystemPromptSemantics = Literal["append", "replace"] alias next to the config field would fix it.
• docs/agents/ANTIGRAVITY.md doesn't mention the empty-string case despite that being the behavior this PR changed there.

What's Missing

Parallel paths

• 🟡 noop_agent.py and out-of-tree SPI agents don't emit system_prompt_semantics, but REPORT_SCHEMA.md documents absence as a specific regime — triggered by the three get_environment_info() overrides.
• ✅ Renderers verified not divergent: reports_html.py and reports_experiment.py share the fixed collect_agent_settings_rows; reports_stats.py, reports_junit.py, and evalboard/ have zero references.

Tests

• 🟠 No test loads replace + system_prompt_file through the real loader — which is why blocker 1 ships green.
• 🟡 Antigravity's system_instructions forwarding untested, with no CI extra.
• 🔵 Codex SDK-signature drift; _effective_prompt_mode fail-open and _unwrap_system_prompt fallthrough untested.

Downstream consumers

• 🟠 External consumers in coder-eval-uipath / eval-runner that string-handle sdk_options.system_prompt break on the str → dict change. There's no in-repo canary (nothing in-repo reads it directly), so the REPORT_SCHEMA.md prose warning is the only signal — worth a heads-up to that repo's owners.

Display & mapping dicts

• Nothing identified — the new Literal has no icon/label/counting map, and its three branch sites were checked for exhaustiveness.

Nightly impact — answered explicitly: no container rebuild, no schema migration. docker_runner.py is untouched and prompt-agnostic; the Bedrock judge route bypasses the SDK. One residual: the prompt baseline now tracks the pinned CLAUDE_CODE_VERSION=2.1.177, so bumping it becomes a score-affecting change (correctly attributable via environment_info.claude_code_cli).

Harness & Lint Improvements

Static checks (lint / type)

• Extend CE030 to the agent configs (ClaudeCodeAgentConfig → CLAUDE_CODE.md, CodexAgentConfig → CODEX.md, antigravity → ANTIGRAVITY.md, plus BaseAgentConfig) — recommended last round, still open; would gate the new system_prompt_mode doc row.
• New CEnnn: every class registered in AgentRegistry must emit system_prompt_semantics from get_environment_info() (registry-based whole-tree check, same shape as CE025) — or give the base a default so the rule isn't needed.
• New CEnnn: forbid bare assert as type-narrowing in src/coder_eval/agents/, where the codebase's own convention is stated (sub_agent.py:82-84) but unenforced. Would catch the _build_options assert (also the one new bandit B101 delta).
• A shared SystemPromptSemantics Literal alias — type tightening rather than a rule.
• Not statically reachable: the load-order crash needs Pydantic's validate_assignment runtime behavior; the migration and A/B-lever items need semantic judgment.

Harness improvements

• A loader-level matrix over system_prompt × system_prompt_mode × system_prompt_file run through load_task — the construction-only test is what let the crash ship green.
• Add the antigravity extra to pr-checks.yml, or extract _build_local_agent_kwargs() so the forwarding is assertable without the SDK.
• An inspect.signature(thread_start) assertion for developer_instructions.
• A log.warning on _effective_prompt_mode's fail-open branch.

Top 5 Priority Actions

1. Fix the replace + system_prompt_file load crash — batch both field updates into one model_copy(update=...) in resolve_agent_system_prompt so no half-updated state is validated, then add a loader-level test for the combination.
2. Re-baseline or pin echo_simulated_judged.yaml — set system_prompt_mode: replace to preserve its baseline, or reword it for append semantics and note the re-baseline.
3. Add the migration surface — a "Migrating tasks that set system_prompt" callout in CLAUDE_CODE.md, a "Migration" note in CODEX.md (ignored → injected, with no opt-out on CodexAgentConfig), and a BREAKING CHANGE: footer so it lands in CHANGELOG.md.
4. Make the system_prompt_semantics contract true — give Agent.get_environment_info() a base default (or stamp it in the orchestrator) so every agent emits it, and correct REPORT_SCHEMA.md so "absent" means "unknown" rather than "pre-append regime".
5. Close the two small correctness gaps — reject empty/whitespace-only prompts under replace, and tighten the SubAgentRunner guard to require both a prompt and replace mode.

---

Change class: complex — it changes prompt-construction semantics on a shared config field across three agents, adds a validator and two guards, and alters a persisted run-record wire format; correctness requires reasoning about consumers outside the diff.
Stats: 0 🔴 · 2 🟠 · 7 🟡 · 6 🔵 across 8 axes reviewed. (Previous review: 1 🔴 · 4 🟠 · 2 🟡 · 5 🔵.)