fix(eval): trial log dirs are world-writable

[SebTardif]

What does this PR do?

Trial directories created for native eval are no longer world-writable (0o777). They are created as 0o755 so only the owner can write logs and artifacts.

Why?

DockerTaskEnvironment.start chmod'd agent, verifier, and artifacts/logs/artifacts to 0o777. On a shared host another user could plant or rewrite trial logs. Cleanup paths in the same tree already use 0o755 for directories.

Changes

• Add prepare_trial_dirs() and chmod 0o755 instead of 0o777
• Cover the mode in test_prepare_trial_dirs_are_owner_writable_only

Tests

• Focused check tests/test_native_eval_runner.py::test_prepare_trial_dirs_are_owner_writable_only passes locally
• python -m ruff check clawbench app.py scripts tests passes locally

Red: with 0o777 the new check reports mode 511 (0o777) instead of 493 (0o755). Green: dirs are 0o755.

$ python3 -m pytest -q tests/test_native_eval_runner.py::test_prepare_trial_dirs_are_owner_writable_only
1 passed
$ python3 -m ruff check scripts/native_eval/runtime.py tests/test_native_eval_runner.py
All checks passed!

What Problem This Solves

Fixes an issue where native-eval trial log directories were world-writable, so another user on the same host could plant or rewrite agent, verifier, or artifact logs.

Evidence

Terminal output from the patched helper:

$ python3 -m pytest -q tests/test_native_eval_runner.py::test_prepare_trial_dirs_are_owner_writable_only
1 passed
mode 0o755

Real behavior proof

• Behavior or issue addressed: Native-eval start created trial dirs with mode 0o777, which lets any local user write logs.

• Real environment tested: macOS 15, Python 3.14, worktree /tmp/sb-chmod on fix/trial-dir-mode.

• Exact steps or command run after this patch:

  python3 -m pytest -q tests/test_native_eval_runner.py::test_prepare_trial_dirs_are_owner_writable_only

• Evidence after fix: terminal output from the patched helper. Trial dirs are owner-writable only:

  1 passed
  mode 0o755

• Observed result after fix: prepare_trial_dirs creates agent, verifier, and artifacts/logs/artifacts as 0o755.

• What was not tested: A live docker compose trial against a remote registry.

[clawsweeper]

🦞👀
ClawSweeper picked this up.

Pull request received. I will update this pull request when review starts.

[clawsweeper]

Codex review: needs real behavior proof before merge. Reviewed August 23, 2026, 12:58 PM ET / 16:58 UTC.

ClawSweeper review

What this changes

This PR changes native-evaluation trial directories from mode 0777 to 0755 and adds a regression test for those modes.

Merge readiness

⛔ Blocked until real behavior proof from a real setup is added - 7 items remain

Keep open: the previously reported P1 remains. The tightened host-owned directories prevent the image-declared non-root user from writing required native-evaluation logs.

Priority: P1
Reviewed head: f88349413ce4dbd046603c3b74851328af0fec02
Owner decision: Required. See Decision needed.

Review scores

Measure	Result	What it means
Overall readiness	🧂 unranked krab (1/6)	The hardening goal is valid, but the patch has a confirmed compatibility defect and lacks production-boundary proof.
Proof confidence	🦪 silver shellfish (2/6)	Needs real behavior proof before merge: The body shows only a helper-level pytest result; add redacted evidence from a native Docker trial with an image-declared non-root user successfully writing the affected mounts. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.
Patch quality	🦪 silver shellfish (2/6)	1 actionable review finding remain.

Verification

Check	Result	Evidence
Real behavior	Needs proof	Needs real behavior proof before merge: The body shows only a helper-level pytest result; add redacted evidence from a native Docker trial with an image-declared non-root user successfully writing the affected mounts. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.
Evidence reviewed	5 items	PR changes mount access: The proposed mode is 0755 for all three bind-mounted log directories, which makes the host process owner the only directory writer. Harnesses write inside the container: Current main redirects OpenClaw, Hermes, Codex, and Claude Code output to /logs/agent, while the verifier writes under /logs/verifier. Execution preserves image user: The runtime adds --user only when explicitly supplied, but normal harness and verifier calls omit it, so Docker uses the task image's configured user.
Findings	1 actionable finding	[P1] Preserve writes for non-root task containers
Security	None	None.

Live Verification

Command: python3 -c "import stat,tempfile; from pathlib import Path; from scripts.native_eval.runtime import DockerTaskEnvironment; d=tempfile.TemporaryDirectory(); e=DockerTaskEnvironment(object(), Path(d.name), 'trial', 'trial', Path(d.name)); e.prepare_trial_dirs(); print(oct(stat.S_IMODE(e.agent_dir.stat().st_mode)))"

Result: FAIL (failed) — execution before step 1 run: sh -lc pnpm install --ignore-scripts --frozen-lockfile failed: ! Corepack is about to download https://registry.npmjs.org/pnpm/-/pnpm-11.23.0.tgz

sh -lc pnpm install --ignore-scripts --frozen-lockfile failed: ! Corepack is about to download https://registry.npmjs.org/pnpm/-/pnpm-11.23.0.tgz

Assertions:

• FAIL expect_output: 0o755

How this fits together

Native evaluation creates host-side trial folders and bind-mounts them into task containers for harness logs, verifier results, and artifacts. Harness commands run inside the task image and must be able to write those mounted paths.

flowchart LR
A[Native evaluation job] --> B[Trial directory setup]
B --> C[Host bind-mount folders]
C --> D[Task image user]
D --> E[Harness and verifier logs]
E --> F[Trial results]

Loading

Decision needed

Question	Recommendation
Should native evaluation continue supporting task images that declare a non-root user, and what least-privilege UID/GID or ACL mechanism should grant those users access to trial mounts?	Preserve non-root task support: Define per-container least-privilege mount ownership or access and prove an unprivileged image can write required logs.

Why: The patch changes the security boundary between host users and arbitrary task-container users; forcing root execution or defining per-container mount ownership changes the evaluator's supported execution contract.

Before merge

• Add real behavior proof - Needs real behavior proof before merge: The body shows only a helper-level pytest result; add redacted evidence from a native Docker trial with an image-declared non-root user successfully writing the affected mounts. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.
• Preserve writes for non-root task containers (P1) - docker exec uses the task image's configured user unless passed --user, while every harness redirects output into /logs/agent and the verifier writes /logs/verifier. These host-owned 0755 bind mounts reject an ordinary image user, so valid non-root tasks fail before producing logs. Retain least-privilege write access for that user and add an unprivileged-container regression test.
• Resolve merge risk (P1) - Merging this mode change can make any supported task image with a non-root Dockerfile or Compose user fail before it can write harness or verifier output.
• Complete next step (P2) - Maintainers must choose the evaluator's least-privilege container-user contract before a safe repair can be selected.
• Improve patch quality - Implement a least-privilege write policy that preserves image-declared non-root users.
• Improve patch quality - Post redacted native-evaluation evidence showing non-root harness and verifier writes succeed; update the PR body to trigger re-review.

Findings

• [P1] Preserve writes for non-root task containers — scripts/native_eval/runtime.py:113

Agent review details

Security

None.

Review metrics

Metric	Value	Why it matters
Permission surface	3 bind-mounted directories change from 0777 to 0755	All native-evaluation harnesses depend on these directories for logs or results.
Code and test delta	production +5, -2; tests +21	The focused test covers mode bits but not the container write path affected by the production change.

Merge-risk options

Maintainer options:

1. Preserve unprivileged-container writes (recommended)
   Revise the directory-access mechanism and add a regression using an image-declared non-root user that writes agent and verifier outputs.
2. Pause for the evaluator contract decision
   Do not merge until maintainers decide whether root-only task execution is an intentional supported-policy change.

Technical review

Best possible solution:

Use a least-privilege mount-access design that grants each configured container user the required write access without restoring host-wide write access, and cover it with an unprivileged-container regression test.

Do we have a high-confidence way to reproduce the issue?

Yes, source-reproducible: run any harness in a task image declaring an unprivileged USER; its required write to /logs/agent or /logs/verifier is denied by a host-owned 0755 mount.

Is this the best way to solve the issue?

No. Replacing 0777 with host-owned 0755 removes required writes for supported image users; a user-aware least-privilege mount policy is safer.

Full review comments:

• [P1] Preserve writes for non-root task containers — scripts/native_eval/runtime.py:113
docker exec uses the task image's configured user unless passed --user, while every harness redirects output into /logs/agent and the verifier writes /logs/verifier. These host-owned 0755 bind mounts reject an ordinary image user, so valid non-root tasks fail before producing logs. Retain least-privilege write access for that user and add an unprivileged-container regression test.
  Confidence: 0.99

Overall correctness: patch is incorrect
Overall confidence: 0.99

AGENTS.md: not found in the target repository.

Codex review notes: model internal, reasoning high; reviewed against 884dd1bb5511.

Labels

Label changes:

• add rating: 🧂 unranked krab: Overall readiness is 🧂 unranked krab; proof is 🦪 silver shellfish and patch quality is 🦪 silver shellfish.
• remove rating: 🦪 silver shellfish: Current PR rating is rating: 🧂 unranked krab, so this older rating label is no longer current.

Label justifications:

• P1: The proposed permissions can prevent native-evaluation trials from producing required logs and verifier results.
• merge-risk: 🚨 compatibility: The change breaks task images whose configured user differs from the host runner owner.
• merge-risk: 🚨 availability: Permission-denied writes can cause trial setup, harness execution, or verification to fail.
• rating: 🧂 unranked krab: Overall readiness is 🧂 unranked krab; proof is 🦪 silver shellfish and patch quality is 🦪 silver shellfish.
• status: 📣 needs proof: The PR needs real behavior proof before ClawSweeper can clear the contributor ask. Needs real behavior proof before merge: The body shows only a helper-level pytest result; add redacted evidence from a native Docker trial with an image-declared non-root user successfully writing the affected mounts. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.

Evidence

What I checked:

• PR changes mount access: The proposed mode is 0755 for all three bind-mounted log directories, which makes the host process owner the only directory writer. (scripts/native_eval/runtime.py:113, f88349413ce4)
• Harnesses write inside the container: Current main redirects OpenClaw, Hermes, Codex, and Claude Code output to /logs/agent, while the verifier writes under /logs/verifier. (scripts/native_eval/harnesses.py:601, 884dd1bb5511)
• Execution preserves image user: The runtime adds --user only when explicitly supplied, but normal harness and verifier calls omit it, so Docker uses the task image's configured user. (scripts/native_eval/runtime.py:400, 884dd1bb5511)
• Regression test misses container contract: The new test only checks host-side mode bits; it does not start an image with an unprivileged USER and write the mounted agent or verifier paths. (tests/test_native_eval_runner.py:1264, f88349413ce4)
• Native-eval ownership history: Current-main history shows Vincent Koc authored the two recent native-eval runtime and harness changes that define this execution path. (scripts/native_eval/runtime.py:84, 884dd1bb5511)

Likely related people:

• Vincent Koc: Authored the recent current-main native-evaluation runtime and harness work that establishes the bind-mount and execution-user behavior. (role: recent native-eval contributor; confidence: high; commits: 884dd1bb5511, b9acd9f7a010; files: scripts/native_eval/runtime.py, scripts/native_eval/harnesses.py)

Rating scale

Score	Internal tier	Crab rank	Meaning
6/6	S	🦀 challenger crab	Exceptional readiness
5/6	A	🦞 diamond lobster	Very strong readiness
4/6	B	🐚 platinum hermit	Good normal PR; ordinary maintainer review
3/6	C	🦐 gold shrimp	Useful, but confidence is limited
2/6	D	🦪 silver shellfish	Proof or implementation needs work
1/6	F	🧂 unranked krab	Not merge-ready
N/A	NA	🌊 off-meta tidepool	Rating does not apply

Overall follows the weaker of proof and patch quality.
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

Workflow

• ClawSweeper keeps one durable marker-backed review comment per issue or PR.
• Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
• A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
• PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
• Maintainers can also comment @clawsweeper review to request a fresh review only.
• Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
• Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
• Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

History

Review history (1 earlier review cycle)

• reviewed 2026-08-16T23:22:19.185Z sha f883494 :: needs real behavior proof before merge. :: [P1] Preserve writes for non-root task containers