chore: uniform pbench prompts and longrun matched-model rerun#207
Merged
Conversation
Add tools.exec.extra_deny_patterns for host-specific deny regexes when running un-sandboxed; product default deny list unchanged. Co-authored-by: Claude (claude-opus-4-8) <noreply@anthropic.com>
All agents on openrouter qwen3.5-27b; add hermes CLI pbench runner, uniform prompts, run_meta/soft_dnd stamps; refresh READMEs to v3. Co-authored-by: Claude (claude-opus-4-8) <noreply@anthropic.com>
userName20260323
changed the title
cyfyifanchen
approved these changes
Jul 24, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two-part refresh of the proactivity eval, plus a small product fix.
1. pbench: uniform prompt re-test. All three agents re-tested with the
same minimal task prompt (no persona, no decision principles -
runners/prompts/uniform/), each driven black-box through its own CLI onthe same backend (OpenRouter
qwen/qwen3.5-27b). The README main table isnow the 3-run mean under this protocol. New
hermes_cli_pbench.pydrivespbench through the installed hermes CLI; legacy per-agent persona templates
are kept only to reproduce the ablation appendix.
2. longrun: matched-model rerun. Full rerun of the 6-persona x
30-day x 3-agent benchmark as a single batch: raven, hermes, and openclaw
all on OpenRouter
qwen/qwen3.5-27b, withclaude-sonnet-4.5as usersimulator and judge, so the comparison isolates framework architecture
rather than model quality. Supporting harness changes:
run_metatrajectory stamp + scorecardsoft_dndfield so every artifactself-identifies its run mode.
repeat(daily/weekdays/weekly) for openclaw.sonnet judge for Type B, weekend quota aligned to factory default.
with per-dimension caveats disclosed.
3. Product fix. Adds
tools.exec.extra_deny_patterns: operators or evalharnesses running the agent un-sandboxed can append host-specific deny
regexes. The shipped default deny list is unchanged; the eval injects
osascript /
open -ablocks via config only.Type
Verification
uv run pytest tests/test_sandbox_unit.py -q- 68 passed (also re-runafter rebasing onto latest
main).uv run ruff check/uv run ruff format --checkon all changed Pythonfiles - clean.
pbench: 3 independent n=120 runs per agent under the uniform prompt
(2026-07-22); README table reports the mean with per-run F1 range.
longrun: 18 legs completed with zero LLM/API errors; scored via
longrun_scorecard.py --all --compare --aggregate(18/18 scorecards,single judge pass).
Relevant tests pass locally
Relevant lint / type checks pass locally
User-facing docs or screenshots are updated when needed
Risk
extra_deny_patternsis append-only on top of the built-in deny list anddefaults to empty, so shipped behavior is unchanged; both commits revert
cleanly and independently.
Related Issues
N/A