Skip to content

Pin the syberlabs CLI surface Syber's SyberWork builder calls (contract 2's receiving end today) - #175

Closed
sykosyber wants to merge 3 commits into
mainfrom
claude/syber-builder-seam-test
Closed

sykosyber wants to merge 3 commits into
mainfrom
claude/syber-builder-seam-test

Conversation

@sykosyber

@sykosyber sykosyber commented Nov 4, 2026 •

Copy link
Copy Markdown
Member

Admission

Class: R
Agent-platform: claude
Reviewed-by: pending: needs a reviewer on a platform other than claude
Admitted-by: self (Class R)
Evidence: tested

Summary

This PR pins, inside SyberWork, the exact syberlabs CLI surface that Syber's SyberWork builder depends on. That surface is the Syber → SyberWork half of contract 2 as it runs today. SyberLabs/syber lib/factory/syberwork-builder.mjs (read at 5a4667f) runs these steps in the job worktree, with Syber's own environment and no --repo, --home or --actor: init (:399), start <objective> --class R (:402), propose --claude-code --bare --lane provider_3p … (:408), status --json (:411), check <id> --json (:435), diff <id> (:446), accept <id> (:455), status --json (:460) and export (:417). Before this PR, no SyberWork test named that consumer.

tests/test_syber_builder_seam.py has two classes:

  • SyberBuilderSeam (3 tests, in-process). It checks every JSON field and argv Syber reads:

    • a passing candidate end to end;
    • a failing candidate;
    • Syber's exact propose line builds a real ClaudeCodeProvider (bare, provider_3p, no GitHits, no MCP config, no shell or web tool). No Claude Code runs: the executable only answers --version, and the search is stopped before any call.
  • SyberBuilderDefaults (1 test, added for the review finding at c0f292b). It runs each step as a subprocess with cwd set to a temporary job worktree:

    • Syber's environment (mjs:350-364), with a temporary HOME, USERPROFILE and XDG directories and no inherited git identity;
    • the disposable repository made exactly as Syber makes it (mjs:389-393);
    • then init, start --class R, status --json, check --json, accept, status --json and export, with no injected options.

    It asserts:

    • the default home is the worktree's .syberlabs;
    • repository discovery resolves to the worktree's repo, and the thread branch tip equals target;
    • the default actor is developer when there is no user.email, rather than the commit's inline identity;
    • nothing is written beside the worktree.

Acceptance criteria

AC Command Result
AC1: every field and argv Syber reads is asserted PYTHONPATH=. python -m unittest tests.test_syber_builder_seam -v Ran 4 tests … OK
AC2 (review finding): the steps run from a temporary job worktree with the consumer's exact arguments (mjs lines cited) and no --repo, --home or --actor, from init through the later state reads same command, test SyberBuilderDefaults.test_the_steps_run_on_the_defaults_from_the_job_worktree ok
AC3: the default home and identity are isolated from the real environment the precondition inside that test runs git config user.email with the same environment syberlabs gives git (gitspace.Repo._env drops GIT_*) exit 1, empty
AC4: the test fails when default repo discovery breaks temporarily change --repo default "." to ".." in syberlabs/cli.py, run AC1's command, then revert FAIL … AssertionError: 2 != 0 : syberwork-builder.mjs:399 (the other 3 tests stay green)
AC5: it also fails on default-home and default-actor regressions --home default changed to .syberlabs-moved; actor fallback "developer" changed to "anonymous"; both reverted AssertionError: False is not true : the default home is the worktree's .syberlabs; AssertionError: 'anonymous' != 'developer'
AC6: the in-process tests fail on drift in_scope renamed in Thread.status; ClaudeCodeProvider made to refuse --bare with provider_3p; both reverted test 1 errors (KeyError); test 3 fails

Threat model

The change is test-only, and no runtime code changes. The surface it pins includes:

  • the lane and the tool list that Syber's proposer relies on (no shell, no web lookups, provider_3p through Syber's key proxy);
  • the default actor attribution when the host has no identity.

The tests read no credential and print no environment value. The subprocess environment is built from scratch, not copied from the caller.

Tests

  • File touched: tests/test_syber_builder_seam.py (new). No existing test, fixture or gate was edited.
  • Windows 11, Python 3.12: 4 run, 4 passed, 0 failed.
  • Failing-first evidence: this PR adds tests for behavior already on main, so "failing first" is shown by temporarily breaking the unit under test. Output from AC4, with the change applied:
    FAIL: test_the_steps_run_on_the_defaults_from_the_job_worktree (tests.test_syber_builder_seam.SyberBuilderDefaults...)
    AssertionError: 2 != 0 : syberwork-builder.mjs:399
    Ran 4 tests in 5.601s
    FAILED (failures=1)
    
  • A fresh-context, read-only review subagent checked the delta. It found that GIT_CONFIG_NOSYSTEM never reached syberlabs' own git calls, because gitspace.Repo._env drops GIT_*. Fixed in 7d08a7f: the precondition now asks git with syberlabs' view, and the docstring states that the host's system config is part of that view. It also noted that propose uses --from-worktree in the defaults test. That stays as is: _kit resolves repo, home and actor the same way for every propose source, and Syber's propose argv is pinned in SyberBuilderSeam.

CI

See the checks tab for head 7d08a7f, which runs the full suite, the gates and admit.

Rollback

Revert the PR. It adds one test file and changes no runtime behavior.

Not verified

  • These tests run Python as python -m syberlabs, while Syber runs the syberlabs console script.
  • They do not run inside bwrap.
  • They do not cover Syber's second round, where .git or .syberlabs already exists (mjs:390, 398).
  • A host whose system gitconfig sets user.email fails AC3 loudly instead of passing silently.
  • I did not run the full suite locally.

Files outside the allowlist

None. The only file touched is tests/test_syber_builder_seam.py.

Review-tier is critical even though the change is test-only, because it pins the contract 2 (S0) seam, the proposer's lane and tool list, and default actor attribution.

Class: R (policy.json floors over git diff --name-only origin/main...HEAD: tests/test_syber_builder_seam.py → auto (unmatched); AGENTS.md §1: a new unit test is R)
Agent-platform: claude
Reviewed-by: pending: needs a reviewer on a platform other than claude
Review-tier: critical

🤖 Generated with Claude Code

RetriggerView in GreptileConfidence Score: 5/5

The PR appears safe to merge, with non-blocking test reliability and coverage issues.

Fix All in Claude CodeFindings

  1. P2 Machine settings break the test ▶
  2. P2 Scoped shell tools escape detection ▶
Fix with agent prompt
### Issue 1
tests/test_syber_builder_seam.py:210-211
If the host has a system-level `user.email`, this assertion fails before any CLI step runs, even though the CLI behaves correctly. `Repo._env()` removes `GIT_CONFIG_NOSYSTEM` and `GIT_CONFIG_GLOBAL`, so the temporary home does not isolate that setting.

Make the fixture independent of system Git settings while keeping the CLI defaults under test.

### Issue 2
tests/test_syber_builder_seam.py:152-153
The assertion only rejects exact tool names. An entry such as `Bash(git *)` passes, but `built_in_tools()` turns it into `Bash` when building the command. The constructor permits shell tools on this path because neither GitHits nor an MCP config is enabled.

Check scoped entries too, or assert the expected tool list explicitly. Otherwise, adding restricted shell access could leave this no-shell test green.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

Adds tests for the CLI commands and fields Syber's builder uses.

  • Syber's candidate tests follow results through acceptance.
  • Syber's CLI test pins the provider built from its exact flags.
  • Syber's CLI tests run from the job worktree defaults.

Reviews (2) · Last reviewed commit: "Seam test: check the identity preconditi..." · Reviewed by Greptile

…ct 2's receiving end today)

SyberLabs/syber lib/factory/syberwork-builder.mjs runs each job as a Build
Thread by shelling out to syberlabs: start --class R, propose --claude-code
--bare --lane provider_3p, status --json, check --json, diff, accept, export.
Nothing in this repository named that consumer, so a change to a JSON field
or a new propose refusal would surface in Syber first. Three tests pin exactly
the fields and argv Syber relies on: a passing candidate end to end, a failing
one, and Syber's propose line building the provider it asks for (no Claude
Code runs; the executable answers --version only and the search is stopped
before any call). Test-only; no behavior change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 4, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-04T14:25:48.060646Z c0f292b PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c0f292baf0

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

self.assertEqual((provider.name, provider.revision, provider.max_calls), ("syber-line0", "1", 2))
self.assertIs(provider.githits, False)
self.assertEqual(provider.mcp_config, None)
self.assertFalse(any(re.match(r"^(Bash|PowerShell|WebSearch|WebFetch)$", t) for t in provider.allowed_tools),

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Reject scoped shell and web tool entries

If the default tool list later includes a scoped entry such as Bash(git log *), this exact-name regex will not recognize it, so the seam test can pass despite allowing a shell tool. Check the base tool names (or otherwise cover scoped forms) so the test catches that access widening.

AGENTS.md reference: AGENTS.md:L212-L215

Useful? React with 👍 / 👎.

Comment on lines +146 to +147
self.assertFalse(any(re.match(r"^(Bash|PowerShell|WebSearch|WebFetch)$", t) for t in provider.allowed_tools),
"Syber's builder relies on the default tools: no shell, no web lookups")

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Scoped shell tools escape detection

The assertion only rejects exact tool names. An entry such as Bash(git *) passes, but built_in_tools() turns it into Bash when building the command. The constructor permits shell tools on this path because neither GitHits nor an MCP config is enabled.

Check scoped entries too, or assert the expected tool list explicitly. Otherwise, adding restricted shell access could leave this no-shell test green.

Prompt To Fix With AI
This is a comment left during a code review.
Path: tests/test_syber_builder_seam.py
Line: 146-147

Comment:
**Scoped shell tools escape detection**

The assertion only rejects exact tool names. An entry such as `Bash(git *)` passes, but `built_in_tools()` turns it into `Bash` when building the command. The constructor permits shell tools on this path because neither GitHits nor an MCP config is enabled.

Check scoped entries too, or assert the expected tool list explicitly. Otherwise, adding restricted shell access could leave this no-shell test green.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Fix in Claude Code

@sykosyber

Copy link
Copy Markdown
Member Author

Independent review (OpenAI gpt-6-astra): CHANGES REQUESTED at c0f292b

The change is focused and adds useful success, failure, and provider-construction coverage without changing runtime code. However, it does not fully pin the invocation contract it documents: every test supplies repository and home options that Syber explicitly omits. Review is based on the supplied diff; tests were not independently run.

  • medium / tests tests/test_syber_builder_seam.py:SyberBuilderSeam.run_cli: The documented consumer starts the CLI in the job worktree without --repo or --home, but this helper always supplies both, as well as --actor. Consequently, regressions in default repository discovery, state-directory selection, or actor resolution can break Syber while all three tests remain green. Add an isolated test that runs from the temporary job worktree using the consumer's actual arguments and exercises init/start through subsequent state reads without these injected options. Keep any default home or identity setup isolated from the developer's real environment.

Response resp_02a3c5818e3866be006aeb43f13e9087d1b2e246be156c0fe1. Tier: critical. Reviewer: Syber's lib/factory/reviewer.mjs (a433c4b), run by the PM (Claude) on behalf of sykosyber. Author platform: claude.

sykosyber and others added 2 commits November 4, 2026 06:44
… worktree

Review finding on #175: run_cli always passed --repo, --home and --actor,
but Syber's builder (syberwork-builder.mjs:350-370) runs the CLI in the job
worktree with none of them. SyberBuilderDefaults runs the steps as
subprocesses in a temporary worktree with Syber's environment, a temporary
HOME, USERPROFILE and XDG directories, an empty global git config and no
system config: the disposable repository as Syber makes it, init, start
--class R, status --json, check --json, accept, status --json and export.
It asserts the default home, repository discovery and the default actor.

Fails first: --repo default '.' -> '..' fails it at init (exit 2), --home
default moved fails the default-home assertion, and the actor fallback
renamed fails the actor assertion; all three reverted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…git config

Independent read of the previous commit: syberlabs drops every GIT_* variable
before running git (gitspace.Repo._env), so GIT_CONFIG_NOSYSTEM did not reach
its `git config user.email`. The precondition now asks git with the same
environment syberlabs gives it, and the docstring says the host's system
config is in that view, as on a Syber host.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Comment on lines +210 to +211
self.assertEqual((done.returncode, done.stdout.strip()), (1, ""),
"no git identity in syberlabs' view (a host system gitconfig with user.email would set one)")

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Machine settings break the test

If the host has a system-level user.email, this assertion fails before any CLI step runs, even though the CLI behaves correctly. Repo._env() removes GIT_CONFIG_NOSYSTEM and GIT_CONFIG_GLOBAL, so the temporary home does not isolate that setting.

Make the fixture independent of system Git settings while keeping the CLI defaults under test.

Prompt To Fix With AI
This is a comment left during a code review.
Path: tests/test_syber_builder_seam.py
Line: 210-211

Comment:
**Machine settings break the test**

If the host has a system-level `user.email`, this assertion fails before any CLI step runs, even though the CLI behaves correctly. `Repo._env()` removes `GIT_CONFIG_NOSYSTEM` and `GIT_CONFIG_GLOBAL`, so the temporary home does not isolate that setting.

Make the fixture independent of system Git settings while keeping the CLI defaults under test.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Fix in Claude Code

sykosyber added a commit that referenced this pull request Nov 4, 2026
…t (B2-2)

Two tests from #175 (tests/test_syber_builder_seam.py
at 7d08a7f) that #176 did not cover:

- SyberBuilderDefaults, ported whole: Syber's steps as subprocesses in a
  job worktree made the way Syber makes it, in Syber's environment, with
  no --repo/--home/--actor and no inherited git identity. Asserts the
  default home is the worktree's .syberlabs, the thread branch is in the
  worktree's repository, the default actor is "developer", and nothing
  is written beside the worktree. Keeps #175's precondition that fails on
  a host whose system gitconfig sets user.email.
- The proposer-provider test, observed on real stand-in runs instead of
  by mocking Thread.propose (the unit under test): bare, provider_3p with
  cost basis estimated, at most --max-calls calls, default tools without
  Bash/PowerShell/WebSearch/WebFetch, no --mcp-config, no GitHits signal.

Each fails against a deliberate break (local, reverted): provider_3p
billed, Bash in DEFAULT_TOOLS, --max-calls ignored, default actor
changed, default home moved out of the worktree.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@sykosyber

Copy link
Copy Markdown
Member Author

Superseded by #176 (B2-2, gestalt plan): its unique tests are ported there.

@sykosyber sykosyber closed this Nov 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant