Repository navigation
tests: pin the CLI surface Syber's Line 0 drives (contract 2 consumer-driven contract test) - #176
Conversation
…act 2)
Syber's SYBER_BUILDER=syberwork builder (SyberLabs/syber
lib/factory/syberwork-builder.mjs, main 5a4667f) runs each work order as
a Build Thread through this CLI and reads JSON fields back. Nothing here
tested that surface, so a renamed flag or field would break Syber's
factory with every SyberWork test green.
tests/test_syber_line0_contract.py runs Syber's exact argv in Syber's
order, from the worktree with no --repo/--home/--actor, with the Claude
Code stand-in as SYBERLABS_CLAUDE_BIN and ANTHROPIC_BASE_URL set as the
key proxy sets it, and asserts every field Syber reads: status.thread,
candidates[].{id,in_scope,evaluation,provider}, check --json
{acceptable,reason,checks[].{name,state}}, status.target (40-hex commit
in the repository) and status.accepted after accept, export as JSON, and
the recorded lane provider_3p. A second test pins the refusal path.
Test only. No product code changes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
Independent review (OpenAI gpt-6-astra): CHANGES REQUESTED at The change is narrowly scoped and covers the main success and refusal flows, but the successful check invocation leaves its exit status untested, and the PR admission field does not satisfy the repository rules. The new credential-environment setup also triggers your explicit review gate, despite using only a placeholder. I could not independently run the tests or verify the cited consumer source from the supplied material.
Response |
Review at 7ce4993 (medium, tests): the successful `check c1 --json` call captured its exit code without asserting it, so a CLI that printed an acceptable verdict but exited non-zero stayed green. Syber's builder reads only the JSON, but every other caller reads the exit. The refusal test already pins the non-zero side. Failing first: with syberlabs/cli.py's check made to `return 1` for an acceptable verdict, this test fails with `AssertionError: 1 != 0`; the test at 7ce4993 passed against the same mutation. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review at 7ce4993 (low, rules): the test set ANTHROPIC_API_KEY and ANTHROPIC_BASE_URL to synthetic values. The CLI does not need either for `propose --claude-code --bare --lane provider_3p`: the lane comes from Syber's flag, and the key and proxy URL are Syber's runtime environment, not this CLI's contract. The test now sets neither, removes any ANTHROPIC_API_KEY the host has (inside ClaudeCodeCase's hermetic environment patch, so it is restored afterwards), and asserts that the stand-in ran without an API key. The lane is still pinned: the export records provider_3p, and removing provider_3p from --lane's choices makes both tests error. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Dismissed because a newer commit was pushed; Greptile will re-review the current head.
…t (B2-2) Two tests from #175 (tests/test_syber_builder_seam.py at 7d08a7f) that #176 did not cover: - SyberBuilderDefaults, ported whole: Syber's steps as subprocesses in a job worktree made the way Syber makes it, in Syber's environment, with no --repo/--home/--actor and no inherited git identity. Asserts the default home is the worktree's .syberlabs, the thread branch is in the worktree's repository, the default actor is "developer", and nothing is written beside the worktree. Keeps #175's precondition that fails on a host whose system gitconfig sets user.email. - The proposer-provider test, observed on real stand-in runs instead of by mocking Thread.propose (the unit under test): bare, provider_3p with cost basis estimated, at most --max-calls calls, default tools without Bash/PowerShell/WebSearch/WebFetch, no --mcp-config, no GitHits signal. Each fails against a deliberate break (local, reverted): provider_3p billed, Bash in DEFAULT_TOOLS, --max-calls ignored, default actor changed, default home moved out of the worktree. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Dismissed because a newer commit was pushed; Greptile will re-review the current head.
|
Independent review (OpenAI gpt-6-astra): CHANGES REQUESTED at The tests substantially cover the requested CLI contract, including successful and refused candidates, provider options, and worktree defaults. However, the new fixture touches credential handling, which triggers the explicit review restriction, and the defaults test deliberately depends on host Git configuration. Test execution was not independently verified.
Response |
… a host git identity Review at 97e2ade, two mediums: - rules: the test no longer touches credential variables. Every CLI step now runs as a subprocess with an environment built from an allowlist (minimal_env: PATH, the Windows start variables, a temporary HOME/USERPROFILE/XDG, runThread's LANG/TERM/CI/PYTHONDONTWRITEBYTECODE, PYTHONPATH at this checkout, plus SYBERLABS_CLAUDE_BIN for propose). It is never a copy of os.environ. The setUp key removal and the stand-in key assertion are gone, and the file names no credential variable. - tests: SyberBuilderDefaults keeps its no-identity precondition but skips, naming the host property, when syberlabs' view of git (the host's system config) supplies user.email, instead of failing a correct checkout. Evidence (local, scratch edits reverted): with an identity simulated in the temp HOME, the 97e2ade test failed (`(0, 'someone@example.test') != (1, '')`), and this one reports `skipped "the host's system git config sets user.email, ..."`. With a host key set (length 106) the stand-in saw none. These breaks still fail the suite: check exits 1 on an acceptable verdict, in_scope dropped, provider_3p billed, default actor changed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Dismissed because a newer commit was pushed; Greptile will re-review the current head.
|
Independent review (OpenAI gpt-6-astra): CHANGES REQUESTED at The change is focused and provides substantial subprocess-based coverage of the consumer’s CLI contract. However, the refusal-path test leaves a subprocess exit-status gap that can hide a broken CLI interaction. Review was limited to the supplied diff; tests were not executed.
Response |
…ract test Review at 649df0d (medium, tests): the failing-change test captured propose's return code and then overwrote it. The CLI's contract is exit 0 when the search registers candidates that all fail: the propose branch prints them and falls through to `return 0` (syberlabs/cli.py:604-666, :778). Only a Rejected exits 2 (:356-361). The test now asserts exit 0, with stdout and stderr as diagnostics, and checks that the search line is printed. Bites: with propose made to raise after registering candidates (scratch, reverted), this test fails with `AssertionError: 1 != 0`; the 649df0d test passed against the same break. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Dismissed because a newer commit was pushed; Greptile will re-review the current head.
|
Independent review (OpenAI gpt-6-astra): CHANGES REQUESTED at The tests cover the requested CLI workflow, refusal path, provider options, and worktree defaults without changing production code. However, the stand-in command construction is not safe for paths containing spaces, undermining the repository’s required Windows support. This review is based on the supplied diff; tests were not executed.
Response |
…eep their boundaries Review at 1c27b10 (medium, tests): SYBERLABS_CLAUDE_BIN was built with " ".join, which splits a path containing spaces. The CLI runs that variable as a single argv item when it names a file (claude_code.claude_executable). Otherwise it splits it with providers.split_command, which is shlex in non-POSIX mode on Windows and keeps quotes inside the items, so a quoted command line (list2cmdline) cannot carry a spaced path there either; this was probed on this host. The test now writes a wrapper and names it by path: claude.cmd with a list2cmdline-quoted command plus %* on Windows, and an sh script with shlex.join plus "$@" on POSIX. A new test puts the stand-in and its wrapper under "claude code bin" and shows the provider launches it and the candidate passes. With the 1c27b10 construction (scratch, reverted) that test fails: `2 != 0 : error: invalid_provider: could not read Claude Code's version`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Independent review (OpenAI gpt-6-astra): APPROVED at The change stays within the requested contract-test scope. It exercises the consumer’s CLI arguments through subprocesses and checks success, refusal, provider configuration, path handling, and worktree defaults. The provider is replaced with a local stand-in; no credential, deployment, or outside-contact changes are present in the diff. No blocking correctness or repository-rule issues found. Review is based on the supplied diff; tests were not independently executed. No findings. Response |
Admission
Class: R
Agent-platform: claude
Reviewed-by: openai:gpt-6-astra (response resp_0facd6674a5cb82e006aeb6eaa26f087d0bbbaaed680d8148b, head 32525bf, tier critical, run by the PM (Claude) on behalf of sykosyber)
Admitted-by: self (Class R)
Merge-restriction: merging is Mateo's under the overnight rules; no agent merges this.
Evidence: tested
Summary
Syber's
SYBER_BUILDER=syberworkbuilder runs each work order as a Build Thread through this CLI and reads JSON fields back. The builder is in SyberLabs/syberlib/factory/syberwork-builder.mjs,runThread, read at main5a4667f. Syber'sfly.toml:18selects it. This is the only Syber-to-SyberWork interface that runs today. Before this PR nothing on this side tested it: renaming a flag or a JSON field would break Syber's factory while every SyberWork test stayed green.tests/test_syber_line0_contract.pyis a consumer-driven contract test on the producer side.--repo,--homeor--actor.minimal_env), never a copy of this process's environment:PATH, and the Windows start variables (SYSTEMROOT,WINDIR,COMSPEC,PATHEXT).HOME,USERPROFILEand XDG directories.runThreadsets:LANG,TERM,CI,PYTHONDONTWRITEBYTECODE.PYTHONPATHset to this checkout.SYBERLABS_CLAUDE_BIN, naming the Claude Code stand-in fromtests/test_claude_code.py.SyberLine0Contract.test_a_work_order_runs_through_the_steps_syber_drivesinit,start --class R,propose --claude-code --bare --lane provider_3p …,status --json,check --json,diff,acceptandexport. Every field Syber reads is asserted, including thatcheckexits 0 on an acceptable verdictSyberLine0Contract.test_a_failing_change_reaches_syber_as_no_passing_candidateacceptable: falsewith areasonand a non-zero exitSyberLine0Contract.test_syber_propose_builds_the_provider_it_asks_forprovider_3pwith cost basisestimated, and makes at most--max-callscalls. It uses the default tools, with no Bash, PowerShell, WebSearch or WebFetch, no--mcp-configand no GitHits. This is observed on real stand-in runs, not by mockingThread.proposeSyberBuilderDefaults.test_the_steps_run_on_the_defaults_from_the_job_worktree7d08a7f. The worktree is made the way Syber makes it. The default home is the worktree's.syberlabs, the thread branch is in the worktree's repository, the default actor isdeveloper, and nothing is written beside the worktree. If syberlabs' view of git (which includes the host's system git config) suppliesuser.email, the test is skipped, with the reason naming that host propertyRevision history
7ce4993, medium, testscheck c1 --jsonexits 0, with stdout and stderr as diagnosticscfbb6f6Admitted-by: self (Class R), with the merge restriction on its own linef4645dd97e2ade97e2ade, medium, rulesminimal_env. The setUp removal and the stand-in key assertion are gone, and the file names no credential variable649df0d649df0d649df0d, medium, testspropose's exit code: 0 when the search registers only failing candidates. The propose branch falls through toreturn 0(syberlabs/cli.py:604-666,:778), and only aRejectedexits 2 (:356-361). The test also asserts the search line is printed1c27b101c27b10, medium, tests (final round)SYBERLABS_CLAUDE_BINnow names a one-file wrapper by path:claude.cmd(alist2cmdline-quoted command plus%*) on Windows, and an sh script (shlex.joinplus"$@") on POSIX. The CLI runs a file path as one argv item (claude_code.claude_executable). A quoted command line cannot work on Windows:providers.split_commandisshlexin non-POSIX mode there, which keeps the quotes inside the split items (probed on this host). New test: the stand-in underclaude code binlaunches32525bfAcceptance criteria
PYTHONPATH=. timeout 300 python -m unittest -v tests.test_syber_line0_contractRan 5 tests in 20.850s,OK(Windows 11, Python 3.12.6)check c1 --jsonexits 1 with an acceptable verdictsyberlabs/cli.py:686changed toreturn 1(scratch, reverted)FAILED (failures=2):AssertionError: 1 != 0in the in-process and defaults testsin_scoperenamed insyberlabs/build.py(scratch)FAILED (failures=1, errors=2):KeyError: 'in_scope'--repo,--homeor--actorSyberBuilderDefaults.step; run the AC1 commandcwdset to the worktree and Syber's arguments only;okgit diff origin/main --name-onlytests/test_syber_line0_contract.pyAdmitted-by: self (Class R)grep -niE "anthropic|api_key|token|secret|password|credential" tests/test_syber_line0_contract.pyFalse)HOMEa.gitconfigwithuser.email, then run…SyberBuilderDefaultsskipped "the host's system git config sets user.email, so syberlabs inherits a git identity and the default actor cannot be observed on this host",OK (skipped=1). The97e2adeversion under the same scratch:AssertionError: Tuples differ: (0, 'someone@example.test') != (1, ''),FAILEDproposewith only failing candidates exits 0, and the test bitesRuntimeErrorafter registering candidates, then run…test_a_failing_change_reaches_syber_as_no_passing_candidateAssertionError: 1 != 0 : c1: 2 files, 557 diff bytes, provisional …,FAILED (failures=1). The649df0dtest printedOKagainst the same break; the unbroken test printsOK…test_a_stand_in_under_a_path_with_spaces_still_launches; scratch: the1c27b10" ".joinconstructionok. Under the scratch:AssertionError: 2 != 0 : error: invalid_provider: could not read Claude Code's version …,FAILEDFailing first: these breaks were run at
649df0dand each made the suite fail. All were scratch edits, reverted withgit checkout -- syberlabs.LANE_COST["provider_3p"]set to"billed":Tuples differ: ('provider_3p', 'billed') != ('provider_3p', 'estimated')."anonymous"(build.py:594):'anonymous' != 'developer'.These breaks, run at
97e2ade, also each failed it:"Bash"added toDEFAULT_TOOLS.--max-callsignored.--homemoved out of the worktree.status.acceptedrenamed.reasondropped fromcheck --json.provider_3premoved from the--lanechoices.Tests
tests/test_syber_line0_contract.py. Changed files: none.CI
97e2ade: all 26 checks pass, includingadmit(run 41854982721).32525bf:admitpasses (run 41871104874), as do test 3.11/3.12/3.13, compose and G2-G6.tests/test_console.py, which this PR does not touch:ConsoleWitness…TimeoutError: timed out;ConsolePreview.test_preview_appends_nothing_for_anyoneConnectionAbortedError [WinError 10053].test_builder_httpTimeoutError). The shard took 817 s against a planned 314 s.1c27b10: all 25 GitHub Actions checks pass, includingadmit(run 41866662145), windows and both G8 jobs. Greptile Review (a third-party app, the 26th check on earlier heads) posted no status on this head.649df0d: all 26 checks pass, includingadmit(run 41859118535), windows, and both G8 existing-suite jobs (strict and plain). G8 fails on any skip whose reason is not allowed, so the defaults test ran, not skipped, on those runners.Backlog
SYBERLABS_CLAUDE_BINgiven as a command line cannot carry a path with spaces on Windows (providers.split_commandkeeps quotes in non-POSIXshlex). That is a CLI fix, outside this allowlist; operators should name a single executable path.provider_3pkey pass-through is pinned intests/test_claude_code.pyas a follow-up (outside this allowlist).ALLOWED_SKIPS(scripts/g8_existing_suite.py:40) needs an entry for the host-identity skip reason, or a host whose git config setsuser.emailfails G8. That is a separate packet, outside this allowlist.user.emailrunsscripts/g8_existing_suite.py, G8 counts the new skip reason as "not allowed" (ALLOWED_SKIPS,scripts/g8_existing_suite.py:40), and that gate script is outside this allowlist. A plainunittestrun reports the test as skipped, not failed. CI runners show no identity: G8 passed on649df0d, so the defaults test ran there.Rollback
Revert the commits, or delete the file. It is test only and has no state, so retries and idempotency do not apply. #175 can be reopened.
Not verified
test/syberwork-builder.test.mjs, which was not run because it hangs on Windows.--name syber-line0-<k>) and theselectpath.Files outside the allowlist
None.
Class: R (
git diff origin/main --name-only→tests/test_syber_line0_contract.py. SyberWorkpolicy.jsonmatches no rule for it, so the floor is auto, and AGENTS.md section 1 classes a new unit test as R)Agent-platform: claude
Reviewed-by: pending: needs a reviewer on a platform other than claude
Review-tier: critical
🤖 Generated with Claude Code