Skip to content

v0.9.1: Make Codewhale upstream-ready for a Verifiers v0.2.1 harness #4641

Description

@Hmbown

Goal

Make the public codewhale exec surface reproducible enough for a small future pull request adding Codewhale as a built-in Verifiers v1 harness.

Codewhale only needs to own its launch contract. Verifiers owns interception, traces, model-call timing, token accounting, retries, rollout limits, and runtime orchestration. Do not add a Verifiers runtime, trace parser, receipt schema, or integration package to Codewhale.

Do not submit the external PR or publish anything in this issue.

Baseline

  • Codewhale: current origin/main when assigned (contract audited at 6cd2953956fd69406aa126f3f58e631332d90abe; rebase before review)
  • Verifiers: v0.2.1, commit ab65b6e8d34b03d162408d4bcb854430a86809e6

Verifiers v1 built-in harnesses install a pinned CLI in setup(), launch it against the supplied interception endpoint and secret in launch(), and call runtime.run_program(...). Match that existing shape.

Codewhale changes

  1. Add public --no-project-config support to the codewhale dispatcher and forward it before the TUI subcommand.
  2. Make codewhale --no-project-config exec ... skip the workspace-specific user-config overlay that Exec currently applies unconditionally.
  3. Document and test one isolated headless launch using existing generic surfaces:
    • an explicit temporary config with [providers.openai] base_url, model, and api_key_env;
    • CODEWHALE_HOME set to a fresh per-run directory;
    • CODEWHALE_SECRET_BACKEND=file;
    • CODEWHALE_MCP_CONFIG pointing to a generated per-run MCP JSON file;
    • CODEWHALE_MEMORY=false and CODEWHALE_TELEMETRY=false;
    • CODEWHALE_ALLOW_INSECURE_HTTP=1 only when Verifiers supplies a trusted http:// interception endpoint (container/tunnel endpoints are not always loopback);
    • exec --auto --sandbox danger-full-access --output-format stream-json;
    • --append-system-prompt and --disallowed-tools when supplied by the caller.

The interception secret must remain in the child environment. The temporary route config stores only the environment-variable name, never the secret:

provider = "openai"

[providers.openai]
base_url = "<interception-endpoint>"
model = "<model-id>"
api_key_env = "VF_CODEWHALE_API_KEY"

The generated MCP file contains only the task servers supplied by the caller:

{"mcpServers":{"task-tools":{"url":"<mcp-url>"}}}

The exact argument order is:

codewhale \
  --config .vf-codewhale/config.toml \
  --workspace . \
  --no-project-config \
  --skip-onboarding \
  exec \
  --auto \
  --sandbox danger-full-access \
  --output-format stream-json \
  -- "<prompt>"

Files

  • crates/cli/src/lib.rs — parse/forward --no-project-config and focused dispatcher tests
  • crates/tui/src/main.rs — honor the flag in Exec and focused unit tests
  • crates/tui/tests/verifiers_harness_contract.rs — provider-free launch/MCP acceptance
  • docs/AGENT_RUNTIME.md — concise public contract and future-upstream checklist

Do not change crates/tui/src/config.rs or crates/tui/src/mcp.rs unless the focused test finds a real defect; route-bound api_key_env, CODEWHALE_MCP_CONFIG, the mcpServers alias, and URL-based Streamable HTTP/SSE fallback already exist.

Acceptance

  • The public dispatcher accepts --no-project-config, forwards it before exec, and rejects misplaced usage clearly.
  • Exec does not apply a matching global [workspace]/[projects] overlay when that flag is present.
  • A provider-free test runs the real headless engine against a loopback OpenAI-compatible fixture from a fresh CODEWHALE_HOME and reaches the exact configured endpoint/model.
  • The same test loads only the generated MCP URL, completes one MCP tool round-trip, applies an appended system prompt and a denied-tool entry, and exits without input or a modal.
  • A sentinel interception secret is absent from Codewhale argv, stdout/stderr, stream JSON, and files; a fixture model failure exits nonzero.
  • No non-loopback request or credential is required.

Verification

cargo test -p codewhale-cli --locked no_project_config
cargo test -p codewhale-tui --bin codewhale-tui --locked no_project_config
cargo test -p codewhale-tui --test verifiers_harness_contract --locked
cargo test -p codewhale-tui --all-features --locked
cargo fmt --all -- --check
cargo clippy --workspace --all-targets --all-features --locked -- \
  -D warnings \
  -A clippy::uninlined_format_args \
  -A clippy::too_many_arguments \
  -A clippy::unnecessary_map_or \
  -A clippy::collapsible_if \
  -A clippy::assertions_on_constants
git diff --check

Cloud worker handoff

Start from a fresh clone of Hmbown/CodeWhale; fetch current origin/main, create agent/4641-verifiers-ready, record the base SHA, and read AGENTS.md, CONTRIBUTING.md, and crates/tui/AGENTS.md. Use temporary homes, loopback fixtures, and placeholder secrets only. No machine-local handoff, existing worktree, installed Codewhale, external checkout, live credential, or paid resource is required.

Use signed-off commits (git commit -s) and open one focused PR to Codewhale main with exact provider-free receipts. Rebase before review, wait for hosted CI, and resolve every review thread. Stop at the tested PR; maintainers own independent audit, merge, installed dogfood, publication, and the later upstream submission.

Future upstream PR (after the public v0.9.1 artifact exists)

The expected Verifiers change is limited to:

  • verifiers/v1/harnesses/codewhale/__init__.py
  • verifiers/v1/harnesses/codewhale/harness.py
  • verifiers/v1/harnesses/__init__.py
  • tests/v1/conftest.py (add codewhale to the existing harness matrix)
  • pyproject.toml (register the strict pytest marker)
  • docs/v1/harnesses.md

CodewhaleHarnessConfig pins 0.9.1. setup() downloads the matching GitHub Release archive plus codewhale-bundles-sha256.txt, verifies the archive, and installs it into the runtime. launch() writes the temporary route/MCP files above, conditionally allows the Verifiers-owned insecure HTTP endpoint, and calls runtime.run_program(...). Set APPENDS_SYSTEM_PROMPT = True, SUPPORTS_MCP = True, and leave message prompts/user simulation unsupported.

Reuse Verifiers' existing tests/v1/test_e2e.py::test_agentic matrix; upstream guidance explicitly says not to add unit tests. Do not add a Codewhale-specific trace parser.

Holdouts and out of scope

  • The upstream install path cannot be completed until a public, immutable Codewhale v0.9.1 GitHub Release and checksum manifest exist.
  • Opening/submitting the upstream PR and running its credentialed PRIME_API_KEY E2E matrix require separate approval. No live provider, hosted-runtime, or paid credential is used here.
  • Do not tag, publish, create a release, or mutate PrimeIntellect-ai/verifiers in this issue.
  • Do not claim runtime/architecture support until the exact released archive has run in that upstream runtime.

Metadata

Metadata

Assignees

No one assigned

    Labels

    agent-readySelf-contained for a fresh-clone cloud agent; implement in a tested PR with no live credentialsbugSomething isn't workingdocumentationImprovements or additions to documentationenhancementNew feature or requestrelease-blockerMust be fixed before the next releasereliabilityReliability, flaky behavior, retries, fallbacks, and robustnessteacher-harnessTeacher/student replay, GEPA-style promotion, and cached-main learningtoolsTool execution, tool schemas, tool UX, and built-in tool behaviortuiTerminal UI behavior, rendering, or interactionv0.9.1Targeting v0.9.1

    Projects

    Status
    Done

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions