Skip to content

Quote holds with the run's options and accept webhook binding keys - #51

Open
rawwerks wants to merge 1 commit into
mainfrom
fix/hold-quote-parameters
Open

rawwerks wants to merge 1 commit into
mainfrom
fix/hold-quote-parameters

Conversation

@rawwerks

@rawwerks rawwerks commented Oct 7, 2026

Copy link
Copy Markdown

I'm an AI agent (Claude Code) opening this PR on behalf of Ray (@rawwerks). He asked for this fix and approved opening the PR, but he did not write the text below. The service-side facts come from the hosted service's maintainers.

Why

The hosted service's run hold is no longer flat. It depends on the model, reasoning effort, environment, declared tools and whether a repository is bound. GET /run/quote accepts model, reasoning_effort, environment, repositories and tools. With no parameters it quotes the default model at its default effort: builtin, no repositories, no tools.

The CLI still behaved as if the hold were flat:

  • Hardcoded wording. It printed holdBasis: "flat hold, independent of program and model", "The same for every program and model", and "the hold does not depend on it" on --model, --reasoning-effort and --repo. The guide, SPEC and docs said the hold is flat and "currently $1.02".
  • Ignored options. run quote, the run submit / run create plan and the job create plan quoted without parameters. run quote --model <cheaper model> therefore showed the default model's hold.

Separately, job create refused model, reasoning_effort and repository_url in webhook specs, though the service accepts them on a webhook with a program_ref. A webhook that reviews pull requests on a given model with a read-only repository therefore could not be created from the CLI.

What changes

Quote requests. Both ports send only what the user gave; nothing is defaulted.

  • run quote and the run submit / run create plan: --model → model, --reasoning-effort → reasoning_effort, --environment → environment, and repositories=1 for any --repo or --commit-output.
  • job create plan: the spec's nonempty model, reasoning_effort and environment, and repositories=1 for a nonempty repository_url or context_repository_url. This applies to every job type.
  • What stays the same:
    • A flagless quote stays parameter-free.
    • program draft still quotes without parameters.
    • Tools are never sent, because the CLI does not inspect the program.
    • The plan quote stays advisory for run submit and required for job create, as before.

Wording.

  • holdBasis is now depends on model, reasoning effort, environment, declared tools and repositories; quoted from the options given, without the program's own run settings or declared tools; a run's price is known only after it settles.
  • The human run quote output gains a Depends on: line.
  • The help, guide, SPEC and docs/service/* match. A program that sets its own model, effort or environment, or declares tools, can hold a different amount, and they all say so.

Webhook spec. The job.create webhook variant accepts model, reasoning_effort, repository_url, repository_branch and output, with the same rules as a schedule. Without a program_ref the service refuses these keys, and its message reaches the user as details.serviceMessage.

Files.

  • Shared contract:
    • The manifest: quote query templates, the run.quote and job.create text, and the webhook spec keys.
    • The runs.schema.json holdBasis const.
    • The regenerated help.v1.json and service-guide-* cases.
    • The three hand-maintained framework cases with the manifest digest (service-capabilities-json, service-capabilities-jsonl, help-json-group-records), refreshed from identical Rust and Bun output.
  • Corpus:
    • Nine edited runs/* cases.
    • Six new cases: runs/quote-sends-hold-options, runs/quote-commit-output-only, runs/submit-preview-repo-quote, jobs/job-create-webhook-binding-quote, jobs/job-create-schedule-environment-quote and jobs/job-create-schedule-empty-strings-quote. Each fixture pins the exact quote query.
  • Rust: service/{runs,jobs,render,programs}.rs.
  • Bun: service/{http,runs,jobs,render,programs}.ts.
  • cli/CHANGELOG.md: an [Unreleased] entry.

Compatibility notes

  • holdBasis text. The schema const is a visible contract change.
  • run quote --model. It still does no local model check. An invalid model now comes back from the service as SERVICE_REQUEST_REJECTED, where before the option was silently ignored.
  • A follow-up is possible. The service maintainers are adding a program-aware quote (program_ref, and a POST /run/quote that takes the run body). Once it lands, the CLI can include a program's own settings and declared tools without parsing it.
  • Overlap with Document webhook delivery signing and fix endpoint_url name #49. That PR also edits guide.v1.md and the service-guide-* cases. Whichever merges second needs python3 cli/ci/render_service_help.py --write.

Evidence

Local, on macOS with Python 3.10.21, Rust from rust-toolchain.toml and Bun 1.4.2. These gates pass:

  • shared-contracts, service-help, public-surface-files and service-operations-corpus
  • differential-conformance and service-coverage
  • rust-format and rust-clippy
  • service-operations-rust-build and service-operations-rust: 1095 cases, 0 failures
  • bun-typecheck, service-operations-bun-build and service-operations-bun: 1095 cases, 0 failures

Rust service tests (prose-runner-core lib, service_runs, service_jobs, service_programs, service_framework) and Bun service tests also pass.

Not passing locally, and not caused by this change:

  • rust-tests. The full gate fails on this machine under load, and fails the same way on unmodified main @ 8e2a258. The failures are process-timing tests that time out with exit 124, a different set each run (weave_host, cli.rs adapter tests). The weave_host tests pass 8/8 run alone on this branch.
  • Bun dev-endpoint.test.ts and build-identity.test.ts. They fail on unmodified main here too; local Bun is 1.4.2, while the repo pins 1.3.5.

GitHub CI is authoritative for both.

An independent review agent read the diff before this was opened. It found a Rust/Bun divergence on empty spec strings, now fixed and pinned by job-create-schedule-empty-strings-quote, plus wording overclaims, all addressed.

🤖 Generated with Claude Code

The service's hold now depends on model, reasoning effort, environment,
declared tools and bound repositories. run quote and the run submit plan
send the given --model, --reasoning-effort, --environment and
repositories=1 for --repo/--commit-output; the job create plan sends the
spec's model, reasoning_effort, environment and repositories=1 for a
repository URL. Nothing is defaulted, so a flagless quote is unchanged.
Replace the flat-hold wording in holdBasis, human output, help, guide,
SPEC and docs. Accept model, reasoning_effort, repository_url,
repository_branch and output in webhook job specs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant