diff --git a/plugins/oz-harness-support/README.md b/plugins/oz-harness-support/README.md index f9d461b..3f7b645 100644 --- a/plugins/oz-harness-support/README.md +++ b/plugins/oz-harness-support/README.md @@ -20,6 +20,29 @@ The plugin ships skills the agent uses to talk to the Oz platform: - **oz-notify-user** — send a progress notification to the triggering user - **oz-report-pr** — report a created pull request back to Oz - **oz-upload-file** — upload a local file as a conversation artifact +- **factory-files** — author and validate file-based Warp software factory definitions + +### factory-files is a mirror + +`skills/factory-files` is a byte-for-byte copy of +`resources/bundled/skills/factory-files` in +[warpdotdev/warp](https://github.com/warpdotdev/warp), mirrored at commit +`f6f4ceac8`. Warp bundles that skill for its own clients; Claude Code loads +filesystem skills from this plugin instead, so it is copied here rather than +resolved from a bundle. Change it in `warpdotdev/warp` and re-mirror; edits +made here are lost on the next sync. + +The skill validates only against warp-server, which owns the Factory file +format. It carries no copy of that format: a bundled copy ships inside a +release, goes stale, and then reports valid fields as unknown, which invites an +agent to delete working configuration. When the server cannot be reached the +skill reports that the tree was not checked rather than guessing. + +That also keeps this mirror cheap. There is no schema here to drift, so a stale +copy costs a stale workflow document, not a wrong verdict. +`tests/test-factory-files.sh` checks the copy arrived complete, carries no +schemas, and reports a missing verdict correctly; its behavioural corpus lives +in Warp. ## Requirements diff --git a/plugins/oz-harness-support/skills/factory-files/SKILL.md b/plugins/oz-harness-support/skills/factory-files/SKILL.md new file mode 100644 index 0000000..abe549d --- /dev/null +++ b/plugins/oz-harness-support/skills/factory-files/SKILL.md @@ -0,0 +1,169 @@ +--- +name: factory-files +description: Create and edit file-based Warp software factory definitions, in a repository tree rooted at a factory.yaml. Use when authoring or changing that factory.yaml, Agent, Automation, Scorer, or Runner files under that root, or its factory and agent skill trees, and when fixing Factory file diagnostics. Do not use for agent-definition Markdown that belongs to another tool, for a tree with no factory.yaml, or to operate a live factory or hand work to one through Factory MCP. +--- + +# Factory Files + +A software factory can be defined by files in a repository. This skill covers +authoring and editing those files, and validating them before you open a pull +request. + +warp-server owns the format. It publishes the schema for each version it +supports and validates a tree with the same parser the apply path uses. This +skill carries no copy of the format: a copy ships inside a Warp release, goes +stale against the server, and then reports confident, wrong diagnostics. When +the server cannot be reached, the answer is that the tree was not checked. + +Use this skill for repository files. It is not the skill for operating a live +factory: use `factory-mcp` to send work to a factory, inspect task status, or +pull a task down locally. Playbooks under a factory's own `skills/` directories +tell that factory's agents how to do their job; editing one is a prompt change, +not a schema change, so this skill's rules do not apply to their contents. + +## Locate the Factory root +Every Factory tree is rooted at the directory containing `factory.yaml`. All +paths below are relative to that root. A repository may register a +subdirectory as the root, so find `factory.yaml` rather than assuming the +repository root. Do not follow symlinks while looking: the server parses the +repository tree, where a symlink is stored as its target path rather than its +target's content. + +If there is no `factory.yaml`, this is not a Factory tree and nothing here +applies. `agents//agent.md` and similar paths are also used by other +agent tooling; stop and say so rather than imposing this schema on them. + +``` +factory.yaml required, exactly one +agents//agent.md at least one; exactly one must be MAIN +agents//skills/** skills only that agent can use +automations//automation.md optional +runners/.yaml optional +scorers//scorer.md optional; Markdown body is the rubric +skills/** skills every agent in the factory can use +``` + +Resource names come from the path, never from a field inside the file. Renaming +an agent means moving its directory. + +`automations/.md` is a legacy flat form the parser still accepts. Create +the directory form; when editing an existing flat file, leave it where it is +unless the user asks you to normalize the tree. + +## Before you edit +1. Read the files you are about to change, plus `factory.yaml`, so you can see + what is inherited and what is overridden. +2. Preserve fields and Markdown bodies you were not asked to change. The body + after an Agent's or Automation's closing `---` fence is its prompt; a + Scorer's body is its rubric. Never fold either into frontmatter. +3. Prefer the smallest edit that satisfies the request. + +## Author against the server's schema +Read the tree's `schemaVersion` from `factory.yaml`; a tree that omits it is +`v1alpha1`. Then fetch the schema for that version: + +```bash +curl -s https://app.warp.dev/api/v1/factory-files/schemas +curl -s https://app.warp.dev/api/v1/factory-files/schemas/ +``` + +The registry lists the versions the server supports. The version endpoint +returns every document describing one version, keyed by file name: +`factory.schema.json` for `factory.yaml`, `agent.schema.json`, +`automation.schema.json`, `runner.schema.json` and `scorer.schema.json` for the +corresponding resources, and `common.schema.json` for the definitions they +share. Both endpoints are unauthenticated. They are exact for the version they +describe: an unknown field is an error, and each enumerated value is one the +server accepts today. + +If the server does not publish the declared version, stop. Do not measure the +tree against a version it does not claim to be, and never lower +`schemaVersion` to make a check pass. + +Read `references/examples.md` for worked examples of each resource, and +`references/scorers.md` before writing or changing a Scorer. The field-by-field +catalogue is not duplicated here any more; the fetched schema carries it, with +a description on each field. + +## Validate before opening a pull request +Run the bundled validator with Python 3.8 or newer, using the host's command +(`python3`, `python`, or `py -3`). Quote both paths because an app-bundle path +can contain spaces. + +```bash +python3 "{{skill_dir}}/scripts/validate_factory_files.py" "" +``` + +It selects the tree's resource files and submits them to the server, which runs +the real parser. Add `--json` for machine-readable output and `--server-root +`, or `WARP_SERVER_ROOT`, to point at a local, staging, or self-hosted +server. No credential is required; `WARP_API_KEY` is forwarded when the +environment already carries one, as an agent sandbox does. + +The exit code distinguishes three outcomes, and so must you: + +- `0` the server checked the tree and found no problem. +- `1` the server checked the tree and reported diagnostics. Fix every one and + re-run until it is clean. +- `2` the tree was **not** checked. This is not a pass and not a failure; it + says nothing about the files at all. + +### Never imply a check that did not happen +On exit `2`, say plainly that validation did not run and why. Do not describe +the files as valid, correct, or ready, and do not substitute your own reading +of the schema for a verdict. If you cannot reach a server and the change +matters, say so and let the user decide. + +On exit `0`, repeat the sentence the validator prints rather than paraphrasing +it into something stronger. A pass means the parser and the state-independent +checks agreed; it does not mean the tree will apply. + +Validation resolves no server state. Model IDs, environment IDs, secret names, +runner names, Scorer model IDs, MCP server IDs, integration availability, and +the values of Linear and Slack name aliases are all checked when the plan is +applied. The response lists what it did not check, including any deferred name +aliases; report that distinction rather than claiming a tree is fully verified. + +If no Python 3 interpreter is available, do not install one or claim the tree +was validated without the user's approval. Check the changed document against +the fetched schema by hand and report that automated validation was +unavailable. + +When the Factory is already registered, a server plan remains the strongest +available check. See `references/validation.md` for diagnostic codes and how to +read them. + +## Rules that are easy to get wrong +- Exactly one agent declares `agentType: MAIN` (or `FOREMAN`, its canonical + spelling). Zero or two is an error. +- `model` and `harness` are mutually exclusive everywhere. `model: ` is + shorthand for the Oz harness. +- `agentDefaults` must declare one of them; agents and automations may declare + neither and inherit. +- Declaring `secrets` or `mcpServers` at agent or automation level replaces the + inherited value; it does not merge. +- An automation needs at least one trigger, and every trigger needs `provider` + and `event`. +- A `schedule.cron_fired` trigger needs either an inline `schedule.cron` or a + non-empty `filter.schedule_ids`, and never both. +- Linux runners require `platform.linux.dockerImage`. A runner with no + `platform` section defaults to Linux and will fail for that reason. +- Trigger filter keys depend on the `(provider, event)` pair. Some fields have + a friendlier authoring spelling that the server rewrites for you: GitHub + `baseBranches` and `prNumbers`, Linear `teams`, `projects`, `states` and + `issues`, and Slack `channels`, `users` and `itemUsers`. Each stands in for + its canonical key, and declaring both is an error. The Linear and Slack ones + name objects the server looks up at apply time, so they take a plain list of + names rather than an `in`/`not_in` matcher. + +## Do not add a local copy of the format +It is tempting to bundle the schema, or to reimplement a few checks here so +authoring works offline. Both have been tried and removed. A copy inside a Warp +release is routinely older than the server it is used against, and a stale copy +does not fail quietly: it reports a valid field as unknown, and an agent trying +to get to a clean run deletes working configuration to satisfy it. That has +already happened once, to Linear and Slack trigger aliases the server accepts. + +Reporting that a tree was not checked costs a little. Reporting the wrong +answer costs correct configuration. Fetch the format when you need it; say +nothing when you cannot. diff --git a/plugins/oz-harness-support/skills/factory-files/references/examples.md b/plugins/oz-harness-support/skills/factory-files/references/examples.md new file mode 100644 index 0000000..7917661 --- /dev/null +++ b/plugins/oz-harness-support/skills/factory-files/references/examples.md @@ -0,0 +1,208 @@ +# Examples +Every value below is a placeholder. Never copy IDs, secret names, ARNs, +project numbers, or images from another factory into a new one. + +## Minimal factory +``` +factory.yaml +agents/foreman/agent.md +``` + +`factory.yaml`: + +```yaml +schemaVersion: v1alpha1 +name: platform-ops +description: Keeps the platform repositories healthy. +repositories: + - owner: acme + name: platform +agentDefaults: + model: auto +``` + +`agents/foreman/agent.md`: + +```markdown +--- +description: Entry point for platform work. +agentType: MAIN +--- +You are the platform foreman. Triage incoming work, decide whether it needs a +spec, and delegate implementation. +``` + +## Factory with secrets, MCP, and integrations +```yaml +schemaVersion: v1alpha1 +name: platform-ops +alias: Platform Ops +credentialStrategy: EXECUTOR +repositories: + - owner: acme + name: platform + - owner: acme + name: platform-config +secrets: + - DEPLOY_TOKEN +mcpServers: + github: + warpId: mcp_placeholder_github +integrations: + - type: linear + - type: slack +cloudProviders: + gcp: + projectNumber: "000000000000" + workloadIdentityFederationPoolId: example-pool + workloadIdentityFederationProviderId: example-provider + serviceAccountEmail: factory@example.iam.gserviceaccount.com +agentDefaults: + model: auto + runner: linux-standard + secrets: + - DEPLOY_TOKEN +``` + +## Agent on a non-Oz harness +```markdown +--- +description: Implements approved specs. +agentType: IMPLEMENT +harness: + type: claude + model: opus + reasoningLevel: high + auth: + source: managedSecret + secretName: ANTHROPIC_API_KEY +runner: linux-standard +--- +Implement the approved spec. Keep the change scoped to what the spec describes. +``` + +The Oz shorthand is a single field, and is mutually exclusive with `harness`: + +```markdown +--- +agentType: REVIEW +model: auto +--- +Review the change against the repository's conventions. +``` + +## Agent that clears an inherited value +```markdown +--- +description: Runs on the shared workspace default host. +workerHost: null +secrets: [] +--- +Investigate the failure and report what you find. +``` + +`workerHost: null` clears an inherited host. `secrets: []` replaces the +inherited list with nothing; it does not merge. + +## Event-driven automation +`automations/pr-review/automation.md`: + +```markdown +--- +agent: reviewer +triggers: + - provider: github + event: pull_request_opened + filter: + repos: [acme/platform] + base_branches: [main] + labels: + not_in: [wip] + - provider: github + event: pull_request_synchronized + filter: + repos: [acme/platform] +--- +Review the pull request that triggered this run. +``` + +## Scheduled automation +```markdown +--- +enabled: true +triggers: + - provider: schedule + event: cron_fired + schedule: + name: nightly-sweep + cron: 0 3 * * * +--- +Sweep for stale branches and open a cleanup pull request if any are found. +``` + +## Linux runner +`runners/linux-standard.yaml`: + +```yaml +description: Standard Linux build runner. +setupCommands: + - apt-get update -y + - apt-get install -y build-essential +instanceShape: + vcpus: 4 + memoryGb: 8 +platform: + os: linux + arch: x86_64 + linux: + dockerImage: ubuntu:24.04 +``` + +## macOS runner +`runners/macos-standard.yaml`: + +```yaml +description: macOS runner for Apple platform builds. +instanceShape: + vcpus: 6 + memoryGb: 14 +platform: + os: macos + arch: aarch64 + mac: + version: "15" +``` +## Scorer +`scorers/tests-run/scorer.md`: + +```markdown +--- +description: Checks whether implementation runs include test evidence. +agents: + - implementer +labels: + - value: tests_run + score: 1 + - value: tests_skipped + score: 0 +passingScore: 1 +samplingRate: 25 +model: claude-4-5-haiku +--- +Evaluate whether the agent ran the relevant tests before finishing. Return +`tests_run` when the transcript contains the command and result; otherwise +return `tests_skipped`. +``` + +## Scoped skills +A skill under `skills/` is available to every agent in the factory. A skill +under `agents//skills/` is available only to that agent. + +``` +skills/release-checklist/SKILL.md every agent +agents/reviewer/skills/review-style/SKILL.md the reviewer agent only +``` + +These are agent playbooks, not Factory schema. Moving one between the two +locations changes who can use it, so do not relocate a skill unless that is +what was asked for. diff --git a/plugins/oz-harness-support/skills/factory-files/references/scorers.md b/plugins/oz-harness-support/skills/factory-files/references/scorers.md new file mode 100644 index 0000000..2e47b8f --- /dev/null +++ b/plugins/oz-harness-support/skills/factory-files/references/scorers.md @@ -0,0 +1,52 @@ +# File-defined Scorers + +A Scorer lives at `scorers//scorer.md`. Its directory provides its name, +YAML frontmatter defines the classification contract, and the non-empty +Markdown body is the rubric evaluated against eligible runs. + +```markdown +--- +description: Checks whether implementation runs include test evidence. +agents: + - implementer +labels: + - value: tests_run + description: The transcript contains a test command and its result. + score: 1 + - value: tests_partial + description: Only a relevant subset of tests ran. + score: 0.5 + - value: tests_skipped + score: 0 +passingScore: 1 +samplingRate: 25 +model: claude-4-5-haiku +selfImprovement: true +--- +Evaluate whether the agent ran the repository's relevant tests before +finishing. Return exactly one declared label. +``` + +## Fields + +- `agents` — required non-empty list of Agent names declared in this Factory. + Names are trimmed and must be unique. +- `enabled` — optional boolean, default true. Use `enabled: false` rather than + a zero sampling rate to pause scoring. +- `output` — optional output form. `classification` is the current known form. +- `labels` — required non-empty list of classifications; the current server + accepts at most 20. Each label requires a non-empty `value` and a numeric + `score` from 0 through 1; `description` is optional. Label values are + trimmed and unique. +- `passingScore` — required numeric threshold from 0 through 1. At least one + label must score at or above it, and at least one below it. +- `samplingRate` — optional percentage, default 25. Values are rounded to two + decimal places and must resolve to 0.01–100. +- `model` — required model ID. The server validates availability. +- `selfImprovement` — optional boolean, default false. + +The Markdown body must not be empty. + +The field list above is a summary for authoring; the server's schema is the +contract. If a Scorer already contains a field this page does not mention, +leave it alone and validate against the server rather than deleting it. diff --git a/plugins/oz-harness-support/skills/factory-files/references/validation.md b/plugins/oz-harness-support/skills/factory-files/references/validation.md new file mode 100644 index 0000000..07ce4f4 --- /dev/null +++ b/plugins/oz-harness-support/skills/factory-files/references/validation.md @@ -0,0 +1,162 @@ +# Validating and reading diagnostics + +## Layers of validation +1. **The server's parser** (`logic/factoryfile` in `warp-server`) is the only + authority. `POST /api/v1/factory-files/validate` runs it over a tree you + submit as paths and content, and adds the state-independent rules the apply + path enforces next: runner platforms and instance shapes, and trigger filter + keys and matchers. Its diagnostics carry `FF_*` codes with a path, line, and + column. +2. **Resolution and apply** validate everything that needs server state: model + IDs, environment IDs, secret names, runner names, MCP server IDs, + integration providers, harness model catalogues, worker-host entitlement, + and the values of Linear and Slack name aliases. + +There is no third layer, and deliberately no local one. A clean result from the +endpoint means the files pass the structural and state-independent semantic +checks. It does not mean the plan will apply. The response lists the checks it +did not run; say so rather than overstating what was checked. + +## Running the validator +The script lives at `scripts/validate_factory_files.py` inside this skill's +directory; `SKILL.md` shows its resolved path. It does not parse the format. It +selects the tree's resource files by path, refuses symlinks, submits the bytes, +and relays what comes back. + +```bash +python3 "/scripts/validate_factory_files.py" "" +python3 "/scripts/validate_factory_files.py" "" --json +``` + +`--server-root ` or `WARP_SERVER_ROOT` selects a local, staging, or +self-hosted server. The endpoint needs no credential; `WARP_API_KEY` is +forwarded when the environment already has one, which makes the request +attributable inside an agent sandbox. + +Use Python 3.8 or newer via the host's command (`python3`, `python`, or +`py -3`). If none is available, do not install an interpreter or claim +automated validation without the user's approval; inspect the changed document +against the fetched schema and report the gap. + +## The three outcomes +- `0` — the server checked the tree and found no problem. +- `1` — the server checked the tree and reported diagnostics. +- `2` — the tree was **not** checked. + +Exit `2` is not a pass and not a failure. It happens when the server is +unreachable, answers with an error or a malformed body, the directory is not a +Factory root, or the tree is larger than the endpoint accepts. In every case +the correct report is that validation did not run, with the reason. Saying +anything about whether the files are correct would be inventing a verdict. + +With `--json`, `validated` distinguishes the cases: a run that reached no +verdict carries `validated: false` and no `valid` key at all, so there is +nothing to misread. + +Each problem reports the file, the field path, and what is wrong. Fix them all +and re-run; do not stop at the first one, since one wrong field often produces +several messages. + +A resource file that is a symlink is reported and never uploaded. The server +parses the repository tree, where a symlink is stored as its target path rather +than its target's content, so it never follows one either; a Factory resource +has to be a real file. Reading the target locally would also let a repository +aim a resource at any readable path on the machine. + +## Deferred resolutions +A response can carry `deferred_resolutions` alongside its diagnostics. A +deferred entry is not a problem: it names an authored value the endpoint +deliberately did not prove, because proving it needs provider state. Linear and +Slack name aliases are the current case — the endpoint checks that `teams` or +`channels` is a list of non-empty names applicable to that event, and leaves +whether those names exist to apply time. + +Report deferred entries. They are the difference between "this parses" and +"this will work". + +## Diagnostic codes +The server reports these from the validation endpoint and when a plan is run +against a registered Factory. + +- `FF_MISSING_FACTORY` — no `factory.yaml` at the Factory root. +- `FF_UNSUPPORTED_VERSION` — `schemaVersion` names no registered tree adapter. + The server stops rather than applying another version's rules. Correct the + version; never lower it to make a check pass. +- `FF_UNSUPPORTED_PATH` — a file that resembles an Agent, Automation, Runner, + or Scorer resource is at a non-canonical path. Other unrelated files under + those directories are intentionally ignored. +- `FF_DUPLICATE_PATH` — the same resource name is declared twice, most often an + automation declared in both the flat and directory forms. +- `FF_INVALID_DOCUMENT` — a file is empty or its root is not a YAML mapping. +- `FF_MALFORMED_FRONTMATTER` — a Markdown resource is missing an opening or + closing `---` fence. +- `FF_INVALID_YAML` — the YAML could not be parsed. +- `FF_DUPLICATE_KEY` — a mapping repeats a key. Also reported when two agents + declare `MAIN`/`FOREMAN`, or a secret is listed twice. +- `FF_ANCHOR`, `FF_ALIAS`, `FF_TAG` — YAML anchors, aliases, and explicit tags + are not permitted. +- `FF_UNKNOWN_FIELD` — a field the schema does not define. Check spelling and + the fetched schema. +- `FF_MISSING_REQUIRED` — a required field is absent or empty. +- `FF_TYPE_MISMATCH` — a value has the wrong YAML type. +- `FF_INVALID_VALUE` — a value violates a format or exclusivity rule, such as + declaring both `model` and `harness`, or an alias with disallowed characters. +- `FF_INVALID_REFERENCE` — a named reference does not resolve, such as an + Automation or Scorer naming an Agent the tree does not declare, or an + unknown current `agentType` or `credentialStrategy`. +- `FF_INVALID_MCP` — an MCP entry is not exactly a non-empty `warpId`. +- `FF_INVALID_TRIGGER` — a trigger is structurally wrong, such as an inline + schedule on a non-schedule trigger, or a `schedule.cron_fired` trigger that + declares both or neither of `schedule.cron` and `filter.schedule_ids`. +- `FF_INVALID_EVENT`, `FF_INVALID_FILTER` — the event is unknown, or a filter + key or value is outside its valid domain. + +## Fetching the schema +The schema endpoints are unauthenticated and cacheable: + +```bash +curl -s https://app.warp.dev/api/v1/factory-files/schemas +curl -s https://app.warp.dev/api/v1/factory-files/schemas/v1alpha1 +``` + +They are ordinary JSON Schema 2020-12 documents, exact for the version they +describe, so any standard validator works if a tree is already converted to +JSON. `x-warp-*` annotations carry constraints JSON Schema cannot express +portably, such as trimmed Unicode alias rules and the power-of-two Linux +compute sizes; only a Warp validator enforces those. + +Fetching a schema is not validation. A successful fetch says the server is +reachable, nothing more. + +## Why there is no offline mode +This skill used to bundle the schemas and a local validator so authoring worked +without a server. That was removed, and should not be reintroduced. + +The copy shipped inside a Warp release, so it was routinely older than the +server a Factory syncs against. A stale copy does not degrade gracefully: it +reports a field the server accepts as unknown, and an agent trying to reach a +clean run resolves that by deleting working configuration. It happened — an +earlier revision rejected the Linear and Slack trigger aliases (`teams`, +`projects`, `states`, `issues`, `channels`, `users`, `itemUsers`) that the +apply path rewrites and accepts, on a tree taken from the server's own +`testdata/valid`. + +The trade is deliberate: never checking is recoverable, and the report says so. +Checking wrongly costs correct configuration and is not obviously wrong to the +agent acting on it. + +## Fixing a diagnostic +Change the file the diagnostic names, at the field it names. Do not silence a +diagnostic by deleting the resource or moving a file to a path the parser +ignores. + +## Checking against the parser directly +When `warp-server` is checked out locally, its parser tests are the closest +thing to ground truth. Run them from that checkout, not from the Factory +repository: + +```bash +go test ./logic/factoryfile/... +``` + +Fixtures under `logic/factoryfile/testdata` show accepted and rejected trees. diff --git a/plugins/oz-harness-support/skills/factory-files/scripts/validate_factory_files.py b/plugins/oz-harness-support/skills/factory-files/scripts/validate_factory_files.py new file mode 100644 index 0000000..9b528fc --- /dev/null +++ b/plugins/oz-harness-support/skills/factory-files/scripts/validate_factory_files.py @@ -0,0 +1,376 @@ +#!/usr/bin/env python3 +"""Validate a Factory file tree against warp-server. + +Usage: + python3 validate_factory_files.py [FACTORY_ROOT] [--json] [--server-root URL] + +FACTORY_ROOT defaults to the current directory and must contain factory.yaml. + +warp-server owns the Factory file format, so it is the only thing that decides +whether a tree is valid. This script collects the tree's resource files and +submits them to the validation endpoint, which runs the real parser plus the +state-independent checks the apply path would run next. + +There is deliberately no local fallback. A bundled copy of the format is +routinely older than the server it is used against, and a stale copy does not +degrade gracefully: it reports confident, wrong diagnostics that invite an +agent to "repair" valid configuration by deleting it. Reporting that a tree +could not be checked is strictly safer than reporting the wrong answer, so when +the server cannot be reached this script says so and validates nothing. + +That also means this script never parses YAML. It decides which files are +resource files from their paths alone and sends their bytes verbatim, so it +cannot disagree with the parser about what a document means. + +The endpoint does not resolve server state. Model IDs, environment IDs, secret +names, runner references, MCP server IDs, integration availability, and the +values of provider name aliases are all checked when the plan is applied. + +Exit codes: + 0 the server validated the tree and found no problem + 1 the server validated the tree and reported diagnostics + 2 the tree was not validated; the reason is printed +""" + +from __future__ import annotations + +import argparse +import json +import os +import sys +import urllib.error +import urllib.request +from pathlib import Path +from typing import Any, Optional + +DEFAULT_SERVER_ROOT = "https://app.warp.dev" +VALIDATE_PATH = "/api/v1/factory-files/validate" + +# Bounded so an unreachable or slow server reports quickly rather than stalling +# an authoring session. +REQUEST_TIMEOUT_SECONDS = 10.0 +MAX_RESPONSE_BYTES = 8 * 1024 * 1024 + +# Mirrors the caps the endpoint enforces, so an oversized tree is reported here +# rather than collecting a 400. +MAX_REMOTE_FILES = 256 +MAX_REMOTE_FILE_BYTES = 256 * 1024 +MAX_REMOTE_CONTENT_BYTES = 2 * 1024 * 1024 + +SYMLINK_REFUSED = ( + "resource file is a symlink, or resolves outside the Factory root, and was not " + "read. The server parses the repository tree, so it sees the link itself rather " + "than its target and cannot accept this either. Replace it with a real file." +) + +EXIT_VALID = 0 +EXIT_DIAGNOSTICS = 1 +EXIT_NOT_VALIDATED = 2 + + +class NotValidated(Exception): + """The tree was not checked. This is never a pass.""" + + +class Problem: + """One reported failure, located as precisely as the server allows.""" + + def __init__(self, path: str, message: str, line: Optional[int] = None, pointer: str = ""): + self.path = path + self.message = message + self.line = line + self.pointer = pointer + + def as_dict(self) -> dict[str, Any]: + return { + "path": self.path, + "line": self.line, + "pointer": self.pointer, + "message": self.message, + } + + def render(self) -> str: + location = self.path + if self.line is not None: + location += f":{self.line}" + if self.pointer: + return f"{location}: {self.pointer}: {self.message}" + return f"{location}: {self.message}" + + +# --------------------------------------------------------------------------- +# Selecting the tree to submit +# --------------------------------------------------------------------------- + + +def classify(relative: str) -> tuple[str, str]: + """Mirror the server's path classification. Returns (kind, name). + + This decides only which files are worth submitting. The server classifies + them again and owns the verdict, so a disagreement here costs a wasted + upload rather than a wrong answer. + """ + if relative == "factory.yaml": + return "factory", "" + segments = relative.split("/") + if len(segments) >= 2 and segments[0] == "skills": + return "skill", "" + if len(segments) >= 4 and segments[0] == "agents" and segments[2] == "skills": + return "skill", "" + if len(segments) == 3 and segments[0] == "agents" and segments[2] == "agent.md": + return ("agent", segments[1]) if _valid_name(segments[1]) else ("invalid", "") + if len(segments) == 3 and segments[0] == "automations" and segments[2] == "automation.md": + return ("automation", segments[1]) if _valid_name(segments[1]) else ("invalid", "") + if len(segments) == 2 and segments[0] == "automations" and segments[1].endswith(".md"): + name = segments[1][: -len(".md")] + return ("automation", name) if _valid_name(name) else ("invalid", "") + if len(segments) == 2 and segments[0] == "runners" and segments[1].endswith(".yaml"): + name = segments[1][: -len(".yaml")] + return ("runner", name) if _valid_name(name) else ("invalid", "") + if len(segments) == 3 and segments[0] == "scorers" and segments[2] == "scorer.md": + return ("scorer", segments[1]) if _valid_name(segments[1]) else ("invalid", "") + if len(segments) == 2 and segments[0] == "scorers" and segments[1].endswith(".md"): + return "invalid", "" + base = segments[-1] + if segments[0] == "agents" and base == "agent.md": + return "invalid", "" + if segments[0] == "automations" and base == "automation.md": + return "invalid", "" + if segments[0] == "runners" and base.endswith(".yaml"): + return "invalid", "" + if segments[0] == "scorers" and base == "scorer.md": + return "invalid", "" + return "unrelated", "" + + +def _valid_name(name: str) -> bool: + return name not in ("", ".", "..") and "/" not in name + + +def _resource_files(root: Path) -> list[Path]: + files = [root / "factory.yaml"] + for directory_name in ("agents", "automations", "runners", "scorers"): + resource_root = root / directory_name + if not resource_root.is_dir(): + continue + for directory, child_directories, filenames in os.walk(resource_root): + relative_directory = Path(directory).relative_to(root) + parts = relative_directory.parts + if directory_name == "agents" and len(parts) == 2: + child_directories[:] = [name for name in child_directories if name != "skills"] + files.extend(Path(directory) / filename for filename in filenames) + return sorted(files) + + +def _leaves_factory_root(path: Path, root: Path) -> bool: + """Report whether reading path would follow a link out of the tree. + + The server never resolves links: it parses an in-memory git tree, where a + symlink is a blob whose content is the target path, so it sees the link + itself. Following one here would both diverge from that and read a file the + Factory does not contain - an untrusted repository could otherwise point a + resource at any readable path and have its content uploaded. + """ + if path.is_symlink(): + return True + try: + path.resolve().relative_to(root.resolve()) + except (OSError, ValueError, RuntimeError): + return True + return False + + +def collect_tree(root: Path) -> tuple[list[dict[str, str]], list[Problem]]: + """Collect the resource files to submit, refusing symlinks as the server does.""" + if not (root / "factory.yaml").is_file(): + raise NotValidated(f"{root} has no factory.yaml, so it is not a Factory root") + files: list[dict[str, str]] = [] + problems: list[Problem] = [] + total = 0 + for absolute in _resource_files(root): + relative = absolute.relative_to(root).as_posix() + kind, _ = classify(relative) + if kind in ("unrelated", "skill", "invalid"): + continue + if _leaves_factory_root(absolute, root): + problems.append(Problem(relative, SYMLINK_REFUSED)) + continue + try: + content = absolute.read_text(encoding="utf-8") + except (OSError, UnicodeError) as error: + problems.append(Problem(relative, f"could not read UTF-8 resource: {error}")) + continue + encoded = len(content.encode("utf-8")) + if encoded > MAX_REMOTE_FILE_BYTES: + raise NotValidated(f"{relative} is larger than the endpoint accepts") + total += encoded + if total > MAX_REMOTE_CONTENT_BYTES or len(files) >= MAX_REMOTE_FILES: + raise NotValidated("the tree is larger than the endpoint accepts") + files.append({"path": relative, "content": content}) + if not files: + raise NotValidated("the tree has no resource files to submit") + return files, problems + + +# --------------------------------------------------------------------------- +# Server-backed validation +# --------------------------------------------------------------------------- + + +class Outcome: + """What the server found, and what it deliberately did not check.""" + + def __init__( + self, + schema_version: str, + problems: list[Problem], + deferred: Optional[list[dict[str, Any]]] = None, + ): + self.schema_version = schema_version + self.problems = problems + self.deferred = deferred or [] + + def disclosure(self) -> str: + """The sentence the agent must repeat. Never claim more than ran.""" + return ( + f"Validated with the warp-server parser for {self.schema_version}; " + "state-dependent apply checks were not run." + ) + + +def server_root(argument: Optional[str]) -> str: + """Resolve the server to ask, so a local or staging root needs no code change.""" + chosen = argument or os.environ.get("WARP_SERVER_ROOT") or DEFAULT_SERVER_ROOT + return chosen.rstrip("/") + + +def _request_json(url: str, token: Optional[str] = None, payload: Optional[Any] = None) -> Any: + """Post or fetch JSON, turning every failure class into NotValidated.""" + data = None + headers = {"Accept": "application/json"} + if payload is not None: + data = json.dumps(payload).encode("utf-8") + headers["Content-Type"] = "application/json" + if token: + headers["Authorization"] = "Bearer " + token + request = urllib.request.Request(url, data=data, headers=headers) + try: + with urllib.request.urlopen(request, timeout=REQUEST_TIMEOUT_SECONDS) as response: + body = response.read(MAX_RESPONSE_BYTES + 1) + except urllib.error.HTTPError as error: + raise NotValidated(f"the server answered HTTP {error.code}") from error + except Exception as error: # DNS, TLS, connection, timeout, proxy, ... + raise NotValidated(f"the server could not be reached: {error}") from error + if len(body) > MAX_RESPONSE_BYTES: + raise NotValidated("the server response was implausibly large") + try: + return json.loads(body.decode("utf-8")) + except (UnicodeError, ValueError) as error: + raise NotValidated(f"the server response was not JSON: {error}") from error + + +def validate(root: Path, base_url: str) -> Outcome: + """Submit the tree to the server, or raise NotValidated. + + The endpoint reads the declared schemaVersion itself and reports an + unrecognized one as a diagnostic, so there is nothing to pre-flight and no + reason for this script to read the tree's YAML. + + The endpoint needs no credential. WARP_API_KEY is forwarded when the + environment already carries one, as an Oz sandbox does, so the request is + attributable there; nothing requires it, because a local authoring agent + runs in a shell that cannot see the Warp client's session. + """ + token = os.environ.get("WARP_API_KEY") + files, problems = collect_tree(root) + response = _request_json(base_url + VALIDATE_PATH, token=token, payload={"files": files}) + if not isinstance(response, dict) or not isinstance(response.get("diagnostics"), list): + raise NotValidated("the validation response was malformed") + + for diagnostic in response["diagnostics"]: + if not isinstance(diagnostic, dict): + raise NotValidated("the validation response was malformed") + code = str(diagnostic.get("code", "")) + message = str(diagnostic.get("message", "")) + line = diagnostic.get("line") + problems.append( + Problem( + str(diagnostic.get("path", "")), + f"{code}: {message}" if code else message, + line=line if isinstance(line, int) else None, + ) + ) + deferred = [ + entry for entry in response.get("deferred_resolutions", []) if isinstance(entry, dict) + ] + reported = response.get("schema_version") + return Outcome( + reported if isinstance(reported, str) and reported else "unknown", + problems, + deferred, + ) + + +def main() -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("root", nargs="?", default=".", help="Factory root containing factory.yaml") + parser.add_argument("--json", action="store_true", help="emit machine-readable output") + parser.add_argument( + "--server-root", + default=None, + help="warp-server root to validate against; defaults to $WARP_SERVER_ROOT then " + + DEFAULT_SERVER_ROOT, + ) + args = parser.parse_args() + + root = Path(args.root).resolve() + try: + outcome = validate(root, server_root(args.server_root)) + except NotValidated as reason: + report = ( + f"This tree was NOT validated: {reason}. Nothing here says the files are " + "correct or incorrect. Validate against a reachable warp-server, and say " + "plainly that validation did not run." + ) + if args.json: + print(json.dumps({"validated": False, "reason": str(reason)}, indent=2)) + else: + print(report, file=sys.stderr) + return EXIT_NOT_VALIDATED + + problems = outcome.problems + if args.json: + print( + json.dumps( + { + "validated": True, + "valid": not problems, + "schema_version": outcome.schema_version, + "disclosure": outcome.disclosure(), + "problems": [problem.as_dict() for problem in problems], + "deferred_resolutions": outcome.deferred, + }, + indent=2, + ) + ) + return EXIT_DIAGNOSTICS if problems else EXIT_VALID + + if problems: + print(f"{len(problems)} problem(s) in {root}:", file=sys.stderr) + for problem in problems: + print(f" {problem.render()}", file=sys.stderr) + print(outcome.disclosure(), file=sys.stderr) + return EXIT_DIAGNOSTICS + + print(f"{root}: factory files are valid.") + print(outcome.disclosure()) + for entry in outcome.deferred: + print( + f" deferred: {entry.get('path', '')} {entry.get('field', '')} " + f"({entry.get('kind', '')}) is resolved when the plan is applied" + ) + return EXIT_VALID + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/plugins/oz-harness-support/tests/test-factory-files.sh b/plugins/oz-harness-support/tests/test-factory-files.sh new file mode 100755 index 0000000..d5ffd06 --- /dev/null +++ b/plugins/oz-harness-support/tests/test-factory-files.sh @@ -0,0 +1,117 @@ +#!/bin/bash +# +# Smoke test for the mirrored factory-files skill. +# +# The skill tree under skills/factory-files is a byte-for-byte copy of +# resources/bundled/skills/factory-files in warpdotdev/warp at the commit named +# in this plugin's README. Correctness of the Factory file format is owned by +# warp-server, and the skill's own behavioural corpus lives in warp; this +# checks the two things a copy can get wrong on its way over: arriving +# incomplete, and arriving with a local copy of the format that should not +# exist. +# +# There is no offline mode to exercise. CI has no warp-server, so the runnable +# assertion here is that the validator reports a tree as NOT validated rather +# than guessing at a verdict. + +set -uo pipefail + +PLUGIN_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +SKILL="$PLUGIN_ROOT/skills/factory-files" +VALIDATOR="$SKILL/scripts/validate_factory_files.py" + +PASSED=0 +FAILED=0 + +pass() { + echo " ✓ $1" + PASSED=$((PASSED + 1)) +} + +fail() { + echo " ✗ $1" + [ -n "${2:-}" ] && echo " $2" + FAILED=$((FAILED + 1)) +} + +echo "factory-files mirror" + +for required in \ + "$SKILL/SKILL.md" \ + "$VALIDATOR" \ + "$SKILL/references/examples.md" \ + "$SKILL/references/scorers.md" \ + "$SKILL/references/validation.md"; do + if [ -f "$required" ]; then + pass "present: ${required#"$SKILL/"}" + else + fail "missing: ${required#"$SKILL/"}" + fi +done + +# A bundled copy of the format is the failure this design removed: it ships +# inside a release, goes stale against the server, and then reports valid +# fields as unknown. +if find "$SKILL" -name '*.schema.json' | grep -q .; then + fail "the mirror carries bundled schemas, which go stale against the server" +else + pass "no local copy of the format" +fi + +if ! command -v python3 >/dev/null 2>&1; then + fail "python3 is required to run the validator" + echo " $PASSED passed, $FAILED failed" + exit 1 +fi + +if python3 -c "import ast,sys; ast.parse(open(sys.argv[1]).read())" "$VALIDATOR" 2>/dev/null; then + pass "the validator parses" +else + fail "the validator does not parse" +fi + +WORKSPACE="$(mktemp -d)" +trap 'rm -rf "$WORKSPACE"' EXIT + +mkdir -p "$WORKSPACE/tree/agents/main" +cat >"$WORKSPACE/tree/factory.yaml" <<'YAML' +schemaVersion: v1alpha1 +name: mirror-smoke +repositories: + - owner: warpdotdev + name: warp +agentDefaults: + model: auto +YAML +printf -- '---\nagentType: MAIN\n---\nDo the thing.\n' >"$WORKSPACE/tree/agents/main/agent.md" + +# Port 9 (discard) is reliably closed, so this exercises the unreachable-server +# path without depending on the network. +output="$(python3 "$VALIDATOR" "$WORKSPACE/tree" --server-root http://127.0.0.1:9 2>&1)" +status=$? +if [ "$status" -eq 2 ]; then + pass "an unreachable server exits 2" +else + fail "an unreachable server should exit 2, got $status" "$output" +fi + +case "$output" in + *"was NOT validated"*) + pass "an unreachable server is reported as not validated" + ;; + *) + fail "the missing verdict was not reported" "$output" + ;; +esac + +case "$output" in + *"Validated with the warp-server parser"*) + fail "a run with no server claimed a server verdict" "$output" + ;; + *) + pass "no verdict is claimed without a server" + ;; +esac + +echo " $PASSED passed, $FAILED failed" +[ "$FAILED" -eq 0 ] || exit 1