Skip to content

Validate Factory files against the server only (REMOTE-2868) #15219

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
advait-m merged 3 commits into from
Aug 17, 2026
Merged

Validate Factory files against the server only (REMOTE-2868) #15219

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
ddd8df1
Point REMOTE-2727 to the server schema spec
warp-agent-staging[bot] Aug 17, 2026
85e89ea
Make the factory-files skill fetch the format from the server
warp-agent-staging[bot] Aug 17, 2026
f6f4cea
Validate Factory files against the server only
advait-m Aug 17, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
130 changes: 38 additions & 92 deletions app/src/ai/skills/bundled_tests.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -86,9 +86,7 @@ fn factory_files_bundled_skill_is_always_active_and_scoped_to_authoring() {
));

for reference in [
"references/schema.md",
"references/scorers.md",
"references/triggers.md",
"references/examples.md",
"references/validation.md",
"scripts/validate_factory_files.py",
Expand All @@ -104,104 +102,52 @@ fn factory_files_bundled_skill_is_always_active_and_scoped_to_authoring() {
}
}

/// The schemas are the contract the skill tells agents to author against, so
/// they have to stay parseable and keep the Factory document's shape.
/// The skill must not carry a copy of the Factory file format.
///
/// A bundled copy ships inside a Warp release and goes stale against the
/// warp-server it is used against. A stale copy does not fail quietly: it
/// reports fields the server accepts as unknown, and an agent clearing that
/// diagnostic deletes working configuration. An earlier revision did exactly
/// that to the Linear and Slack trigger aliases. The format is fetched from
/// the server now, so nothing here should describe it.
#[test]
fn factory_files_schemas_are_parseable_and_keep_the_factory_contract() {
let schemas_dir = Path::new(env!("CARGO_MANIFEST_DIR"))
.join("../resources/bundled/skills/factory-files/schemas");
fn assert_refs_resolve(value: &serde_json::Value, current_name: &str, schemas_dir: &Path) {
match value {
serde_json::Value::Object(object) => {
if let Some(reference) = object.get("$ref").and_then(serde_json::Value::as_str) {
let (name, fragment) = reference.split_once('#').unwrap_or((reference, ""));
let target_name = if name.is_empty() { current_name } else { name };
let raw = std::fs::read_to_string(schemas_dir.join(target_name))
.unwrap_or_else(|error| panic!("read $ref target {target_name}: {error}"));
let target: serde_json::Value = serde_json::from_str(&raw)
.unwrap_or_else(|error| panic!("parse $ref target {target_name}: {error}"));
assert!(
fragment.is_empty() || target.pointer(fragment).is_some(),
"{current_name} contains unresolved $ref {reference}"
);
}
for child in object.values() {
assert_refs_resolve(child, current_name, schemas_dir);
}
}
serde_json::Value::Array(values) => {
for child in values {
assert_refs_resolve(child, current_name, schemas_dir);
}
fn factory_files_skill_carries_no_copy_of_the_format() {
let skill_dir = Path::new(env!("CARGO_MANIFEST_DIR"))
.join("../resources/bundled/skills/factory-files")
.canonicalize()
.expect("factory-files skill directory");

let mut schemas = Vec::new();
let mut pending = vec![skill_dir.clone()];
while let Some(directory) = pending.pop() {
for entry in std::fs::read_dir(&directory).expect("read skill directory") {
let path = entry.expect("read skill entry").path();
if path.is_dir() {
pending.push(path);
} else if path.to_string_lossy().ends_with(".schema.json") {
schemas.push(path);
}
_ => {}
}
}

for name in [
"common.schema.json",
"factory.schema.json",
"agent.schema.json",
"automation.schema.json",
"runner.schema.json",
"scorer.schema.json",
] {
let raw = std::fs::read_to_string(schemas_dir.join(name))
.unwrap_or_else(|error| panic!("read {name}: {error}"));
let schema: serde_json::Value = serde_json::from_str(&raw)
.unwrap_or_else(|error| panic!("{name} should be valid JSON: {error}"));
assert_eq!(
schema.get("$id").and_then(serde_json::Value::as_str),
Some(name),
"{name} should use a relative $id so sibling $refs resolve locally"
);
assert_refs_resolve(&schema, name, &schemas_dir);
}

let factory: serde_json::Value = serde_json::from_str(
&std::fs::read_to_string(schemas_dir.join("factory.schema.json")).unwrap(),
)
.unwrap();
let required: Vec<&str> = factory["required"]
.as_array()
.expect("factory schema declares required fields")
.iter()
.map(|value| value.as_str().expect("required entries are strings"))
.collect();
assert_eq!(
required,
["schemaVersion", "name", "repositories", "agentDefaults"]
);
// These schemas ship inside a Warp release and are routinely older than the
// warp-server they validate against, so they stay open on purpose: a closed
// schema would reject configuration a newer server accepts. Flipping either
// assertion to `false` is a regression, not a tightening. See
// specs/REMOTE-2727/TECH.md.
assert_eq!(
factory["additionalProperties"],
serde_json::Value::Bool(true)
);
// Both the current key and the legacy alias stay accepted; the server reads
// cloudProviders first and falls back to providers.
assert!(factory["properties"].get("cloudProviders").is_some());
assert!(factory["properties"].get("providers").is_some());

let scorer: serde_json::Value = serde_json::from_str(
&std::fs::read_to_string(schemas_dir.join("scorer.schema.json")).unwrap(),
)
.unwrap();
assert_eq!(
scorer["additionalProperties"],
serde_json::Value::Bool(true)
assert!(
schemas.is_empty(),
"the skill has regrown bundled schemas, which go stale against the server \
and produce false rejections; fetch the format instead: {schemas:?}"
);
for required in ["agents", "labels", "passingScore", "model"] {

let validator = std::fs::read_to_string(skill_dir.join("scripts/validate_factory_files.py"))
.expect("read the validator");
for banned in ["import yaml", "def load_yaml", "jsonschema"] {
assert!(
scorer["required"].as_array().is_some_and(|fields| {
fields.iter().any(|field| field.as_str() == Some(required))
}),
"scorer schema should require {required}"
!validator.contains(banned),
"the validator parses the format again ({banned}); it should send bytes \
to the server and relay the verdict"
);
}
assert!(
validator.contains("/api/v1/factory-files/validate"),
"the validator should reach the server's validation endpoint"
);
}

#[test]
Expand Down
159 changes: 85 additions & 74 deletions resources/bundled/skills/factory-files/SKILL.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,12 @@ A software factory can be defined by files in a repository. This skill covers
authoring and editing those files, and validating them before you open a pull
request.

warp-server owns the format. It publishes the schema for each version it
supports and validates a tree with the same parser the apply path uses. This
skill carries no copy of the format: a copy ships inside a Warp release, goes
stale against the server, and then reports confident, wrong diagnostics. When
the server cannot be reached, the answer is that the tree was not checked.

Use this skill for repository files. It is not the skill for operating a live
factory: use `factory-mcp` to send work to a factory, inspect task status, or
pull a task down locally. Playbooks under a factory's own `skills/` directories
Expand All @@ -19,7 +25,9 @@ not a schema change, so this skill's rules do not apply to their contents.
Every Factory tree is rooted at the directory containing `factory.yaml`. All
paths below are relative to that root. A repository may register a
subdirectory as the root, so find `factory.yaml` rather than assuming the
repository root.
repository root. Do not follow symlinks while looking: the server parses the
repository tree, where a symlink is stored as its target path rather than its
target's content.

If there is no `factory.yaml`, this is not a Factory tree and nothing here
applies. `agents/<name>/agent.md` and similar paths are also used by other
Expand Down Expand Up @@ -50,66 +58,78 @@ unless the user asks you to normalize the tree.
Scorer's body is its rubric. Never fold either into frontmatter.
3. Prefer the smallest edit that satisfies the request.

## Author against the schema
The current server parser rejects unknown fields, but bundled schemas can be
older than the server. They therefore validate known fields while preserving
unknown properties and newer catalogue values. Do not invent a field when a
documented one exists, and do not delete an existing unknown field. Duplicate
keys, YAML anchors, aliases, explicit tags, and multiple documents remain
invalid.

The bundled JSON Schemas are the machine-readable contract:
## Author against the server's schema
Read the tree's `schemaVersion` from `factory.yaml`; a tree that omits it is
`v1alpha1`. Then fetch the schema for that version:

```
schemas/factory.schema.json factory.yaml
schemas/agent.schema.json agents/<name>/agent.md frontmatter
schemas/automation.schema.json automations/<name>/automation.md frontmatter
schemas/runner.schema.json runners/<name>.yaml
schemas/scorer.schema.json scorers/<name>/scorer.md frontmatter
schemas/common.schema.json shared definitions referenced by the above
```bash
curl -s https://app.warp.dev/api/v1/factory-files/schemas
curl -s https://app.warp.dev/api/v1/factory-files/schemas/<schemaVersion>
```

Read `references/schema.md` for the field-by-field reference, defaults, and
inheritance rules. Read `references/triggers.md` before writing or changing an
automation trigger: filter keys are specific to each provider and event, and
the parser does not catch a wrong one. Read `references/scorers.md` before
writing or changing a Scorer. Read `references/examples.md` for worked
examples of each resource.
The registry lists the versions the server supports. The version endpoint
returns every document describing one version, keyed by file name:
`factory.schema.json` for `factory.yaml`, `agent.schema.json`,
`automation.schema.json`, `runner.schema.json` and `scorer.schema.json` for the
corresponding resources, and `common.schema.json` for the definitions they
share. Both endpoints are unauthenticated. They are exact for the version they
describe: an unknown field is an error, and each enumerated value is one the
server accepts today.

If the server does not publish the declared version, stop. Do not measure the
tree against a version it does not claim to be, and never lower
`schemaVersion` to make a check pass.

Read `references/examples.md` for worked examples of each resource, and
`references/scorers.md` before writing or changing a Scorer. The field-by-field
catalogue is not duplicated here any more; the fetched schema carries it, with
a description on each field.

## Validate before opening a pull request
Run the bundled validator with Python 3.8 or newer, using the host's command (`python3`,
`python`, or `py -3`). Quote both paths because an app-bundle path can contain
spaces.
Run the bundled validator with Python 3.8 or newer, using the host's command
(`python3`, `python`, or `py -3`). Quote both paths because an app-bundle path
can contain spaces.

```bash
python3 "{{skill_dir}}/scripts/validate_factory_files.py" "<factory-root>"
```

Add `--json` for machine-readable output. A non-zero exit means at least one
problem; fix every reported problem and re-run until it is clean.
It selects the tree's resource files and submits them to the server, which runs
the real parser. Add `--json` for machine-readable output and `--server-root
<url>`, or `WARP_SERVER_ROOT`, to point at a local, staging, or self-hosted
server. No credential is required; `WARP_API_KEY` is forwarded when the
environment already carries one, as an agent sandbox does.

If no Python 3 interpreter is available, do not install one or claim the tree
was validated without the user's approval. Check the changed document against
the corresponding JSON Schema manually and report that automated validation
was unavailable.
The exit code distinguishes three outcomes, and so must you:

The validator checks known field structure, mutual exclusions, trigger and
Scorer semantics, cron syntax, runner platform rules, and tree-level rules
(exactly one MAIN agent, Agent references, duplicate resource names). Unknown
properties and newer catalogue values pass through for version skew. It does
not resolve server state: model IDs, environment IDs, secret names, runner
names, Scorer model IDs, MCP server IDs, and integration availability are all
validated when the plan is applied. Report that distinction rather than
claiming a tree is fully verified.
- `0` the server checked the tree and found no problem.
- `1` the server checked the tree and reported diagnostics. Fix every one and
re-run until it is clean.
- `2` the tree was **not** checked. This is not a pass and not a failure; it
says nothing about the files at all.

If a `warp-server` checkout is available, its parser tests are the authority.
Run them from that checkout, not from the Factory repository:
### Never imply a check that did not happen
On exit `2`, say plainly that validation did not run and why. Do not describe
the files as valid, correct, or ready, and do not substitute your own reading
of the schema for a verdict. If you cannot reach a server and the change
matters, say so and let the user decide.

```bash
go test ./logic/factoryfile
```
On exit `0`, repeat the sentence the validator prints rather than paraphrasing
it into something stronger. A pass means the parser and the state-independent
checks agreed; it does not mean the tree will apply.

When the Factory is already registered, a server plan is the strongest
Validation resolves no server state. Model IDs, environment IDs, secret names,
runner names, Scorer model IDs, MCP server IDs, integration availability, and
the values of Linear and Slack name aliases are all checked when the plan is
applied. The response lists what it did not check, including any deferred name
aliases; report that distinction rather than claiming a tree is fully verified.

If no Python 3 interpreter is available, do not install one or claim the tree
was validated without the user's approval. Check the changed document against
the fetched schema by hand and report that automated validation was
unavailable.

When the Factory is already registered, a server plan remains the strongest
available check. See `references/validation.md` for diagnostic codes and how to
read them.

Expand All @@ -128,31 +148,22 @@ read them.
non-empty `filter.schedule_ids`, and never both.
- Linux runners require `platform.linux.dockerImage`. A runner with no
`platform` section defaults to Linux and will fail for that reason.

## Schema drift and version skew
The format is `v1alpha1` and still changing. `logic/factoryfile` in
`warp-server` is the authority; these schemas only mirror it.

They also ship inside your Warp version rather than coming from the server, so
they can be older than the server the Factory syncs against. A field the server
added after your version was built will be reported here as unknown.

Because of that:

- Never delete, rename, or rewrite a field only because the validator calls it
unknown. On a file you did not author, that is at least as likely to be a
newer field as a mistake. Leave it, and say the schemas may be behind.
- Treat unknown-field reports on your own new edits as real. You are the one
who just introduced the field.
- If the server and these schemas disagree, the server is right. Say the
bundled schemas look stale rather than working around the validator by
skipping it.
- If the validator reports that it does not describe the tree's
`schemaVersion`, it stopped instead of applying `v1alpha1` rules to a format
it does not know. Validate with the server; never downgrade `schemaVersion`
to make the local run pass.

If you are editing the bundled schemas themselves rather than a Factory tree,
their openness is deliberate and load-bearing. Read the "If you are changing
these schemas" section of `references/validation.md` before tightening
anything.
- Trigger filter keys depend on the `(provider, event)` pair. Some fields have
a friendlier authoring spelling that the server rewrites for you: GitHub
`baseBranches` and `prNumbers`, Linear `teams`, `projects`, `states` and
`issues`, and Slack `channels`, `users` and `itemUsers`. Each stands in for
its canonical key, and declaring both is an error. The Linear and Slack ones
name objects the server looks up at apply time, so they take a plain list of
names rather than an `in`/`not_in` matcher.

## Do not add a local copy of the format
It is tempting to bundle the schema, or to reimplement a few checks here so
authoring works offline. Both have been tried and removed. A copy inside a Warp
release is routinely older than the server it is used against, and a stale copy
does not fail quietly: it reports a valid field as unknown, and an agent trying
to get to a clean run deletes working configuration to satisfy it. That has
already happened once, to Linear and Slack trigger aliases the server accepts.

Reporting that a tree was not checked costs a little. Reporting the wrong
answer costs correct configuration. Fetch the format when you need it; say
nothing when you cannot.
Loading
Loading