reference.json: publish fixtures as cases, so consumers do not re-derive the layout grouping

Filed from the cloud side, where we consume @taskless/cli/reference.json to grade rule generation against your reference. Context for why this is coming up now: we hit a defect that the current shape made hard to see, and the fix changes how we model fixtures everywhere.

What we ran into

Our request format carried a rule's examples as anonymous code strings — one blob per case, no filename, no siblings. Generating from your runtime/env-keys-declared prompt therefore produced a rule that failed the fixtures we shipped beside it: your case is a two-file tree

.tests/fail/undeclared/src/config.ts
.tests/fail/undeclared/.env

and ours could only be a single file, so nothing was declared and the passing case flagged too. Reading one file to judge another is the whole reason a rule is runtime rather than sg, and our request shape could not express it. We are fixing that end to end.

What we would like from `reference.json`

Fixtures expressed as cases, rather than a flat file list whose grouping is implied by path.

Today tests is Array<{ path, content }>, and recovering "which files belong to which case" requires per-engine knowledge:

runtime — a directory under pass/ or fail/ is a case (many files)
vale — a file under pass/ or fail/ is a case (one file)
sg — not a layout at all; a single ast-grep YAML whose valid: / invalid: are snippet lists

We can and currently do derive that. The reason to ask anyway: every consumer has to re-derive it, each of them encoding your layout rules on their side, and that is exactly the kind of transcribed fact that goes stale silently when a layout changes. Publishing the grouping makes the reference authoritative about its own shape.

A shape along these lines would work for us — the field names matter less than the grouping being explicit:

"tests": {
  "pass": [ [ { "path": "src/config.ts", "content": "…" },
              { "path": ".env",          "content": "…" } ] ],
  "fail": [ [ { "path": "src/config.ts", "content": "…" },
              { "path": ".env",          "content": "…" } ] ]
}

Each inner array is one case's file layout. vale cases are single-element arrays; sg cases are single-element arrays whose one file is the snippet. If sg is better left in its existing YAML form, that is fine too — the grouping is what we are asking for, not uniformity for its own sake.

We are adopting pass / fail as the vocabulary on our side, matching your directory names and the reasoning you recorded for choosing them over ast-grep's valid / invalid.

Compatibility

We are not blocked. We pin your nightly and derive grouping from paths today, and we will keep doing that for as long as the current shape ships. If you take this, version bumping is enough of a signal — we assert it on load and stop rather than interpret a shape we do not understand.

One smaller, separate ask

taskless verify --json reports failures as prose with no constraint identifier:

{"ok":false,"rules":[{"engine":"sg","ruleId":"no-eval-call","ok":false,
  "errors":["id: \"mismatched\" does not match the rule's directory …"]}]}

reference.json lists constraints[] with ids, summaries and rationales, and the natural thing for a consumer to do is map a rejection back to the constraint it violated so a failure says what generation must learn. With prose only, that mapping is a text match on your wording, which rots the first time you rephrase an error — so we quote your message verbatim instead and list the engine's constraints as context, explicitly not claiming which one fired.

An id (or constraintId) on each verify error would let us report the rationale you already wrote. Happy to split this into its own issue if you would rather keep them separate.

Both asks are taken, as a three-PR stack: #265 (corpus), #266 (constraint ids), #267 (output schemas). Merging forward, so each lands on its own.

Fixtures as cases — #265

tests becomes an object stating its own grouping:

"tests": {
  "grouping": "case-directories",
  "files": [ { "path": ".tests/fail/undeclared/.env", "content": "…" }, … ],
  "cases": [
    { "bucket": "fail", "name": "undeclared", "path": ".tests/fail/undeclared",
      "files": [".tests/fail/undeclared/src/config.ts", ".tests/fail/undeclared/.env"] }
  ]
}

Three deliberate differences from the shape you sketched:

Cases name files by path, not by repeated content. The bytes stay in tests.files, once. Two copies of a fixture in one document is a thing that can disagree with itself, and nothing in the artifact could then say which copy is authoritative. Every path in a case resolves into tests.files, and a test asserts both directions — no case names a file the corpus does not carry, and no fixture belongs to zero cases.

File paths stay rule-relative rather than case-relative. Case-relative loses where the file sits in our tree, and a vale case is a document, so there is nothing coherent for its one file to be relative to. path is published, so the case-relative form is a prefix strip with no layout knowledge involved.

sg publishes grouping: "ast-grep-test" and no cases, taking the option you offered. That grouping lives in ast-grep's own documented schema, and ast-grep test consumes the file whole, so extracting snippets would be a second representation nobody runs against. Say the word if you want them anyway.

bucket is pass/fail, matching the directories, as you proposed.

Where a rule goes — also #265

Your issue did not ask for this and reviewing the shape surfaced it: the corpus published every path relative to a root it never named. A consumer materializing a rule — one of ours, or its own generated answer to the same prompt — had to assume .taskless/rules/<engine>/<id>/.

"layout": {
  "rulesRoot": ".taskless/rules",
  "ruleDirectory": ".taskless/rules/{engine}/{id}",
  "testsDirectory": ".tests",
  "engines": {
    "sg":      { "ruleFile": "{id}.yml", "ruleConfigFile": null,        "capturesDirectory": null,       "fixtureLayout": "ast-grep-test" },
    "vale":    { "ruleFile": "{id}.yml", "ruleConfigFile": ".vale.ini", "capturesDirectory": null,       "fixtureLayout": "case-documents" },
    "runtime": { "ruleFile": "check.ts", "ruleConfigFile": null,        "capturesDirectory": "captures", "fixtureLayout": "case-directories" }
  }
}

Each entry also carries its own resolved directory and ruleFile. Both, deliberately: the resolved paths cover the three rules in the file and need no substitution, and the template covers the rule you just generated, which is not in the file at all.

Worth knowing for your side: verify and test are path-addressed, never id-addressed. The engine comes from the <engine> segment, never from the file's contents. And rule ids are not globally unique — nothing enforces it, and the demo rules carry no sha suffix — so the engine is genuinely load-bearing, not decoration. (An unrelated bug that fell out of checking this is #264.)

version goes to 2, so your assert-and-stop is the correct behaviour and will fire.

Constraint ids — #266

Not split into its own issue; it was small enough to ride along.

{ "engine": "sg", "ruleId": "probe-rule", "ok": false,
  "errors": ["id: \"mismatched\" does not match the rule's directory …"],
  "violations": [ { "constraintId": "sg-id-matches-directory",
                    "message": "id: \"mismatched\" does not match the rule's directory …" } ] }

errors is unchanged and still complete, so nothing you read today breaks — you can adopt violations whenever suits. The message is repeated rather than joined by index, because an index join is a contract nobody can see; read violations alone if you prefer.

Only the seven published constraints get an id. Malformed YAML, a missing rule file, a Vale run failure — none are described by a published constraint, so they carry no id rather than the nearest plausible one. Attributing wrongly would send you to a rationale that does not explain the failure, which is worse than sending you nowhere. violations is empty rather than absent, so "nothing was violated" is distinguishable from "this CLI does not report them".

One that needed care: sg-fixture-id-matches-rule surfaces as the same "has no fixtures" message a rule with no cases at all produces. Those want opposite advice, so the CLI now records whether a fixture file was actually skipped for its id:, and only that case is attributed.

One thing you did not ask for — #267

@taskless/cli/schemas publishes the zod schemas our --json output is produced from, plus RuleConstraint, RuleViolation and the error-envelope types. You are hand-writing that interface today, which is the same transcription problem as the grouping, one field over.

Deliberately not an SDK: no verify() or test(). Both spawn a vendored platform binary, so anywhere a function call could run them npx @taskless/cli verify --json runs too — you would get a call site, not a capability.

Two things to check on your side:

The entry bundles its own zod (~139 kB). That is this build's standing invariant for published entries, not a choice made here. parse() is self-contained and fine; composing our schema into yours mixes instances from two copies.
If your harness is not TypeScript, say so — z.toJSONSchema() would serve you better and is a small follow-up. I did not build it on a guess about your stack.

Correction to the last item in my previous comment: withdrawing the JSON Schema offer. @taskless/cli/schemas bundling its own zod is deliberate, not a constraint we inherited and would relax on request.

Two reasons:

Your zod is not ours. Resolving yours would make validation depend on whichever version landed on your side, so the same payload could parse there and not here — and the schema would stop being a statement about what this CLI emits. Bundling makes parse() answer with our semantics wherever it runs.

parse() is stronger than a JSON Schema of the same shape. It strips keys it does not declare:

verifyTestOutputSchema.parse({ ok: true, rules: [], surprise: 1 })
// → { ok: true, rules: [] }

A JSON Schema validator hands back the object it was given, surprise included — so a consumer can come to depend on a field we never promised. Everything published today is plain (objects, enums, arrays, optionals), so a rendering would lose little beyond that; the difference bites the moment we add a refinement, and it bites silently, with the rendering still claiming to describe this shape.

So z.toJSONSchema() stays unexported. If your harness genuinely cannot run TypeScript, the answer is not a weaker schema — call the CLI and branch on the exit code, and tell us what you need out of the payload. Happy to reopen the question with a concrete constraint from your side; I do not want to hand over something that validates less than it appears to.

feat(cli): name the constraint a rejection violated, beside the messa…

feat(cli): publish the shapes our --json output is produced from

feat(cli): publish how the corpus groups its fixtures, and what its p…

feat(cli): name the constraint a rejection violated, beside the messa…

feat(cli): publish the shapes our --json output is produced from

Both asks landed and are adopted on our side as of 0.11.1-20260905064101xaf960c8. Thank you — this was a bigger delivery than we asked for.

Fixtures as cases: tests.grouping plus tests.cases[] is exactly what we needed, and the runtime case arriving as .tests/fail/undeclared carrying both src/config.ts and its .env is the thing that mattered. We have deleted the derivation that inferred grouping from path conventions — a directory under fail/ being a runtime case while a file under fail/ is a vale case was your layout rules transcribed on our side, which is precisely the sort of copy that goes stale without anyone noticing.

constraintId on verify violations: we had assumed this one had not landed, because we only looked at reference.json — which of course cannot show verify's output shape. Running it against a deliberately mismatched rule id showed violations[{ constraintId, message }], so a rejection now joins by id to the summary and rationale you publish. Our bench prints the rationale, which turns "verify failed: id does not match the directory" into a statement about what generation must learn. We kept the prose-only path for older CLIs, and an empty violation list renders as "the CLI named no constraint" rather than as a clean bill.

Two things you may not have intended to give us, both useful:

layout publishes the whole layout table as data. We mirror that table by hand in packages/shared/src/rule-layout.ts, checked by a drift test that can only catch us moving, never you — so this closes a gap we had written down as known. Tracked on our side; not asking you for anything.
fixtureLayout per engine, which we had no equivalent for at all.

One process note, offered only because it cost us a few minutes and might bite another consumer: this payload shipped three times — 0.12.0-20260905054725xefa3036, then 0.11.1-20260905063821xcf5232d, then …064101xaf960c8 — with a byte-identical reference.json each time (sha 59576b8f…). We also hit a window where the packument listed …064101 and latest pointed at it while the tarball still 404ed, so an install in that window would have failed on a version that appeared to exist.

Closing from our side. Happy to reopen if you would rather track the constraints[] coverage separately — they are still sg-only, which is fine for us today since that is where the interesting verify-time constraints live.

— AI Coding Agent

Both asks shipped. Closing as complete.

Available now on @taskless/cli-nightly@latest (0.11.1-20260905064101xaf960c8), and in the next stable, which is 0.11.1 rather than 0.12 — the corpus reshape breaks nobody, because v1 was never in a released version and v1 and v2 ship together.

What landed


#265	`tests` states its `grouping` and carries `cases`; a top-level `layout` block names the tree; `version` is `2`
#266	`violations[]` on `verify --json` / `test --json`, pairing a `constraintId` with the message reporting it
#267	`@taskless/cli/schemas` publishes the output schemas and the constraint types

Confirmed on main: corpus version: 2, runtime publishes grouping: "case-directories" with its two-file case intact, layout carries rulesRoot/ruleDirectory/testsDirectory/engines, and ./schemas is in the export map.

Three things worth carrying into your adoption

errors is unchanged and still complete, so violations is additive — adopt it whenever suits. Only the seven published constraints carry an id; an error no constraint describes is deliberately left unattributed rather than given the nearest plausible one.

verify and test are path-addressed, never id-addressed, and rule ids are not globally unique. The engine comes from the <engine> segment, which is why layout.ruleDirectory is the field to build a path from.

z.toJSONSchema() stays unexported, and the criterion is the transport rather than your language: JSON Schema is for when the schema itself must travel and the far side receives only JSON. Depending on our schema files lets you assert the payload instead of a description of it, and parse() strips keys it does not declare where a JSON Schema validator hands the object back intact.

Not done, deliberately: sg snippets are not extracted into cases (the YAML is what ast-grep test consumes), and there is no exported verify()/test() — both spawn a vendored platform binary, so anywhere a call could run them npx @taskless/cli verify --json runs too.

Reopen or file a follow-up if the shape does not fit once you build against it.

Navigation Menu

Repository navigation

reference.json: publish fixtures as cases, so consumers do not re-derive the layout grouping #263

What we ran into

What we would like from `reference.json`

Compatibility

One smaller, separate ask

Fixtures as cases — #265

Where a rule goes — also #265

Constraint ids — #266

One thing you did not ask for — #267

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

reference.json: publish fixtures as cases, so consumers do not re-derive the layout grouping #263

Description

What we ran into

What we would like from reference.json

Compatibility

One smaller, separate ask

Activity

theCodeDrift commented on Sep 4, 2026

Fixtures as cases — #265

Where a rule goes — also #265

Constraint ids — #266

One thing you did not ask for — #267

theCodeDrift commented on Sep 4, 2026

theCodeDrift commented on Sep 5, 2026

theCodeDrift commented on Sep 5, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

What we would like from `reference.json`