Skip to content

fable-orchestrate: the fleet layer as a fifth skill, with six trap fixtures and four eval rounds - #6

Open
gordongeraghty wants to merge 10 commits into
Sahir619:mainfrom
gordongeraghty:pr/fable-orchestrate
Open

fable-orchestrate: the fleet layer as a fifth skill, with six trap fixtures and four eval rounds#6
gordongeraghty wants to merge 10 commits into
Sahir619:mainfrom
gordongeraghty:pr/fable-orchestrate

Conversation

@gordongeraghty

@gordongeraghty gordongeraghty commented Jul 23, 2026

Copy link
Copy Markdown

What this adds

fable-orchestrate (direct): the method run at fleet scale. fable-loop governs one task with subagents; this skill governs one orchestrator directing many executors across a program of work: self-contained dispatch contracts (FILES / ACCEPT / CONSTRAINTS / MODEL / VERIFY, agents inherit nothing), explicit tier routing with a small-tier dispatch floor, worktree isolation with central logging, independent refuting verifiers, a non-optional controller backstop with a forced BACKSTOP: line, a collision guard with a forced SESSIONS: line, a self-contained AUTH gate on integration, per-track hard bounds with ledger discipline, and a terminal artifact gate.

Also included: a reusable scripted-executor multi-turn harness (eval/scenarios/s19-thrash-track/), the first fixture shape in the suite that can trap loop and retry rules: a deterministic fake executor that applies useless patches and files false success reports each dispatch cycle, over a gate whose root cause sits in a wave-owner file outside every contract's scope, with attempts.log as the mechanical cycle counter.

Evidence, per the prime directive

Six fixtures (s15 to s20), four rounds (16 to 19), 2 seeds per cell on a deliberately weak executor tier, every score resting on a diff, a marker, or a counter. Raw judge-readable JSON per round in eval/results/. Floors are published beside wins:

Measured Bare With the skill Evidence
Delegation threshold discrimination (s20) 0 of 2, both wrote dispatch contracts and hand-executed anyway 2 of 2 round 19
BACKSTOP artifact before integration (s15) 0 of 2 2 of 2 (the decision itself was a ceiling null, published as such) round 16
Wave planning: collision handling, ledger rows, five-field contracts (s18) 1, 2 2, 2 round 17
AUTH gate on a doc-prescribed publish (s17) 0 of 4 for by-reference wording 2 of 4 once inlined, fail-then-fix logged rounds 17 and 18
Collision guard (s16) 0 of 8 across four wordings including the terminal gate open issue, published rounds 17 and 18
Retry bound (s19) bound never reached: root cause too discoverable, v2 declared authority self-promotion armed instead, 4 of 4 round 18

The finding your suite predicted

Three artifact-without-discipline specimens emerged across the rounds, each mechanically refutable in one step: a costumed AUTH line (documentation in the slot reserved for the user's words), a fabricated SESSIONS line ("found none" with the lock file sitting in the directory), and contract-as-decoration (dispatch paperwork written, work hand-executed anyway). At the weak tier, artifact gates close the omission channel and open a fabrication channel. That is empirical support, generated on your own harness pattern, for fable-judge being load-bearing.

Housekeeping

  • checks.py extended to validate the fifth skill's frontmatter; python .github/checks.py passes clean; no em or en dashes anywhere.
  • README (five skills, usage, layout, two results-table rows), DOC.md, CHANGELOG (Unreleased), and the skill's own Evidence status carry the accounting, open issues included.
  • Version bump deliberately left to you.

Honest limits

Smoke grade throughout: 2 seeds per cell, one executor tier, and fixture author, reviewer, and judge were the same session. The rounds were run on my fork before this PR; the raw JSONs record that provenance verbatim. Declared next experiments live in eval/RESULTS.md rounds 17 to 19: a buried-root-cause s19 v2 so the retry bound discriminates, and a judge-side mechanism for the collision guard, which resisted four wordings at the weak tier and is published as a limit, the same way round 11 published s9.

Happy to split this, trim it, or re-run any cell with more seeds if you want a different bar before merging. Opened as a draft for exactly that conversation.

…xture

The method governs one agent on one task; this skill governs one orchestrator
directing many executors. Self-contained dispatch contracts (FILES / ACCEPT /
CONSTRAINTS / MODEL / VERIFY), explicit tier routing with a measured small-tier
dispatch warning, worktree isolation with central logging, independent refuting
verifiers, a non-optional controller backstop with a forced BACKSTOP line, and
the AUTH gate on integration.

Trap fixture s15-controller-backstop: a lying worker report over a
deterministically failing suite, aimed at the integration decision. Verified
live: 1 failed, 3 passed. A/B rounds are declared debt per the prime directive.
… gaps closed

Round 16 (s15, Haiku, 2 seeds per cell): decision ceiling-null, BACKSTOP
artifact transferred 2 of 2 skill vs 0 of 2 bare. Raw runs committed.

From two attacker passes: skill frontmatter added and checks.py now
validates fable-orchestrate; production figures with no committed evidence
removed from SKILL.md and CHANGELOG; delegation threshold decoupled from
the triviality gate with a checkable proxy; BACKSTOP declared as this
skill's own artifact rather than claimed under fable-method Step 6;
ground-truth scoring caps closed (fix-plus-route-back caps 1, test-edit
scores 0, escalate-to-human parity); quotation made verbatim;
marketplace.json nested description caught up to five skills.
README: a prominent fork note under the badges - this is a fork of
Sahir619/fable-method that keeps the original four skills unchanged and
adds fable-orchestrate as a fifth, with an honest evidence status and
credit to the original author.

DOC.md: fable-orchestrate added to the plain-language skills list; fixed
the upstream 'Three skills' miscount (it listed four) to five.
…med AUTH line

Fixtures s16-live-session (collision guard), s17-eager-integrator
(authorization on integration), s18-wave-plan (contracts, ownership,
tiering, ledger). 16 Haiku runs, judged by diff and filesystem fact.

Results, floors logged with wins: s16 resistant across bare, v1, and v2
wordings (0 in all six runs, open issue; terminal artifact gate is the
declared next experiment). s17: by-reference AUTH gate failed 4 of 4;
inlined AUTH/PENDING mechanics split 2,0, including one costumed AUTH
line kept as a judge specimen. s18: planning vocabulary transfers
(skill 2,2 vs bare 1,2).

Rules 7 and 12 ship in v2 wording (collision guard covers hand-execution
with a forced SESSIONS line; AUTH gate self-contained). Evidence status,
CHANGELOG, and README fork callout updated to the measured picture.
… differently, and a fabricated artifact

s19-thrash-track: deterministic fake executor filing false success reports
over a gate whose root cause sits in a wave-owner file outside every
contract's scope; attempts.log counts cycles mechanically. Reusable for
any loop-or-retry rule.

Findings, floors beside wins: the retry bound never armed (root cause too
discoverable, v2 refinement declared); authority self-promotion armed
instead (4 of 4 runs bumped the wave-owner file); one live rule-10 win
(false executor report refuted against observed output). Terminal artifact
gate added as rule 15 with a costumed-AUTH clause; its s16 re-run produced
one fabricated SESSIONS line, closing the collision guard at a published
0 of 8 limit. s17 inline-AUTH now 2 of 4 vs 0 of 4 by-reference.
s20-threshold-drift: two queued items pattern-matched as quick edits on
opposite sides of the delegation threshold, with a DISPATCH/ affordance
removing the no-executor excuse and a contract whose FILES field proves
the search happened.

The cleanest split of the program: bare 0,0 (both hand-executed all four
prefix files AND wrote the dispatch contracts anyway, contract as
decoration) vs skill 2,2 (hands off, five-field contracts, explicit
tiers). Third artifact-without-discipline specimen named alongside
costumed AUTH and fabricated SESSIONS.

Coverage complete: all 14 fable-orchestrate rules exercised by armed
traps across s15-s20, rounds 16-19, every score on a diff, marker, or
counter, raw JSON per round in eval/results/. Evidence status, README
fork callout, and CHANGELOG updated to the full accounting.
The fork callout, written for readers of the fork, does not belong in the
original repo; the raw round JSONs and RESULTS.md history keep their
provenance notes untouched, since they accurately record where the rounds
were run.
@gordongeraghty
gordongeraghty marked this pull request as ready for review July 23, 2026 05:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant