From 359741593db7a644a999dcbf70595345da85b7be Mon Sep 17 00:00:00 2001 From: Jacek Tomaszewski Date: Tue, 25 Aug 2026 12:48:34 +0200 Subject: [PATCH 1/4] feat(prior-art): research how others solve it, with every claim sourced MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit "How do other big players do this?" is the highest-confabulation question we ask an LLM: it returns fluent, sourceless prose about what Stripe/Postgres/ everyone does, and that prose then gets pasted into a spec as justification. No skill covered it — `brainstorm` and `zoom-out` both generate alternatives from inside our own head, and `om-spec-writing` gestures at "research against market leaders" without saying how. The skill's value is the discipline, not the prompt: - every claim about a named system is graded documented / inferred / folklore, and folklore may appear but may never justify a decision; - comparables are chosen by shared constraint, never by fame — the big-player fallacy is this question's main failure mode; - the format can express "they arrange things so the question never arises", "they do it and regret it", and "there is no consensus", which a fill-in-the- options table cannot; - our own repo, specs and tracker are searched first — we have often already decided it; - search queries are outbound to third parties, so they carry no client- identifying detail, same standing rule as the publishing skills; - it closes on a verdict against our constraints plus an explicit unknowns list, sized to drop into the spec/ADR/ticket the user asked for. Read-only throughout. Pairs with `zoom-out` as the other tool for seeing past your own frame — zoom-out gets a fresh context, prior-art gets the outside world. --- .claude-plugin/plugin.json | 4 +- README.md | 3 +- skills/prior-art/SKILL.md | 80 ++++++++++++++++++++++++++++++++++++++ 3 files changed, 84 insertions(+), 3 deletions(-) create mode 100644 skills/prior-art/SKILL.md diff --git a/.claude-plugin/plugin.json b/.claude-plugin/plugin.json index 037d58b..3c23c81 100644 --- a/.claude-plugin/plugin.json +++ b/.claude-plugin/plugin.json @@ -1,7 +1,7 @@ { "name": "fsh", - "description": "Full Stack House Claude Code skills (idea brainstorming & kickoff, delivery workflow, bug/flake hunting, change explanation, hands-on test drives, spec/ticket/PR polishing, documentation & agent-instruction audit, status reporting, design, document redlines)", - "version": "1.2.2", + "description": "Full Stack House Claude Code skills (idea brainstorming & kickoff, prior-art research, delivery workflow, bug/flake hunting, change explanation, hands-on test drives, spec/ticket/PR polishing, documentation & agent-instruction audit, status reporting, design, document redlines)", + "version": "1.3.0", "author": { "name": "Full Stack House", "url": "https://fullstack.house" diff --git a/README.md b/README.md index 355c18a..061e5be 100644 --- a/README.md +++ b/README.md @@ -29,8 +29,9 @@ read what a counterparty changed in a contract. | [`docx-diff`](./skills/docx-diff/SKILL.md) | Reconstruct a redline between two `.docx` versions when the counterparty edited without tracked changes: pandoc → sentence-level unified diff → a classification of which changes are material and who they favour. Needs `pandoc`. | | [`bro`](./skills/bro/SKILL.md) | Restate the last message in plain human language — no jargon, one human talking to another. Manual-invoke only. | | [`zoom-out`](./skills/zoom-out/SKILL.md) | Break mid-task tunnel vision: restate the goal from the original request, mark sunk work ignorable, measure the decision space, get a fresh-context second opinion (subagent that never sees the current approach), present 2–3 options-in-kind + a recommendation. Analysis only until the user picks. | +| [`prior-art`](./skills/prior-art/SKILL.md) | "How do other big players do this?" — answered in a checkable form: sharpen the question into a decision + our constraints, look inside our own repo first, pick comparables by constraint rather than fame, sweep by source class (docs / source / postmortems / the dissent), grade every claim documented / inferred / folklore, close on a verdict plus what's still unknown. Read-only. | -`explain`, `brainstorm`, `kickoff`, `overnight`, `deliver`, `upstream-pr`, `pr-polish`, `ticket-refresh`, `ticket-polish`, `spec-polish`, `docs-audit`, `review-queue`, `bug-hunt`, `flake-hunt`, `project-status`, `design-polish`, and `design-explore` are **repo-agnostic** — they derive +`explain`, `brainstorm`, `kickoff`, `overnight`, `deliver`, `upstream-pr`, `pr-polish`, `ticket-refresh`, `ticket-polish`, `spec-polish`, `docs-audit`, `review-queue`, `bug-hunt`, `flake-hunt`, `project-status`, `design-polish`, `design-explore`, and `prior-art` are **repo-agnostic** — they derive project-specific commands, paths, and policy at runtime (see [Skill profile](#skill-profile) below). A repo with its own sharper, hardcoded variant can keep it in its `.claude/skills/` alongside these (plugin skills are namespaced, so they don't collide — see Install). diff --git a/skills/prior-art/SKILL.md b/skills/prior-art/SKILL.md new file mode 100644 index 0000000..fb72391 --- /dev/null +++ b/skills/prior-art/SKILL.md @@ -0,0 +1,80 @@ +--- +name: prior-art +description: Research how other systems actually solve a design problem, with every claim tied to a primary source. Use when invoked as "/prior-art", when the question is "how do other big players/systems do this?", "is X a good pattern?", "what's standard practice for Y?", "has anyone solved this already?", before committing to a non-obvious architectural decision, or when a spec/ADR/PR asserts what the industry does without citing anything. Read-only — it researches and recommends, it never implements. +--- + +# prior-art + +You are running the **prior-art** skill. Someone is about to decide something and wants to know how the rest of the world decided it. + +**The premise: this is the highest-confabulation question anyone asks an LLM.** "How does Stripe handle idempotency", "do big systems enable RCSI", "what do most teams do for X" — all of it produces fluent, confident, sourceless prose that is often part-invented, and it is *dangerous* precisely because it then gets pasted into a spec as justification. A wrong answer here doesn't lose an argument; it gets cited. + +Your job is not to answer the question. It is to answer it **in a form the reader can check**. + +**Hard rule: read-only.** Research and recommend. Don't implement, don't edit the spec, don't post the comment — hand back text the user places. + +## 1. Sharpen the question into a decision + +Restate what is actually being decided, in one sentence, as a choice between options: *"Should we enable read-committed snapshot isolation on the sync database, or keep the current locking behaviour and fix contention another way?"* Not *"is RCSI good?"* — no design property is good in the abstract, only under constraints. + +Then write down **our constraints** — the three or four facts that make an outside answer transferable or not. Scale, write pattern, consistency requirement, team size, uptime budget, whether we control the schema, what we can't change. If you can't name them, ask; a survey without them ranks by fame and will mislead. + +## 2. Look inside before you look outside + +Cheapest source first, and the one most often skipped: search the consuming repo's `docs/`, specs, ADRs, `CLAUDE.md`/`AGENTS.md`, and the tracker for this exact decision. It is common to find that we already decided it, decided it once and drifted, or hit the failure mode the outside world is about to warn us of. Report that first — it outranks any external finding. + +## 3. Choose comparables by constraint, not by fame + +Pick systems that **share our constraint**, then say why each qualifies. A niche database with our write pattern is worth more than a household name with a different one. + +**The big-player fallacy is the main failure mode of this skill.** "Stripe does X" is only evidence if Stripe's problem is our problem — usually they operate at a scale that justifies costs we can't carry, or under consistency requirements we don't have. Where a comparable is famous but not comparable, say so and use it as a *contrast*, not support. + +Favour targets whose behaviour is inspectable: open-source systems (you can read the code), vendors with real reference docs, and teams that publish postmortems. A closed system you can only guess about produces exactly the folklore this skill exists to filter. + +## 4. Sweep by source type, not by search box + +Fan out subagents — one per source class, because each finds what the others structurally cannot: + +- **Primary docs** — the vendor's or project's own documentation, including its "when not to use this" and limitations pages, which is where the honest trade-off usually hides. +- **Source and configuration** — what the code, default config, or migration actually does. Beats every blog post. +- **Field reports** — engineering blogs, postmortems, conference talks, mailing-list/RFC threads, issue trackers. Postmortems are the highest-value genre here: they describe the pattern *failing*, which docs never do. +- **The dissent** — deliberately search for who argues against it and why. A survey that finds only advocates hasn't finished. + +Each subagent returns claims with URLs, never prose summaries you'd have to re-verify. + +**Queries go to third parties.** Phrase every one generically — no client names, repo names, internal ticket or spec IDs, module or env-var prefixes, hostnames, or paths. "SQL Server snapshot isolation for an ERP sync workload", never the client's system by name. This is the same standing rule as the publishing skills; searching is publishing. + +## 5. Grade every claim, and let the weak ones die + +Label each finding: + +- **Documented** — a specific URL says this. Quote or link it. +- **Inferred** — you read the code/config and concluded it. Say what you read. +- **Folklore** — widely repeated, no source found. + +**Folklore may appear in the report but may never be used as justification**, and label it as such rather than dropping it silently — "everyone says X and nobody documents it" is itself a finding about how well-founded the practice is. If a claim you'd have liked to make has no source, say "couldn't verify" and move on. An honest three-source answer beats a confident ten-source one, and *"there is no consensus"* is a legitimate, common, and useful result. + +Two findings this format must be able to express, because a fill-in-the-options table can't: + +- **They avoid the situation.** The most valuable answer is often that comparable systems arranged things so the question never arises. That reframes the decision instead of settling it. +- **They do it, and regret it.** Adoption is not endorsement. Prefer sources that report the outcome. + +## 6. Verdict, against our constraints + +Close with the part that makes the research usable: + +- **What comparable systems do**, in a few lines — grouped by the *reason* they chose it, not by company. +- **What that implies for us**, given the step-1 constraints — including where our situation genuinely differs and the majority answer therefore doesn't apply. +- **One recommendation**, with the strongest argument against it stated fairly. +- **What remains unknown**, and what would settle it — a benchmark to run, a doc to find, a person to ask. This list is not optional; it's what stops the reader treating the survey as complete. + +Size it to drop into the spec section, ADR, ticket comment, or PR reply the user asked for — not a standalone essay. Hand it back; let them place it. + +## Hard rules + +1. **No unsourced claim about a named system.** Documented, inferred, or explicitly folklore. There is no fourth category, and "I know this" isn't one. +2. **Comparable by constraint, never by fame.** Every included system carries one line on why its problem is our problem. +3. **"No consensus", "they avoid it", and "couldn't verify" are results.** Manufacturing a clean majority out of thin evidence is the failure this skill exists to prevent. +4. **Search queries carry no client-identifying detail.** Generic phrasing, always. +5. **Read-only.** No edits, no posts, no implementation — the user places the output. +6. **End with a verdict and an unknowns list.** A survey without a recommendation hands the work back; a recommendation without unknowns oversells it. From 61af5ba2245ddf1d5814f891a2bcfc5f71dd433b Mon Sep 17 00:00:00 2001 From: Jacek Tomaszewski Date: Tue, 25 Aug 2026 13:13:26 +0200 Subject: [PATCH 2/4] fix(prior-art): bind the fan-out and add the verification step MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Ran the skill on five problems on fresh contexts — a DB isolation setting, an e-commerce order-status model, durable background work, a financial ledger, and ERP module structure — and graded what came back. Five findings reproduced on all five runs; each maps to a change here. What held, and is now stated more strongly: - §2 "look inside before you look outside" was the highest-value step every single time — it found decisions we had already made and incidents we had already survived, which outrank anything published outside. Promoted. - "They avoid the situation", "they do it and regret it" and the licence to report no consensus all produced the best findings in at least one run. Kept, with the licence explicitly not an instruction to hedge: one run correctly reported a real consensus where one exists. - "Comparable by constraint, not fame" earned its place, but every run had to derive the ONE axis that actually sorted the evidence rather than being prompted for it. §3 now asks for the axis, not a constraint list. What broke: - The mandated subagent fan-out is the skill's most reliable failure. It exhausted the shared search budget in minutes on every run, leaving nothing for the orchestrator's own checks — in one run the dissent sweep, which a hard rule protects, issued zero queries and its silence was nearly reported as "no dissent found". §4 now budgets the fan-out, runs dissent first, requires a tool-failure to be distinguishable from an absence, and tells you to cross-check sweeps that disagree. - Hard rule 4 was unenforceable through the very mechanism the skill mandates. Sweeps mined the inherited repo CLAUDE.md, inferred the client's ERP product and vendor, and searched for them by name — despite being told to stay generic. The rule now says where the leak comes from and requires the sweep prompts to be sanitized explicitly. - The Documented/Inferred/Folklore ladder is a self-report, and passes a well-formatted wrong attribution straight through: one sweep credited one commerce platform's source code to another under a correctly-applied label. All three runs that hit this invented the same missing category. Grading is now two axes — strength × whether you opened it — and a new §5 requires you to open every source the verdict rests on. - The output contract asked for something "sized to drop into a spec section" and produced a ~4,500-word essay on all five runs, converging within 15%. One document that is both a verdict and an evidence file always becomes the evidence file. Split into two artifacts. Net +223 words: the additions are load-bearing, paid for partly by cutting rhetoric that argued for rules instead of stating them. --- README.md | 2 +- skills/prior-art/SKILL.md | 98 ++++++++++++++++++++++----------------- 2 files changed, 57 insertions(+), 43 deletions(-) diff --git a/README.md b/README.md index 061e5be..5cb2470 100644 --- a/README.md +++ b/README.md @@ -29,7 +29,7 @@ read what a counterparty changed in a contract. | [`docx-diff`](./skills/docx-diff/SKILL.md) | Reconstruct a redline between two `.docx` versions when the counterparty edited without tracked changes: pandoc → sentence-level unified diff → a classification of which changes are material and who they favour. Needs `pandoc`. | | [`bro`](./skills/bro/SKILL.md) | Restate the last message in plain human language — no jargon, one human talking to another. Manual-invoke only. | | [`zoom-out`](./skills/zoom-out/SKILL.md) | Break mid-task tunnel vision: restate the goal from the original request, mark sunk work ignorable, measure the decision space, get a fresh-context second opinion (subagent that never sees the current approach), present 2–3 options-in-kind + a recommendation. Analysis only until the user picks. | -| [`prior-art`](./skills/prior-art/SKILL.md) | "How do other big players do this?" — answered in a checkable form: sharpen the question into a decision + our constraints, look inside our own repo first, pick comparables by constraint rather than fame, sweep by source class (docs / source / postmortems / the dissent), grade every claim documented / inferred / folklore, close on a verdict plus what's still unknown. Read-only. | +| [`prior-art`](./skills/prior-art/SKILL.md) | "How do other big players do this?" — answered in a checkable form: sharpen into a decision, look inside our own repo first, name the one axis that decides transferability, sweep by source class on a budget (dissent first), **open every source the verdict rests on yourself**, grade on strength × who-checked-it, and split the output into a paste-ready verdict + a separate evidence file. Read-only. | `explain`, `brainstorm`, `kickoff`, `overnight`, `deliver`, `upstream-pr`, `pr-polish`, `ticket-refresh`, `ticket-polish`, `spec-polish`, `docs-audit`, `review-queue`, `bug-hunt`, `flake-hunt`, `project-status`, `design-polish`, `design-explore`, and `prior-art` are **repo-agnostic** — they derive project-specific commands, paths, and policy at runtime (see [Skill profile](#skill-profile) diff --git a/skills/prior-art/SKILL.md b/skills/prior-art/SKILL.md index fb72391..74244f8 100644 --- a/skills/prior-art/SKILL.md +++ b/skills/prior-art/SKILL.md @@ -1,80 +1,94 @@ --- name: prior-art -description: Research how other systems actually solve a design problem, with every claim tied to a primary source. Use when invoked as "/prior-art", when the question is "how do other big players/systems do this?", "is X a good pattern?", "what's standard practice for Y?", "has anyone solved this already?", before committing to a non-obvious architectural decision, or when a spec/ADR/PR asserts what the industry does without citing anything. Read-only — it researches and recommends, it never implements. +description: Research how other systems actually solve a design problem, with every load-bearing claim opened and checked by you. Use when invoked as "/prior-art", when the question is "how do other big players/systems do this?", "is X a good pattern?", "what's standard practice for Y?", "has anyone solved this already?", before committing to a non-obvious architectural decision, or when a spec/ADR/PR asserts what the industry does without citing anything. Read-only — it researches and recommends, it never implements. --- # prior-art -You are running the **prior-art** skill. Someone is about to decide something and wants to know how the rest of the world decided it. +Someone is about to decide something and wants to know how the rest of the world decided it. -**The premise: this is the highest-confabulation question anyone asks an LLM.** "How does Stripe handle idempotency", "do big systems enable RCSI", "what do most teams do for X" — all of it produces fluent, confident, sourceless prose that is often part-invented, and it is *dangerous* precisely because it then gets pasted into a spec as justification. A wrong answer here doesn't lose an argument; it gets cited. +**This is the highest-confabulation question anyone asks an LLM.** It returns fluent, sourceless prose about what Stripe or Postgres or "most teams" do, and it is dangerous precisely because it doesn't lose an argument — it gets *cited*. Your job is not to answer the question. It is to answer it in a form the reader can check. -Your job is not to answer the question. It is to answer it **in a form the reader can check**. - -**Hard rule: read-only.** Research and recommend. Don't implement, don't edit the spec, don't post the comment — hand back text the user places. +**Read-only.** Research and recommend; don't implement, don't edit the spec, don't post the comment. ## 1. Sharpen the question into a decision -Restate what is actually being decided, in one sentence, as a choice between options: *"Should we enable read-committed snapshot isolation on the sync database, or keep the current locking behaviour and fix contention another way?"* Not *"is RCSI good?"* — no design property is good in the abstract, only under constraints. +Restate what is being decided as a choice between named options: *"Grow our existing job record into a durable lease, or adopt an off-the-shelf durable-execution engine?"* Not *"is X good?"* — no design property is good in the abstract. -Then write down **our constraints** — the three or four facts that make an outside answer transferable or not. Scale, write pattern, consistency requirement, team size, uptime budget, whether we control the schema, what we can't change. If you can't name them, ask; a survey without them ranks by fame and will mislead. +A vague question cannot be researched, only answered plausibly. If you can't state the options, that *is* the first deliverable: sharpen it, say what you assumed, and continue. ## 2. Look inside before you look outside -Cheapest source first, and the one most often skipped: search the consuming repo's `docs/`, specs, ADRs, `CLAUDE.md`/`AGENTS.md`, and the tracker for this exact decision. It is common to find that we already decided it, decided it once and drifted, or hit the failure mode the outside world is about to warn us of. Report that first — it outranks any external finding. +Search the consuming repo — `docs/`, specs, ADRs, `CLAUDE.md`/`AGENTS.md` — and the tracker for this exact decision. + +**This step outranks everything below it.** We have often already decided this, decided it and drifted, or already hit the failure the outside world is about to warn us of. An internal incident beats an external blog post: it happened to us, under our constraints, with consequences someone remembers. Report internal findings first and separately. + +## 3. Name the one axis that decides transferability + +List our constraints, then do the work that matters: identify **the single axis along which comparable systems' answers diverge**. "Is this system the book of record?" "Does an external system own the transitions?" "Do they control the whole stack?" -## 3. Choose comparables by constraint, not by fame +That axis, not a constraint list, is what sorts the evidence. Every included system gets one line: where it sits on the axis, and why that makes it comparable to us or a contrast. -Pick systems that **share our constraint**, then say why each qualifies. A niche database with our write pattern is worth more than a household name with a different one. +**Fame is not the axis.** "Stripe does X" is evidence only if Stripe's problem is our problem. Where a famous system sits on the far side of the axis, keep it as an explicit **contrast**, never as support. -**The big-player fallacy is the main failure mode of this skill.** "Stripe does X" is only evidence if Stripe's problem is our problem — usually they operate at a scale that justifies costs we can't carry, or under consistency requirements we don't have. Where a comparable is famous but not comparable, say so and use it as a *contrast*, not support. +## 4. Sweep by source class — with a budget -Favour targets whose behaviour is inspectable: open-source systems (you can read the code), vendors with real reference docs, and teams that publish postmortems. A closed system you can only guess about produces exactly the folklore this skill exists to filter. +Four classes, because each finds what the others structurally cannot: **primary docs** (including the "when not to use this" page, where the honest trade-off hides), **source and config** (what the code actually does — beats every blog post), **field reports** (postmortems, issue trackers, RFC threads — the highest-value genre, because they describe the pattern *failing*), and **the dissent** (who argues against it, and why). -## 4. Sweep by source type, not by search box +Delegating these to parallel subagents is fine, but the fan-out is this skill's most reliable way to fail. Bind it: -Fan out subagents — one per source class, because each finds what the others structurally cannot: +- **Search budget is finite and shared.** Give each sweep a hard query cap and **reserve at least a third of the total for step 5**. An unbudgeted fan-out spends everything on breadth and leaves nothing for checking. +- **Run the dissent sweep first.** It's the one a hard rule protects and the first casualty of an exhausted budget. +- **Sanitize the sweep prompts explicitly.** Subagents inherit the consuming repo's `CLAUDE.md`/`AGENTS.md` and mine it for specifics; telling them to "stay generic" does not work, because they infer the client's products and vendors from that inherited context and search for them *by name*. Give each sweep the sanitized question text plus an explicit list of terms it may not put in a query. +- **Where two sweeps disagree** about the same system, neither claim is usable until you check it yourself. +- **A sweep that returns nothing has two meanings** — nothing exists, or the tool failed. Record which. "No dissent found" from a sweep whose queries were all refused is not a finding, it's a missing measurement; label it as one and re-run before anyone cites the silence. -- **Primary docs** — the vendor's or project's own documentation, including its "when not to use this" and limitations pages, which is where the honest trade-off usually hides. -- **Source and configuration** — what the code, default config, or migration actually does. Beats every blog post. -- **Field reports** — engineering blogs, postmortems, conference talks, mailing-list/RFC threads, issue trackers. Postmortems are the highest-value genre here: they describe the pattern *failing*, which docs never do. -- **The dissent** — deliberately search for who argues against it and why. A survey that finds only advocates hasn't finished. +If search is unavailable, say so and fall back to fetching known URLs directly — but note that field reports and dissent are exactly the classes you cannot reach by guessing URLs, so the survey is incomplete in its most important dimension. -Each subagent returns claims with URLs, never prose summaries you'd have to re-verify. +## 5. Verify before you cite -**Queries go to third parties.** Phrase every one generically — no client names, repo names, internal ticket or spec IDs, module or env-var prefixes, hostnames, or paths. "SQL Server snapshot isolation for an ERP sync workload", never the client's system by name. This is the same standing rule as the publishing skills; searching is publishing. +**Open, yourself, every source the verdict rests on.** Not a sample of them — all of them. -## 5. Grade every claim, and let the weak ones die +This is the step that separates this skill from asking the question directly, and it is the one under budget pressure to disappear. Fabricated attributions arrive confidently formatted and correctly labelled: a page cited for a claim it never makes, one platform's source code credited to another, a real quote pinned to the wrong URL. Nothing about the shape of a delegated claim reveals this. Only opening it does. -Label each finding: +Budget for this before you spend on breadth. **A verified survey of three systems beats an unverified one of ten.** -- **Documented** — a specific URL says this. Quote or link it. -- **Inferred** — you read the code/config and concluded it. Say what you read. -- **Folklore** — widely repeated, no source found. +## 6. Grade on two axes -**Folklore may appear in the report but may never be used as justification**, and label it as such rather than dropping it silently — "everyone says X and nobody documents it" is itself a finding about how well-founded the practice is. If a claim you'd have liked to make has no source, say "couldn't verify" and move on. An honest three-source answer beats a confident ten-source one, and *"there is no consensus"* is a legitimate, common, and useful result. +Every claim carries both: -Two findings this format must be able to express, because a fill-in-the-options table can't: +- **Strength** — `documented` (a specific page says it), `inferred` (you read the code/config and concluded it; say what you read), or `folklore` (widely repeated, no source found). +- **Provenance of the check** — `opened` (you fetched it) or `secondhand` (a sweep reported it; nobody opened it). -- **They avoid the situation.** The most valuable answer is often that comparable systems arranged things so the question never arises. That reframes the decision instead of settling it. +The second axis exists because the first is a self-report. A well-formatted wrong attribution passes the strength label cleanly. **`documented` + `secondhand` is a claim about a subagent's formatting, not about the world** — it may appear in the evidence, never in the verdict. + +Folklore may be reported and may never justify a decision — "everyone says it and nobody documents it" is itself a finding about how well-founded the practice is. Record what you dropped and why; dropped claims are among the most useful output. + +Three results this format must be able to express, because an options table can't: + +- **They avoid the situation.** Often the most valuable answer: comparable systems arranged things so the question never arises. That reframes the decision instead of settling it. - **They do it, and regret it.** Adoption is not endorsement. Prefer sources that report the outcome. +- **There is no consensus** — legitimate and common. Equally: where a real consensus exists, say so plainly. This licence is not an instruction to hedge. + +## 7. Two artifacts, not one -## 6. Verdict, against our constraints +**The verdict** — about a page, the thing that gets pasted: -Close with the part that makes the research usable: +- what comparable systems do, grouped by the *reason* they chose it, not by company; +- what that implies for us on the step-3 axis — including, when it applies, that **the majority answer doesn't transfer**, which is a headline and not a footnote; +- one recommendation, with the strongest argument against it stated fairly; +- what remains unknown and what would settle each — a measurement, a doc, a person to ask. -- **What comparable systems do**, in a few lines — grouped by the *reason* they chose it, not by company. -- **What that implies for us**, given the step-1 constraints — including where our situation genuinely differs and the majority answer therefore doesn't apply. -- **One recommendation**, with the strongest argument against it stated fairly. -- **What remains unknown**, and what would settle it — a benchmark to run, a doc to find, a person to ask. This list is not optional; it's what stops the reader treating the survey as complete. +**The evidence** — a separate file, as long as it needs to be: graded findings, what was dropped, the sweep record, and every query issued verbatim. -Size it to drop into the spec section, ADR, ticket comment, or PR reply the user asked for — not a standalone essay. Hand it back; let them place it. +Keep them separate. One document that is both a verdict and an evidence file always becomes the evidence file, and nobody pastes a 4,000-word essay into a spec. ## Hard rules -1. **No unsourced claim about a named system.** Documented, inferred, or explicitly folklore. There is no fourth category, and "I know this" isn't one. -2. **Comparable by constraint, never by fame.** Every included system carries one line on why its problem is our problem. -3. **"No consensus", "they avoid it", and "couldn't verify" are results.** Manufacturing a clean majority out of thin evidence is the failure this skill exists to prevent. -4. **Search queries carry no client-identifying detail.** Generic phrasing, always. -5. **Read-only.** No edits, no posts, no implementation — the user places the output. -6. **End with a verdict and an unknowns list.** A survey without a recommendation hands the work back; a recommendation without unknowns oversells it. +1. **Every claim in the verdict is `opened` by you.** `secondhand` stays in the evidence file. +2. **Comparable by axis, never by fame.** Each system carries one line on where it sits and why that transfers. +3. **Budget the fan-out; dissent goes first.** Reserve a third of the search budget for step 5. +4. **Search queries carry no client-identifying detail** — and the leak comes from *inherited repo context*, not from what you type, so sanitize the sweep prompts explicitly. +5. **Distinguish "nothing found" from "couldn't look".** A tool failure reported as an absence is the worst output this skill can produce. +6. **"No consensus", "they avoid it", "couldn't verify" are results.** Manufacturing a majority out of thin evidence is the failure this skill exists to prevent. +7. **Read-only, and two artifacts.** The user places the output. From dd854fcb752c7ad0b2f8ab9550ef17dd766b5463 Mon Sep 17 00:00:00 2001 From: Jacek Tomaszewski Date: Tue, 25 Aug 2026 13:23:34 +0200 Subject: [PATCH 3/4] fix(prior-art): quote the source, and stop treating the grades as a ranking MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Graded 66 load-bearing citations from the five test reports by fetching every one: 55 supported, 9 partial, 2 unsupported, 3 dead — 83% strict, 97% counting partials. Per report, 100% down to 62%; the best run had every numeric default and limit exact to the digit. The important result is not the rate, it is that the labels did not predict it, and for the strongest label the correlation is inverted. Both unsupported claims and seven of the nine partials sat in the `documented` band. Every `inferred (I read the source)` claim that could be diffed verified exactly, and every hedge tested was honestly placed — including one report's hedged reading of a platform's source, which is correct, while its confidently-cited docs quote for the same rule does not exist on the page. The mechanism: citing a page is cheap and looks authoritative, reading source is expensive enough that nobody claims it falsely, and hedging costs the writer something so it is rarely false. Labels record how a claim was gathered, not whether it holds. So: - Every documented claim in the verdict now carries a VERBATIM QUOTE, not just a URL. You cannot quote a page that doesn't say it, and the reader checks it in one click. - State-bearing claims ("still open", "deprecated", "the default is") carry what was seen and when. Two claims the reports said they had personally verified were stale rather than invented — an issue described as "still open" had been closed for years — and staleness is much harder for a reader to spot than fabrication. - §6 says plainly that the grades are not a confidence ranking, and that reading the source outranks citing a doc page. Also confirms the fix in the previous commit was aimed correctly but not sufficient on its own: "opened by you" is itself a self-report, which is why it now has to produce a quote. No cross-product misattribution survived into any final text — the one surviving mix-up is docs-vs-source inside a single product. --- skills/prior-art/SKILL.md | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/skills/prior-art/SKILL.md b/skills/prior-art/SKILL.md index 74244f8..2b4d47e 100644 --- a/skills/prior-art/SKILL.md +++ b/skills/prior-art/SKILL.md @@ -53,15 +53,22 @@ This is the step that separates this skill from asking the question directly, an Budget for this before you spend on breadth. **A verified survey of three systems beats an unverified one of ten.** +Two mechanics make the check real rather than asserted: + +- **Quote, don't cite.** Every documented claim in the verdict carries a verbatim phrase from the page, not just a URL. You cannot quote a page that doesn't say it, and the reader can check it in one click. A citation without a quote is an assertion that you read something. +- **Date the state-bearing claims.** "Still open", "deprecated", "the default is", "no longer supported" — these were true once and rot silently. Record what you saw and when, or the claim's failure mode is being *stale* rather than wrong, which is far harder for a reader to spot. + ## 6. Grade on two axes Every claim carries both: -- **Strength** — `documented` (a specific page says it), `inferred` (you read the code/config and concluded it; say what you read), or `folklore` (widely repeated, no source found). +- **Strength** — `documented` (a specific page says it, quoted), `inferred` (you read the code/config and concluded it; say what you read), or `folklore` (widely repeated, no source found). - **Provenance of the check** — `opened` (you fetched it) or `secondhand` (a sweep reported it; nobody opened it). The second axis exists because the first is a self-report. A well-formatted wrong attribution passes the strength label cleanly. **`documented` + `secondhand` is a claim about a subagent's formatting, not about the world** — it may appear in the evidence, never in the verdict. +**Do not read the strength labels as a confidence ranking.** `inferred` from reading the source is in practice the *most* reliable grade, because reading code is expensive enough that nobody claims it falsely, and because source is what runs while docs drift. `documented` is the least reliable, because citing a page is cheap and confident-looking. Hedges are similarly trustworthy — a hedge costs the writer something, so it is rarely false. Ranking by grade rather than by what you checked inverts the real reliability order. + Folklore may be reported and may never justify a decision — "everyone says it and nobody documents it" is itself a finding about how well-founded the practice is. Record what you dropped and why; dropped claims are among the most useful output. Three results this format must be able to express, because an options table can't: @@ -85,7 +92,7 @@ Keep them separate. One document that is both a verdict and an evidence file alw ## Hard rules -1. **Every claim in the verdict is `opened` by you.** `secondhand` stays in the evidence file. +1. **Every claim in the verdict is `opened` by you, and every documented one is quoted.** `secondhand` stays in the evidence file. 2. **Comparable by axis, never by fame.** Each system carries one line on where it sits and why that transfers. 3. **Budget the fan-out; dissent goes first.** Reserve a third of the search budget for step 5. 4. **Search queries carry no client-identifying detail** — and the leak comes from *inherited repo context*, not from what you type, so sanitize the sweep prompts explicitly. From c38b1c3e31e05b9f2b2a98903132621abf7fbed5 Mon Sep 17 00:00:00 2001 From: Jacek Tomaszewski Date: Tue, 25 Aug 2026 13:29:07 +0200 Subject: [PATCH 4/4] feat(prior-art): weight sources by engineering maturity, and read the weak ones for failures MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit "Comparable by axis, not by fame" told you which systems' answers transfer, but said nothing about how much a given system's judgment is worth. Those are different questions and the skill conflated them. Maturity is the second weight, and it changes what you read a system FOR: - A team with a strong engineering record chose deliberately, so their choice is evidence about the design. - A legacy or poorly-built system's choice is evidence of nothing — but its failures are, and are often the best evidence available, because a mature system's bugs in the same area were found and quietly fixed years ago while a legacy one's are public, reproducible and still open. The test runs already showed this without naming it: the highest-value finding in the order-status run was two open bugs in the domain's legacy incumbent, where partial shipment plus partial refund silently strands fulfilable inventory. Under a fame-or-quality filter alone that source gets dropped; under this rule it is exactly where you go for failure data. Hence: read mature systems for what they chose, read legacy systems for what broke. A survey drawing only on the admirable ones carries no failure data at all. --- skills/prior-art/SKILL.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/skills/prior-art/SKILL.md b/skills/prior-art/SKILL.md index 2b4d47e..9f39654 100644 --- a/skills/prior-art/SKILL.md +++ b/skills/prior-art/SKILL.md @@ -31,6 +31,8 @@ That axis, not a constraint list, is what sorts the evidence. Every included sys **Fame is not the axis.** "Stripe does X" is evidence only if Stripe's problem is our problem. Where a famous system sits on the far side of the axis, keep it as an explicit **contrast**, never as support. +**Engineering maturity is a separate weight — and it changes what you read the system for, not whether you read it.** A team with a strong engineering record chose deliberately, so their choice is evidence about the design. A legacy or poorly-built system's choice is evidence of nothing — but its *failures* are, and they are often the best evidence available anywhere, because a mature system's bugs in this area were found and quietly fixed years ago while a legacy one's are public, reproducible, and frequently still open. **Read mature systems for what they chose; read legacy systems for what broke.** Neither is disqualified — they answer different questions, and a survey drawing only on the admirable ones has no failure data in it at all. + ## 4. Sweep by source class — with a budget Four classes, because each finds what the others structurally cannot: **primary docs** (including the "when not to use this" page, where the honest trade-off hides), **source and config** (what the code actually does — beats every blog post), **field reports** (postmortems, issue trackers, RFC threads — the highest-value genre, because they describe the pattern *failing*), and **the dissent** (who argues against it, and why).