From the stumble batch of 2026-08-26 (3 blind agents, browser only, no repo
access, landing page as the sole source; median 13 steps, all three completed
the task). Three of their gripes were fixed in the same pass — the --budget /
--tau flags now appear as real command rows, a stage click no longer loses to
the autoplay timers, and the hero counters ship their value in the markup
instead of animating up from a permanent "0". Two survive, both reproduced
across runs, and both are content decisions rather than bugs.
1. Nothing on the site is a flag reference
"There's no docs/CLI-reference page in the nav (only pipeline/graph/scoring/
install) — a first-time user has to reverse-engineer the full flag set from
five example commands rather than a real reference."
Adding --budget and --tau answered the one question that blocked the task,
but the shape of the complaint is unchanged: the site teaches by example and
example only. Every agent that "completed" the task did so by guessing a
plausible command (all three produced slightly different ones, and two hedged
with "possibly").
2. The only quantitative evidence is a toy repo baked into the page
"The only quantitative 'it works' evidence (7× denser signal, 48,210 vs 6,930
tokens) comes from a fixed 18-fragment toy repo baked into the page, not any
real-world or benchmark PR."
"no real-world case study or example at PR scale (the only live demo is a
fixed 18-fragment toy repo) — nothing to gauge how it behaves on something
like a 40-file PR."
This one stings because the numbers exist. The paper and the frozen sweep carry
per-instance results over real commits; the page cites the DOI but never lifts a
single measured figure out of it. A reader deciding whether to trust the output
over a raw diff is handed a hand-made scenario and a link to a PDF.
Why this is on the product queue rather than fixed here
Both are editorial calls about what the landing page is for — a mechanism
explainer (what it is today, and it is very good at that: "the 5-stage clickable
pipeline made the tool's mechanism genuinely understandable in under a minute")
versus an adoption page that also has to answer "at my scale, with which flags".
Picking is not a QA decision.
Close condition
The page carries (a) a reachable full flag reference, and (b) at least one
number measured on a real repository, with its denominator stated. Re-run
stumble task 1 from QA.md; the two gripe slugs above must not reproduce.
From the stumble batch of 2026-08-26 (3 blind agents, browser only, no repo
access, landing page as the sole source; median 13 steps, all three completed
the task). Three of their gripes were fixed in the same pass — the
--budget/--tauflags now appear as real command rows, a stage click no longer loses tothe autoplay timers, and the hero counters ship their value in the markup
instead of animating up from a permanent "0". Two survive, both reproduced
across runs, and both are content decisions rather than bugs.
1. Nothing on the site is a flag reference
Adding
--budgetand--tauanswered the one question that blocked the task,but the shape of the complaint is unchanged: the site teaches by example and
example only. Every agent that "completed" the task did so by guessing a
plausible command (all three produced slightly different ones, and two hedged
with "possibly").
2. The only quantitative evidence is a toy repo baked into the page
This one stings because the numbers exist. The paper and the frozen sweep carry
per-instance results over real commits; the page cites the DOI but never lifts a
single measured figure out of it. A reader deciding whether to trust the output
over a raw diff is handed a hand-made scenario and a link to a PDF.
Why this is on the product queue rather than fixed here
Both are editorial calls about what the landing page is for — a mechanism
explainer (what it is today, and it is very good at that: "the 5-stage clickable
pipeline made the tool's mechanism genuinely understandable in under a minute")
versus an adoption page that also has to answer "at my scale, with which flags".
Picking is not a QA decision.
Close condition
The page carries (a) a reachable full flag reference, and (b) at least one
number measured on a real repository, with its denominator stated. Re-run
stumble task 1 from
QA.md; the two gripe slugs above must not reproduce.