Question Tuning preamble points at a relative `scripts/question-registry.ts` that does not resolve from a project cwd, so agents silently fabricate question_ids (21/21 unregistered in one session)

[alexvibecommit]

Summary

The Question Tuning preamble tells the agent to look up question_id at a relative path that does not resolve from a user's project directory, and offers "make one up" as the alternative in the same sentence. The result is that agents silently fabricate question_ids instead of using the registry, and every fabricated id is a singleton that no per-question preference can ever attach to.

scripts/resolvers/question-tuning.ts:26:

Before each AskUserQuestion, choose `question_id` from `scripts/question-registry.ts` or
`{skill}-{slug}`, then run `${bin}/gstack-question-preference --check "<id>"`.

Note the asymmetry inside that one sentence: the bin path is interpolated absolute (${bin} → ~/.claude/skills/gstack/bin), the registry path is hardcoded relative. Agents run with cwd set to the user's project, not the gstack install:

$ pwd
/Users/<me>/Documents/<my-project>
$ ls scripts/question-registry.ts
ls: scripts/question-registry.ts: No such file or directory

The file exists only at ~/.claude/skills/gstack/scripts/question-registry.ts. The lookup fails, the documented or {skill}-{slug} fallback succeeds, and the agent proceeds with an invented id and no error anywhere.

This line is emitted into every generated SKILL.md — I count it in ~40 of them (plan-tune, autoplan, cso, plan-eng-review, design-*, ios-qa, learn, pair-agent, …), so it is not skill-specific.

Observed impact

One /plan-eng-review session on a real plan emitted 21 question_ids, 0 of them registered:

plan-eng-review-arch-1-delta-lineage        plan-eng-review-quality-1-hook-durability
plan-eng-review-arch-2-parity-gate          plan-eng-review-perf-1-cumulative-rebuild
plan-eng-review-ov-3-subagent-transcripts   plan-eng-review-tests-2-client-tier
plan-eng-review-scope-challenge             plan-eng-review-todos-batch            …

Every one was generated from the agent's own section headings. The registry has five perfectly good plan-eng-review-* ids (arch-finding, scope-reduce, test-gap, outside-voice, todo-add) that these map onto cleanly — arch-1-delta-lineage and arch-2-parity-gate are both plan-eng-review-arch-finding, tests-2-client-tier is plan-eng-review-test-gap, and so on.

The one id that was logged in an earlier session, plan-eng-review-step0-scope, is also unregistered — the registry name is plan-eng-review-scope-reduce.

Why this defeats the feature rather than degrading it

Per-question preferences are keyed on question_id. Tuning requires the same id to recur enough times to establish a pattern. With heading-derived ids:

• Each id fires once, ever. The next session produces 21 different ids from different headings.
• No count per id ever exceeds 1, so no threshold — the user's, mine, or any future automated one — is ever reachable.
• door_type is unknown for every fabricated id, so the safety classification falls back to keyword-matching prose (scripts/one-way-doors.ts), which that file's own header says is too weak to be the primary gate.
• /plan-tune's history and stats views show 21 unrelated one-off rows instead of 5 ids with counts.

This is the same failure as the hook-<hash> ids that the skill's "Audit unmarked questions" section exists to surface, except louder: a heading-derived id looks legitimate and registered. hook-<hash> at least announces itself.

Suggested fix

1. Interpolate the registry path the same way the bin path already is. One line, one file, propagates to every skill via bun run gen:skill-docs:

   -choose `question_id` from `scripts/question-registry.ts` or `{skill}-{slug}`
   +choose `question_id` from `${gstackRoot}/scripts/question-registry.ts` (read it — do not
   +guess); only if the question genuinely has no registry entry, fall back to `{skill}-{slug}`

2. Demote the fallback. As written, A or B reads as two equal options and B is the one that always works. Making the fallback explicitly conditional on a failed lookup is most of the fix on its own.

3. Flag marked-but-unregistered in the audit path. /plan-tune's "Audit unmarked questions" currently greps for hook--prefixed ids. It could equally check emitted ids against getAllRegisteredIds() (already exported from scripts/question-registry.ts:623) and surface the unregistered-but-marked ones. That catches this class at inspection time even when the instruction is misread, and it is the louder version of the problem the section already targets.

4. Optionally, have bin/gstack-question-log stamp a registered: true|false field per event so the fragmentation is queryable after the fact rather than needing a transcript audit.

Fix (1) is the one that stops new bad data. (3) is what would have caught this without a hand audit.

[alexvibecommit]

Worth noting for prioritization: this one gates #2490, and it also gates the usefulness of the fix in #2336.

Once answers are parsed correctly (#2336), the log still cannot support per-question tuning while every id is a heading-derived singleton — you get accurate answers attached to ids that occur exactly once each. The path fix here is what makes the other two worth having.