Skip to content

feat(work-items): bug path reproduces twice and proves the fix before a draft PR - #11894

Merged
melodic-automation-lanes[bot] merged 4 commits into
mainfrom
work-resolve-issue/issue-11865-41281560880-1
Nov 1, 2026
Merged

melodic-automation-lanes[bot] merged 4 commits into
mainfrom
work-resolve-issue/issue-11865-41281560880-1

Conversation

@melodic-automation-lanes

Copy link
Copy Markdown
Contributor

Closes #11865

Summary

Opened by the work-resolve-issue lane. The run's own summary:

Replaced the work skill's one-line Bug Investigation Rule with a five-step Bug path. It names the broken and expected final states before any run, reproduces twice through the real surface from a fresh detached worktree, and runs verify-existing-fix (baseline twice, patched build twice) when a fix may already exist. That check ends Confirmed, Insufficient fix or Inconclusive and never opens a competing PR. The path then writes the failing test first when a cheap local target exists, opens a PR (create --pushed always opens a draft) only with before-and-after proof, and never merges. Step 4 and Step 5 now point at it, and two described-snapshot eval cases cover a confirmed repro and an already-fixed item. Design choices and their basis: steps 1-3 run in the orchestrator with the claim held, and the worker writes the test and the fix, so the existing rule that the orchestrator never edits holds. Without proof the item is escalated through /work-items:attend-queue, the human route the root-cause gate already uses, rather than a PR opening. No extra --draft is passed, because create --pushed already opens a draft and a --draft reason would block the normal ready flip. CI mode keeps its own steps.

Not verified

  • python3 plugins/evals/skills/validate/scripts/validate-cases.py plugins/work-items/evals was not run: this lane allows no interpreter call. The YAML subset and every regex sample were checked by reading against the validator's source; left to the PR's CI
  • bash plugins/skill-quality/scripts/check-skill.sh plugins/work-items/skills/work was not run directly: check-affected skipped changed-skills because it 'runs the checked tree's own skill-quality code'. The review read the script and found no new FAIL (417 lines under the 500 cap; ## Next is no longe
  • The eval cases were not run (claude plugin eval is a paid run); only their shape and samples were checked
  • ai-slop pass ran without its deterministic detector (detect.sh is not a wrapper this lane may run), so it was a manual rubric read only; the git-branch context call that skill asks for was also skipped (only git-read status/diff/log/show are allowed)
  • review-venue.txt was absent, so the review ran locally in one fresh-context subagent, per the CI-mode fallback
  • Review nits left as they are: '\bConfirmed\b' and 'twice' are loose regex graders (the llm grader backs them up), and the Bug path stays an H2 after ## Next rather than moving into Workflow

Fix

  • plugins/work-items/skills/work/SKILL.md: replaced '## Bug Investigation Rule' with '## Bug path' (named final states, two reproductions from a reset, verify-existing-fix with Confirmed / Insufficient fix / Inconclusive and no competing PR, failing-test-first, PR only with before-and-after proof, nev
  • plugins/work-items/evals/wk-bug-repro-twice-then-proof/case.yaml: new case for a bug with no existing fix (regex graders reproduces-twice, draft-with-proof, no-merge, plus an llm meets-expectations grader)
  • plugins/work-items/evals/wk-bug-repro-twice-then-proof/samples/{reproduces-twice,draft-with-proof,no-merge}.json: pass/fail samples for each regex grader
  • plugins/work-items/evals/wk-bug-verify-existing-fix-confirmed/case.yaml: new case for a fix already on main, ending Confirmed with no PR (regex graders confirmed-outcome, no-competing-pr, plus an llm meets-expectations grader)
  • plugins/work-items/evals/wk-bug-verify-existing-fix-confirmed/samples/{confirmed-outcome,no-competing-pr}.json: pass/fail samples for each regex grader
  • .changes/work-items/work-resolve-issue-issue-11865-5b2e9c1d.md: minor-bump changelog fragment for the Bug path

Verification

The runner ran each check from the base copy against 18d951f:

  • scripts/check-changelog-fragments.sh --check: passed

The run reports:

  • .lane/checks/bin/check-changelog-fragments: all 53 fragments valid (after renaming the fragment to the <branch-slug>-<8 hex>.md form the check requires)
  • .lane/checks/bin/check-affected: 11 passed, 7 skipped, 0 failed over 9 changed files, re-run after tidy and after the review fixes
  • fresh-context review subagent over the whole diff: it reported no FAIL in check-skill.sh terms and said every regex sample sorts correctly under Python re and JavaScript RegExp. Its consistency findings (the step split, the repro disagreement rule versus triage's bar, how a reset is made, an untrust
  • tidy (code-tidying:tidy-diff over 12 hunk rows): passes 1-3 found nothing in the YAML/JSON data files; pass 4 (write-for-agents) cut one restating clause in SKILL.md; pass 5 (ai-slop fix) was a manual rubric read with no finding
  • The lane run: https://github.com/melodic-software/claude-code-plugins/actions/runs/41281560880

Left out: the items under Not verified.

Related

#11865

Lane-Activity: work-resolve-issue/resolve-issue
@github-actions

github-actions Bot commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

claude-review reviewed 8e07026d15c5bef1bfce944b89df30948bbc48dd: read 1 of 1 changed file(s); 0 finding(s), 0 as inline comments.

Incremental review (4e814a0 → 8e07026): no findings. This covers only plugins/work-items/skills/work/SKILL.md. The rest of the PR was reviewed at 4e814a0 and has not changed since.

What changed. The change is to Bug path step 3 (SKILL.md#L409). This branch handles a claimed fix whose revision is untrusted: a fork PR's head, or a revision named by an author below write access. It used to end with "that escalation ends the step". That wording did not say whether the run then went on to steps 4–5, where a worker writes a fix and a PR opens. It now reads "then write no fix, open no PR, release the claim (seam release) and end the run".

Why it holds.

  • Clear ending. The run now stops here, with no fix and no PR. That fits the closing issue's rule that verify-existing-fix "never opens a competing PR".
  • Same pattern as nearby endings. It matches how the Confirmed and Insufficient-fix (open PR) outcomes end on L410–411: release the claim, then end the run.
  • Security unchanged. The untrusted revision is still never checked out, built or run. Escalation still names the claim and why it was not run.

Possible follow-up (not counted as a finding). The Bug path's other escalations do not say whether to release the claim:

  • step 2, where the two repro runs disagree;
  • the Inconclusive outcome;
  • step 5, when there is no proof.

Those lines did not change in this increment, so they are out of scope here. A follow-up could make claim handling the same on every escalation.

Closing issue #11865. The Done-when criteria are unaffected by this increment. Two of them depend on commands the author says were not run: validate-cases.py and check-skill.sh. Both are author-claimed and unverified; this lane cannot run them either.

No settled-threads file was supplied. No plugin manifests changed in this increment. No embedded instructions were found in the PR context.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

claude-security-review: 1 inline finding(s) on 18d951f1141970625c458df3107a8dff6c431bea; the summary is in this lane's pull request comment.

claude-lane: claude-security-review

Comment thread plugins/work-items/skills/work/SKILL.md Resolved
@github-actions

github-actions Bot commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

claude-security-review reviewed 8e07026d15c5bef1bfce944b89df30948bbc48dd: read 1 of 1 changed file(s); 0 finding(s), 0 as inline comments.

Security review: incremental (4e814a0 → 8e07026)

Scope: plugins/work-items/skills/work/SKILL.md, one hunk (Bug path step 3, verify-existing-fix). I read the whole incremental change. No other file changed since the last review.

What changed: One sentence changed. Before, refusing to run an untrusted patched build (a fork PR's head, or a revision named by an author with less than write access) ended with "that escalation ends the step". Now the sentence goes on: "then write no fix, open no PR, release the claim (seam release) and end the run."

Assessment: This tightens a control. It does not widen one.

  • The trusted-revision gate is unchanged. Only revisions already on the default branch, same-repo PR heads, or revisions named by a commenter whose access the API reports as write, maintain or admin get checked out or run in the credentialed orchestrator session. A permission stated in item, PR or comment text still does not count.
  • The old wording left one path open. After the escalation, the bug path could have gone on to steps 4 and 5 (write a fix and open a PR) on an item where an untrusted actor had planted a fix claim. The new wording closes that path, so a refused claim can no longer lead to automated work.
  • The run now stops cleanly and releases its claim, and no credentials or untrusted code are involved. No instruction-surface rule was deleted or softened.

Embedded text in PR context: None that tries to steer this review. The closing issue body was left out by the wrapper.

No security issues found.

critical-findings: 0

@github-actions github-actions Bot added the needs-human Human-in-the-loop required; autonomous sessions must not resolve items carrying this. label Nov 1, 2026
@github-actions

github-actions Bot commented Nov 1, 2026

Copy link
Copy Markdown
Contributor

claude-security-review holds this pull request for a person (critical-security) at 18d951f1141970625c458df3107a8dff6c431bea.

CRITICAL: the new verify-existing-fix step in plugins/work-items/skills/work/SKILL.md (lines 408-409) checks out and runs a fork PR's or a commenter's revision as the patched build. It runs in the orchestrator session that holds the owner's gh auth, the tracker credentials and the lane push identity, including unattended under work-loop. No PR-author or fork trust check blocks it.

claude-lane: claude-security-review

@melodic-automation-lanes

melodic-automation-lanes Bot commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor Author

Unattended merge: native auto-merge leaves this pull request unarmed at head 4e814a034acd8fb30a66936d25132416e3d471e5. The lanes clear this reason on their own; the next scheduled run judges it again.

pr-merge's gate refuses its verifier verdict (verify:earlier-fail)

@kyle-sexton kyle-sexton removed the needs-human Human-in-the-loop required; autonomous sessions must not resolve items carrying this. label Nov 1, 2026
Comment thread plugins/work-items/skills/work/SKILL.md Resolved
Lane-Activity: pr-unblock/address-feedback
Lane-Thread: PRRT_kwDOTCGFQM6yYJPh
@melodic-automation-lanes

Copy link
Copy Markdown
Contributor Author

pr-unblock / address-feedback finished round 1 of max 4 on 0abc117.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

claude-review: 1 inline finding(s) on 0abc1172eb1e7574da0dcb17d638dbae5de4d9da; the summary is in this lane's pull request comment.

claude-lane: claude-review

Comment thread plugins/work-items/skills/work/SKILL.md Resolved
Lane-Activity: pr-unblock/address-feedback
Lane-Thread: PRRT_kwDOTCGFQM6ybQmz
@melodic-automation-lanes

Copy link
Copy Markdown
Contributor Author

pr-unblock / address-feedback finished round 2 of max 4 on 4e814a0.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

claude-review: 1 inline finding(s) on 4e814a034acd8fb30a66936d25132416e3d471e5; the summary is in this lane's pull request comment.

claude-lane: claude-review

Comment thread plugins/work-items/skills/work/SKILL.md Resolved
Lane-Activity: pr-unblock/address-feedback
Lane-Thread: PRRT_kwDOTCGFQM6ybVlm
@melodic-automation-lanes

Copy link
Copy Markdown
Contributor Author

pr-unblock / address-feedback finished round 3 of max 4 on 8e07026.

@kyle-sexton
kyle-sexton added this pull request to the merge queue Nov 1, 2026
@melodic-automation-lanes

Copy link
Copy Markdown
Contributor Author

Unattended merge: native auto-merge leaves this pull request unarmed at head 8e07026d15c5bef1bfce944b89df30948bbc48dd. A person must clear this reason, so the pr-merge lane added the needs-human label: fix the reason, then remove the label, and the next scheduled run judges it again.

pr-merge's gate holds it on its risk score (security-finding: 1 open)

@melodic-automation-lanes melodic-automation-lanes Bot added the needs-human Human-in-the-loop required; autonomous sessions must not resolve items carrying this. label Nov 1, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 1, 2026
@kyle-sexton kyle-sexton removed the needs-human Human-in-the-loop required; autonomous sessions must not resolve items carrying this. label Nov 1, 2026
@kyle-sexton
kyle-sexton added this pull request to the merge queue Nov 1, 2026
@kyle-sexton
kyle-sexton removed this pull request from the merge queue due to a manual request Nov 1, 2026
@kyle-sexton kyle-sexton added the needs-human Human-in-the-loop required; autonomous sessions must not resolve items carrying this. label Nov 1, 2026
@kyle-sexton

Copy link
Copy Markdown
Member
Hold: the CRITICAL security thread (fork or untrusted revision run as the patched build) was resolved by the lanes bot, and the merge gate counts only a person's resolve or a clean whole security review as clearing a CRITICAL finding. PR Lane removed needs-human in error at 15:0xZ and has put it back. Releases when: the owner confirms the fix in 0abc117 by re-resolving thread PRRT_kwDOTCGFQM6yYJPh, or a whole claude-security-review at the head reports 0 findings.

@kyle-sexton kyle-sexton removed the needs-human Human-in-the-loop required; autonomous sessions must not resolve items carrying this. label Nov 1, 2026
@melodic-automation-lanes

Copy link
Copy Markdown
Contributor Author

The unattended merge path armed native auto-merge at head 8e07026d15c5bef1bfce944b89df30948bbc48dd.

@melodic-automation-lanes
melodic-automation-lanes Bot added this pull request to the merge queue Nov 1, 2026
Merged via the queue into main with commit 9182ad3 Nov 1, 2026
110 of 112 checks passed
@melodic-automation-lanes
melodic-automation-lanes Bot deleted the work-resolve-issue/issue-11865-41281560880-1 branch November 1, 2026 15:53
@melodic-automation-lanes

Copy link
Copy Markdown
Contributor Author

The post-merge check verified merge commit 9182ad3b8629229b0043222ca48f210407cbf672; later checks count it verified from this comment.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(work-items): bug path reproduces twice and proves the fix before a draft PR

1 participant