What problem are you trying to solve?
I went through the repo's CI-related content, and the evaluation side is
actually in good shape: software-fluency scores hosted CI L1–L5,
harness-engineering §8 checks CI/local consistency, and git-hooks.md has a CI
And Merge Stage. But all of it answers "does this project's CI exist, is it
strong, where in the lifecycle should it sit". None of it answers what an
agent should do with an actual verdict.
What I see missing:
- Red-verdict attribution. The failure-repair chain in agent-work-loop
assumes the check is locally runnable and reproducible; hosted CI usually
isn't. I've watched agents burn rounds "fixing" their own diff for
failures that were really infra (hitting other PRs too) or upstream
test-corpus drift (the consumed repo deleted a failing case an hour
earlier).
- Fake-green defense. Nothing warns about green runs that executed zero
cases: silent fallback when a remote fetch fails, scripts dying quietly
under set -e, 0/0 passing.
What would success look like?
One short references doc: attribute the verdict before touching code (infra /
upstream asset drift / local), plus a few evidence-integrity guards
(empty-run must fail, fail-fast over silent fallback). I already sketched it
following the references contract — link-graph check passes — and can send a
PR.
Where would this be used?
references/project-harness/, guidance only, host-agnostic.
Examples or evidence
All from real episodes on OSS and internal CI; happy to provide redacted
write-ups.
Anything else?
The ownership call is genuinely open: separate doc, or extend
agent-verify-loop's "After a Non-Passing Verdict" (I lean separate because
that section assumes locally runnable checks). Whether the guards deserve
their own doc, whether a companion case-study is worth it — also open. This
is my reading of the repo, so if I missed existing coverage or got the
framing wrong, please correct me; happy to discuss.
What problem are you trying to solve?
I went through the repo's CI-related content, and the evaluation side is
actually in good shape: software-fluency scores hosted CI L1–L5,
harness-engineering §8 checks CI/local consistency, and git-hooks.md has a CI
And Merge Stage. But all of it answers "does this project's CI exist, is it
strong, where in the lifecycle should it sit". None of it answers what an
agent should do with an actual verdict.
What I see missing:
assumes the check is locally runnable and reproducible; hosted CI usually
isn't. I've watched agents burn rounds "fixing" their own diff for
failures that were really infra (hitting other PRs too) or upstream
test-corpus drift (the consumed repo deleted a failing case an hour
earlier).
cases: silent fallback when a remote fetch fails, scripts dying quietly
under
set -e, 0/0 passing.What would success look like?
One short references doc: attribute the verdict before touching code (infra /
upstream asset drift / local), plus a few evidence-integrity guards
(empty-run must fail, fail-fast over silent fallback). I already sketched it
following the references contract — link-graph check passes — and can send a
PR.
Where would this be used?
references/project-harness/, guidance only, host-agnostic.Examples or evidence
All from real episodes on OSS and internal CI; happy to provide redacted
write-ups.
Anything else?
The ownership call is genuinely open: separate doc, or extend
agent-verify-loop's "After a Non-Passing Verdict" (I lean separate because
that section assumes locally runnable checks). Whether the guards deserve
their own doc, whether a companion case-study is worth it — also open. This
is my reading of the repo, so if I missed existing coverage or got the
framing wrong, please correct me; happy to discuss.