knowledge: spec/mapping artifact checks — coverage vs value validity, per-check negative controls, GFM pipe parsing (2 insights) - #8
Open
dch0202-rsquare wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Knowledge flush — 1 page from 2 insight(s), 1 candidate deduped
Cross-Check: 독립 claude CLI(headless, --permission-mode plan) 적대검증 1회 —
.strip("|")가 마지막 셀의 escaped pipe를 먹는 실제 버그(GitHub/markdownAPI로 재현 확인), pitest·ESLint 인용 과대주장 2건을 지적받아 전부 수정 반영. 백슬래시 parity 반례 지적은 GitHub 실렌더로 기각됨.Queue drained: 3 pending rows. Two became one page (they share a trigger); one was
already flushed and is a duplicate re-emission — see "Existing-layer check".
Independent adversarial cross-check — 3 defects found and fixed
An independent headless
claudeinstance was asked to break this page's claims. Itfound one real code defect and two overstated citations; all three are fixed in this
PR, and I re-verified each myself rather than taking the review's word.
|a|b\\||gave['a', 'b\\']. Ground truth from GitHub's own renderer (POST /markdown,mode: gfm): the row is two cells,aandb|RuleTesterrequires nothing aboutvalid, which may be emptyinvalid-declares-errorsfact remainsOne challenge was rejected: the reviewer proposed that
(?<!\\)\|mishandles anescaped backslash before a real delimiter (
\\|). GitHub's renderer emits a singlemerged cell there, so the lookbehind matches real GFM behavior; the spec is silent on
backslash parity. No change made.
Verified best-practice
Insight A — coverage of a mapping table is not validity of its cells
Claim. When a check confirms a spec/mapping table covers every item of a source
list (row-per-rule, key-per-field, case-per-enum), pair it with a separate
value-validity check (each cell's value must exist in the canonical enum/schema/id
set), and give each check its own negative control that mutates only what that check
owns.
Sources checked.
split this rests on.
required: "An object instance is valid against this keywordif every item in the array is the name of a property in the instance."
enum: "Aninstance validates successfully against this keyword if its value is equal to one
of the elements in this keyword's array value." Presence and value are separate
keywords in the spec, which is exactly the coverage-vs-validity distinction.
automatically seeded into your code, then your tests are run. If your tests fail
then the mutation is killed, if your tests pass then the mutation lived." And on
why a green run over covered rows proves less than it appears to: line coverage
"measures only which code is executed by your tests. It does not check that
your tests are actually able to detect faults in the executed code."
RuleTester#run()takesvalidandinvalidcase arrays, and an invalid case must declareerrors; themust-pass/must-fail pairing is the established form for validating a checker.
How verified. The two load-bearing quotes (JSON Schema, pitest) were read from
the primary sources, not from memory. The per-check-negative-control refinement —
one mutation per check, requiring the owning check to redden — is the session's
own field evidence (mutating
| EntityDecl | Entity |toBogusand repointing agolden node id to
perf.nonexistentboth left the coverage checks PASS untilkind∈schema and id∈document checks were added).
Confidence: verified for the coverage/validity split and the seeded-fault
mechanic (official docs). The "exactly one check reddens per mutation" discipline is
field evidence layered on those sources; it is stated as a directive, not attributed
to a source it does not come from.
Not cited. The draft this flush inherited cited
testing.googleblog.com/2021/04/mutation-testing.html. That page's body could not beretrieved (the fetch returned only title/comments/sidebar), so it was removed
rather than carried as an unread citation;
pitest.orgsupplies the same mechanic intext I could actually read. Two further draft citations (ploeh red-green checklist,
lostechies "red for the right reason") were dropped with the scope change below.
Insight B — Markdown table rows split on unescaped pipes only
Claim. When validating or parsing Markdown table rows programmatically, split on
unescaped pipes —
re.split(r"(?<!\\)\|", row)— then unescape\|in each cell,before asserting cell counts.
Sources checked.
content by escaping it, including inside other inline spans"; "The header row must
match the delimiter row in the number of cells. If not, a table will not be
recognized"; and for body rows, "If there are a number of cells fewer than the
number of cells in the header row, empty cells are inserted. If there are greater,
the excess is ignored". The last quote corrected an edge-case claim: a fixed
body-row cell count is a repo convention, not a GFM requirement, and the page now
says so.
How verified. Reproduced locally (Python 3.9.6, macOS) on a 5-column row whose
third cell holds
create\|update\|delete:split("|")→ 7 cells,re.split(r"(?<!\\)\|", …)→ 5, matching the 5-cell header. Note the inheriteddraft asserted "6 cells naive vs 4 correct" — those figures did not reproduce (they
came from a differently-shaped row), so the page carries my measured 7-vs-5 with
the interpreter version stated.
Confidence: verified (GFM spec quote + local reproduction).
Existing-layer check
Routed via
INDEX.md→testing→ readwiki/testing/index.mdand everyquality/page whose "load when" overlaps.quality/tests-that-cannot-fail.mdrelated:added both waysquality/minimum-case-set.mdquality/behavior-not-implementation.mdqa/index.md+qa/process/*domain: qatesting; rationale belowDuplicate found — candidate retired, not ingested. The third pending row
(hash
f997639131d8ec9e, "validating a grep/regex gate whose target does not existyet") is already recorded in
~/.dev-loop/queue/.processed.jsonlwithstatus: flushed, and it is the subject of open PR #7(
wiki/testing/quality/checks-that-cannot-pass.md). The pending row is a secondemission of the identical hash from a later session. It is retired without a second
page.
Conflict flagged — PR #7 and this PR touch the same lines. Both append an id to
the same
related:line intests-that-cannot-fail.mdand add a row to the samequalitytable inwiki/testing/index.md. A textual merge conflict is expected onwhichever merges second; the resolution is to keep both ids and both rows.
Scope narrowed against PR #7. An earlier aborted run of this flush left an
untracked draft that folded all three candidates into one page, so its item 1 and one
"Instead of" row re-taught PR #7's subject (validate a gate against a known-good
sibling before adopting it), and it duplicated PR #7's grep-exit-status edge case.
Both were removed — that subject belongs to PR #7. No
related:link totesting-quality-checks-that-cannot-passwas added, because AGENTS.md invariant 4requires every
related:id to resolve and that page does not exist onmainwhilePR #7 is open. If PR #7 merges, adding the link both ways is a one-line follow-up.
Routing decision
testing/quality/spec-artifact-checks.md(new page, existing category)testing's remit perINDEX.md;qualityalready holds the "can this check actually detect a defect" family (tests-that-cannot-fail)Harvested
domain: qaoverridden for insight B.INDEX.mdscopesqatorelease-quality process (gates, regression scoping, bug reports, triage) and
testingto writing automated tests and checks. Parsing a table inside a checker ischeck-authoring code, so it routes to
testing.wiki/qa/was read to confirm nopage there owns doc parsing.
No new category.
testing/qualityfits; nothing was created at the domain orcategory level, so
INDEX.mdis unchanged.Files in this PR
wiki/testing/quality/spec-artifact-checks.md— 92 body lines (≤120), nobanned vague qualifiers, every anti-pattern paired with a replacement in
Instead of,idmatches path, unique across the wikiwiki/testing/index.md—qualitytable row with a "load when" line enumeratingall four distinct uses (mapping coverage, cross-document ids, verdict wording,
Markdown row parsing)
wiki/testing/quality/tests-that-cannot-fail.md— reverserelated:id only;last_verifieddeliberately left at 2026-07-10 since no claim on that page wasre-verified
log.md— appended## [2026-07-29] ingest | testing +1 (quality)entryInvariants checked mechanically:
related:ids resolve, reverse link present, pagelisted in its domain index, no unresolved inline links, body ≤120 lines, id unique.
Decision Log (AI 생성)
의도 — 무엇을 / 왜
testing/quality/spec-artifact-checks.md로 신설. 두 건이 트리거("spec/mapping 산출물을 검사하는 체크를 짠다")를 공유하므로 AGENTS.md의 "one case per page"를 지키면서 한 페이지가 맞다고 판단.flushed상태(열린 PR knowledge: validating a check whose target does not exist yet (1 insight) #7)인 동일 hash의 재방출이라 페이지를 만들지 않고 회수만 했다 — 같은 주제로 두 페이지가 생기는 것을 막기 위함.배제한 대안 — 무엇을 안 했나 / 왜
related:에testing-quality-checks-that-cannot-pass를 넣지 않음: AGENTS.md 불변식 4(모든 related id는 실재해야 함)를 위반하게 된다 — PR #7이 머지되기 전엔main에 그 페이지가 없다. PR knowledge: validating a check whose target does not exist yet (1 insight) #7 머지 후 양방향 링크 추가가 1줄 후속작업.qa로 라우팅하지 않음:INDEX.md가 qa를 릴리즈 프로세스로, testing을 테스트/체크 코드 작성으로 규정하므로 체커 코드는 testing.tests-that-cannot-fail.md의last_verified를 올리지 않음: 역방향 링크만 추가했고 그 페이지의 주장을 재검증한 게 아니라서.리뷰어가 볼 곳 — 신뢰성 판단 포인트
wiki/testing/quality/spec-artifact-checks.md:62-77— 파싱 스니펫. 적대검증에서 잡힌.strip("|")버그를 고친 자리이고, 게시된 스니펫 그대로 실행해|a|b\||→['a','b|']를 확인했다. GitHub/markdownAPI 렌더 결과가 정본.wiki/testing/quality/spec-artifact-checks.md:46-50— pitest 인용 경계. "어느 체크가 붉어지는가"는 pitest의 메커니즘이 아니라 이 페이지의 convention이라고 명시한 부분.wiki/testing/index.md+wiki/testing/quality/tests-that-cannot-fail.md:14— 열린 PR #7과 같은 줄/같은 표를 건드린다. 나중에 머지되는 쪽에서 텍스트 충돌이 예상되며, 해소는 "양쪽 id·양쪽 행 모두 유지".f997639131d8ec9e를 페이지 없이 회수한 판단이 맞는지(PR #7이 실제로 그 주제를 담고 있는지) 확인.