Skip to content

fork-ci: test list reads one line per test; doc-check fails closed (#789, #908) - #942

Merged
scuttletoby merged 4 commits into
mainfrom
fork-ci/tests-formats-and-doc-check
Nov 22, 2026
Merged

scuttletoby merged 4 commits into
mainfrom
fork-ci/tests-formats-and-doc-check

Conversation

@scuttletoby

@scuttletoby scuttletoby commented Nov 22, 2026 •

Copy link
Copy Markdown
Owner

What and why

Two small sets of follow-ups from earlier lens reviews. Both fail closed where they used to pass quietly.

Fork test list (#789, from the #784 lenses)

  1. A test the runner never runs. stage-preview-bundle.test.py is run by a plain python3 step, so the runner never reads its unittest word. The list header no longer claims the word judges it, and a comment sits beside the line. New test in fork-ci-tests.test.cjs: every listed Python test with no expected record (expectedKeys) must have the word its source shows (unittest.main/TestCase or an all passed end line). Today that is only this file.
  2. An ok-lines test counts itself. This is part of the format and is not changed. The header now says it plainly: the count is trusted, and a test that prints a matching end line and then stops early passes, so a new Python test is best a unittest test.
  3. A file listed twice is refused by readList. Before, a part used the last line and --run used the first.
  4. The "cannot read" error now reads: ...; a Python test's line then ends in its format: "unittest" or "ok lines").
  5. The fork CI: each listed Python test declares its format (unittest or ok lines), judged by it alone (#702) #784 body sentence. fork-ci-result never reads a record's format; it judges status, counts, skips and keys only. This is a correction for readers, with no code change.

doc-check (#908, from the #874 lens)

  1. Symlinks. A file turned into a symlink was dropped twice: its change shows as type T, which the diff filter skipped, and a symlink was then only a warning. Type changes are now checked, and a symlink is checked as the file it points to: its size, history rules and links. A symlink that points out of the repository, into .git or to no file fails. With --base, a change that deletes, renames or moves away the file an untouched symlink points to fails as this change removed <path>. A symlink's warning says when the change moved it past warn_at (the base is read through the symlink). When a symlink's target is checked in the same run, what the target already reports is said once, at the target.
  2. Not UTF-8. A file that is not UTF-8, or that cannot be read, now fails, where before it got only a warning. Its size is still checked, because the bytes are kept with surrogateescape.
  3. Deleted or renamed docs. With --base, if the change deletes or renames a file (or empties a folder) that a selected file it did not touch still links to, that link fails as this change removed <path>. Broken links that were already on the base in untouched files are not blamed on the change; mode: all still finds those.
  4. CRLF. Sizes are the bytes on disk. Before, text mode dropped one byte per line. Line counts and the history and link rules still read text as Python's text mode does.
  • Nits:
    • Links inside inline code, or in a fence inside a list item, are no longer reported. Fences follow CommonMark: a fence opens with 0-3 spaces of indent, counted from the content of the list item it is in. A line indented 4 or more spaces (or a tab) is no fence, so it no longer hides the links after it, as on main. A fence in a list item ends with the item.
    • A link that is no path (%00, for example) is a broken link, not a traceback.
    • A ?query is cut like an #anchor.
    • The README and docstring now say that "cannot loosen" covers the config only. The calling workflow file comes from the pull request, so only review guards it.

On this repository, --all changes in three ways. Three false "broken link" reports go away: all three were links inside inline code. The vendored .repos/ SKILL.md files that are symlinks are now checked as their targets. Their targets are also checked, so an over-budget target is reported once, at the target; --all already failed there. No new problem appears outside .repos/.

Fix round (lens reports on e74b63f)

  • Lint (blocker): the issue number is gone from the two test comments (no-low-value-comments). Lint on fork-ci-tests.test.cjs fails at e74b63f with those 2 errors and passes now.
  • Untouched symlink loses its target (major): delete, rename, and the target's folder turned into a symlink out of the repository. Each fails in --base mode. A symlink already dangling on the base is not blamed.
  • Fence indent (major): CommonMark, as above.
  • Minors: the symlink warning wording and once-only report, .git targets, %00 links, and the action.yml description (it now lists the UTF-8 and symlink rules).
  • Shown failing without the fix: the new tests were run against e74b63f's doc_check.py. All 11 new code-fix tests fail: 3 for the symlink target, 1 fence test (5 sub-cases), .git, the warning, --all once, and %00 (a traceback). The 12th, the dangling-on-base guard, passes on both.
  • --all on this repository: 461 problems and 2 warnings before this round, 455 and 1 after. The difference is only the 6 symlinked .repos/ SKILL.md files and 1 warning, now said once at their targets. The fence change adds or removes no link report here.

Tests that fail without each fix (run locally)

  • doc-check: the new tests run against main's doc_check.py. Six of seven fail: the removed-doc test, the symlink to a large file, the symlink out of the repository or to no file, not UTF-8, CRLF, and inline code with a query and a list fence. The seventh, "an old broken link is not this change", is a guard against blaming too much, so it passes on both.
  • fork-ci-tests:
    • With main's fork-ci-tests.cjs, the list test fails on the error wording.
    • With only the duplicate refusal removed, it fails with Missing expected exception: a.test.sh.
    • With stage-preview-bundle's word changed to ok lines, the new source test fails (its source is unittest).

Checks run

Each run was gated by capacity and run through the build-slot wrapper.

Check Result
doc_check.test.py 39 OK
fork-ci-tests.test.cjs 69/69
fork-contracts.test.py 716 ok, 0 skipped (after merging main; no test list changed in the fix round)
Seam check (--config-ref origin/main origin/main HEAD) 0
vp fmt --check (README, action.yml, both .cjs files) clean
vp lint (fork-ci-tests.test.cjs) 0 errors (red at e74b63f)

State question

  • Runs again: both tools are read-only and stateless, so a re-run gives the same output. The new not-UTF-8 test checks this by running twice. The fix round adds only reads (git ls-tree and cat-file at the merge base, readlink).
  • Stops half-way:
    • doc-check writes nothing, and exit 0 only comes after every file and the removed-link scan. A failure in the scan's git ls-files exits 2 and is never a pass.
    • readList throws before any test runs or any record is written.
  • Old, empty or missing state:
    • doc-check with no deletions does not scan the tree. With links: false, the scan checks only untouched symlinks, not links. An unreadable file in the scan reads as empty.
    • A symlink already dangling on the base is not blamed on the change. If a base path cannot be read through its symlinks, the warning just leaves out "this change moved it past".
    • A list that had a duplicate line now fails loudly in every job that reads it; main's list has none.
    • Records from before this change are not read again.
  • Strictness: this can newly fail an existing pull request in these cases, which is the intent of doc-check: four cases where it passes when it should fail #908:
    • a doc-check user whose instruction file is a symlink (out of the repository, into .git or to no file) or not UTF-8;
    • a CRLF instruction file near its budget, which can now fail when touched, because sizes count the CR bytes;
    • a pull request that removes a doc that is still linked, or the target of a symlink;
    • a link after a line indented 4+ spaces that starts with a fence mark. Main checks that link too.

General vs project

  • General: doc-check is a general action. All of its changes, including the symlink, UTF-8, CRLF and removed-doc rules, apply to any repository that uses it; nothing project-specific was added. The test-list rules are fork CI tooling that works the same for any list.
  • Project: none. No config file or project fact changed.

Fixes #789
Fixes #908

🤖 Generated with Claude Code

scuttletoby and others added 2 commits November 22, 2026 10:46
, #908)

fork-ci-tests (#789):
- readList refuses a file listed twice (a part and --run could read different lines).
- The "cannot read" error reads plainly.
- The list says a test the runner does not run (stage-preview-bundle, a plain
  python3 step) is not judged by its format word; fork-ci-tests.test.cjs now
  checks that word against the test's source.
- The header says an ok-lines test's own count is trusted.

doc-check (#908):
- A symlink is checked as the file it points to (a change of type is a change);
  one out of the repository or to no file fails.
- A file that is not UTF-8 or cannot be read fails, and its size is still checked.
- Sizes are the bytes on disk, so CRLF counts.
- With --base, a doc the change deletes or renames that an untouched file still
  links to fails.
- Links in inline code, in a fence inside a list item, or with a ?query are no
  longer reported broken; the README says the workflow file is guarded by review
  only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@scuttletoby

Copy link
Copy Markdown
Owner Author

lens: state · reviewer: lens agent (Opus) · read at: e74b63f
verdict: safe to merge
covered: the whole diff: doc_check.py (read_doc, links_to_removed, changed_files with T and D, lf/usage, FENCE and INLINE_CODE), its README and tests; fork-ci-tests.cjs readList, the list header and the new source-format test.
checks: ran doc_check.test.py (30 OK) and fork-ci-tests.test.cjs (69/69). Mutations, each turned one test red: diff filter back to AMD (the symlink test), not-UTF-8 back to no problem (the UTF-8 test), no emptied-folder clause in links_to_removed (the rename/delete test). In a scratch repo, one change that deletes a linked doc, renames a file, adds a non-UTF-8 file, a symlink out of the repository and a dangling symlink gave 4 problems and exit 1, the same on a second run; deleting a whole linked folder gave 3 "this change removed" problems; --base deadbeef exits 2.
state: runs again: the same output (no writes; the new UTF-8 test checks it). Stops half-way: nothing is written, exit 0 only after every file and the removed-link scan; a failing git ls-files in the scan exits 2. Old or empty state: no deletions or links: false means no scan; a repo with no selected docs scans nothing and passes; a dangling or outside symlink fails; a non-UTF-8 file fails and keeps its size check; a base that is not a commit or has no merge base exits 2; a base blob that cannot be shown reads as a new file, which only changes a warning's wording; a link that was already broken on the base is not blamed. The test list is read only at the checked-out commit, so an old list is never re-read; main's list has no duplicate line.
blocker: none
major: none
minor: FENCE now opens at any indent (doc_check.py:56). A 4-space indented code block, or a tab-indented line, that shows an unbalanced fence line now hides every later link in the file; main read that line as text. One fix: allow more than 3 spaces only when the line before is a list item or a fence was opened in one.
minor: a link with %00 in a selected file that the change did not touch crashes the removed-link scan (os.path.realpath raises ValueError in broken_link, doc_check.py:225) whenever a change deletes any file. It exits non-zero, so it fails closed, but with a traceback and on every deleting change. The same crash was already possible for touched files.
minor: the PR body's "Strictness" list leaves out CRLF: a CRLF instruction file near its budget can newly fail when touched.
minor: action.yml's description still lists only budgets, history, links and reports, not the UTF-8 and symlink rules.

@scuttletoby

Copy link
Copy Markdown
Owner Author

lens: merge-path · reviewer: lens agent · read at: e74b63f
verdict: not safe to merge
covered: the PR body and the full diff (fork-ci-tests.txt, fork-ci-tests.cjs and its test, doc_check.py, its test and README); every caller of doc-check in this repository and in the SkyRPG repository; the test list of every open PR that touches it
checks: doc_check.test.py 30 OK and fork-ci-tests.test.cjs 69/69 at the head; the head merged with current main (45fcf05) reads 125 list entries and passes 69/69; --all with this head against main's doc_check.py on this tree; edge cases in a small test repo; the failing Lint and fork-ci-result logs of this head
state: runs again: both tools are read-only, so a re-run gives the same result; stops half-way: doc-check exits 0 only after every file and the removed-link scan, a failed git ls-files exits 2, and readList throws before anything runs; old state: no open PR's list has a duplicate line (#40, #43, #525 checked), so the new refusal fails nothing open; on main, --all changes only as the body says (6 symlinked .repos/ SKILL.md files over budget, 3 inline-code reports gone), and no workflow here runs doc-check outside its unit tests, so nothing on main is newly refused; SkyRPG runs its own scripts/skyrpg-dev/doc_check.py, not this action, so it is not affected
blocker: Lint fails at this head, and fork-ci-result with it: t3code(no-low-value-comments) refuses the issue number in two new comments, .github/scripts/fork-ci-tests.test.cjs:276 and :290. Dropping "(#789)" from both fixes it.
major: fork-ci/actions/doc-check/doc_check.py:304 (links_to_removed) and :366: deleting or renaming the target of a symlink the change does not touch passes. Base has the file AGENTS.md and CLAUDE.md -> AGENTS.md; the change deletes AGENTS.md; --base prints "0 file(s), 0 problem(s)" and exits 0, although CLAUDE.md now points to no file, which the README says fails. The same happens when the target's folder is replaced by a symlink out of the repository. The scan reads only paths where os.path.isfile is true and ignores read_doc's problem. Fix: in the scan, fail a selected tracked symlink whose target is in gone or under a removed folder, or that read_doc now refuses, as "this change removed "; add a test that fails without the fix.
major: fork-ci/actions/doc-check/doc_check.py:56: FENCE now matches any indent, so a line indented 4 or more spaces that starts with three backticks but is no fence opener (an indented code block that shows a fence, for example) opens a fence that never closes, and every link after it is skipped. Main catches this and the head does not: README.md with an indented fence line and then [gone](nope.md) exits 1 on main's doc_check.py and 0 on the head. Fix, failing closed: when a fence never closes by the end of the file, check the links on the lines after it as well (this also covers an unclosed fence at 0-3 spaces); add a test.
minor: doc_check.py:376: a symlink gets no base_text, so its WARN never says that this change moved it past warn_at, and in --all a symlink to an over-budget file is reported twice, once for the target and once for the link.
minor: doc_check.py:285 (read_doc): a symlink to a file under .git/ is "in the repository" by realpath and is now read and checked. Today only link targets and counts are printed, and Actions masks the checkout token, so nothing leaks. Refusing targets under .git/ would close this for good.

- A change that deletes, renames or moves away the file an untouched symlink
  points to fails in --base mode ("this change removed <path>").
- A fence opens only with 0-3 spaces of indent, counted from the content of
  the list item it is in; 4+ spaces or a tab is no fence, so later links are
  still checked. A fence in a list item ends with the item.
- A symlink's warning says when the change moved it past warn_at (the base is
  read through the symlink); a symlink whose target is checked in the same
  run reports what the target already reports once.
- A symlink into .git fails; a link that is no path (%00) is a broken link,
  not a traceback.
- action.yml lists the UTF-8 and symlink rules.
- Drop the issue number from two test comments (no-low-value-comments).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@scuttletoby

Copy link
Copy Markdown
Owner Author

lens: merge-path r2 · reviewer: lens agent (Opus) · read at: b02479e
verdict: safe to merge
covered: the PR body, both round-1 lens reports, the fix commit b02479e and the full diff against main: fork-ci-tests.txt, fork-ci-tests.cjs and its test, and doc_check.py with its test, README and action.yml
checks: doc_check.test.py 39 OK and fork-ci-tests.test.cjs 69/69 at the head. Lint passes at the head in CI. In scratch repos I ran this head's and main's doc_check.py on: an untouched symlink whose target is deleted, renamed, or whose folder becomes a symlink out of the repository; a symlink already dangling on the base; symlinks into .git (from the root and from a subfolder); %00 links in touched and untouched files while the change deletes a file; the warning wording for a symlink. I ran 13 fence and inline-code cases through link_lines on both versions. I did not re-run the head merged with main: capacity was not OK, and main is still at 45fcf05, which round 1 checked; the fix commit changes only two comments in the cjs test.
state: runs again: both tools are read-only, so a second run gives the same output. Stops half-way: doc-check exits 0 only after every file and the removed-link scan; a failing git ls-files exits 2; readList throws before anything runs. Old, empty or existing state: a symlink already dangling on the base is not blamed; no deletion means no scan; no workflow in this repository runs doc-check outside its unit tests, so nothing on main is newly refused; no open list has a duplicate line.
round 1, fixed: Lint. "(#789)" is gone from both comments in fork-ci-tests.test.cjs, and the Lint check passes at this head.
round 1, fixed: an untouched symlink whose target is deleted, renamed, or whose folder is replaced by a symlink out of the repository. Each case fails in --base with "this change removed docs/AGENTS.md" (or "AGENTS.md"). A symlink already dangling on the base still passes. Tested at the head and by the three new tests.
round 1, fixed: fence indent. A 4-space or tab-indented fence-like line no longer hides the links after it. A fence in a list item counts its indent from the item's content and ends with the item. A "x" line is no fence. I compared 13 cases with main: none is a new false pass from the fence change, and 3 links that main hid are now reported.
round 1, fixed: a symlink into .git is refused, from the root and from a subfolder, and nothing from the file is printed.
round 1, fixed: a %00 link is reported as "not a path (ValueError)" in a touched file. In an untouched file during a deleting change it is no traceback and is not blamed.
round 1, fixed: the symlink WARN says "this change moved it past 90%" when the change did, read through the old target on the base. In --all, what a target already reports is said once, at the target.
blocker: none
major: none
minor: doc_check.py:349-390 (removed_target and links_to_removed): the scan matches removed paths by spelling, and it runs only when the change deletes a file. Three cases still pass that leave an untouched symlink or link pointing at no file. (1) A folder symlink, such as docs -> real, is retargeted out of the repository or to nothing: the diff has only an M on docs, so nothing is scanned. (2) real/AGENTS.md is deleted while CLAUDE.md -> docs/AGENTS.md goes through docs -> real. (3) The same as (2) for a link [a](docs/AGENTS.md). All three pass on main as well, so none is a regression. This repository has one folder symlink (.claude/skills), but no doc links through it. Fix for a follow-up: in --base, re-read every selected tracked symlink and every link through its resolved path, and blame any that read_doc or broken_link refuses at the head but that resolved at the base.
minor: doc_check.py:431: in --base, an untouched symlink is never checked against its own budget. Example: CLAUDE.md -> AGENTS.md, and the change grows AGENTS.md to 61 lines. --base passes: AGENTS.md is inside its 150-line budget. --all fails CLAUDE.md at 61 of 40 lines. Main passes this too, so it is no regression, but the README says a symlink is checked as the file it points to. Fix for a follow-up: also check the selected tracked symlinks whose resolved target is a changed file.
minor: doc_check.py:59: INLINE_CODE can start in the middle of a backtick run, so an unmatched run hides a real link. Main reports nope.md in both lines below and the head reports neither:

`` [g](nope.md) `
close it with ``` and see [g](nope.md) and `x`

One fix: (?<!)` before the opening group.
minor: doc_check.py:216: a fence on a list item's own line (- ```sh) is not seen, so its closing line opens a fence. That fence hides the links in the rest of the item, up to the next line indented less than the item. Main hides them too.

@scuttletoby

Copy link
Copy Markdown
Owner Author

lens: state r2 · reviewer: lens agent (Opus) · read at: b02479e
verdict: safe to merge
covered: the PR body, both round-1 lens reports, the fix commit b02479e and the full diff against main (doc_check.py, its test, README and action.yml; fork-ci-tests.cjs, its test and the list)
checks: doc_check.test.py 39 OK and fork-ci-tests.test.cjs 69/69 at the head; Lint and fork-ci-result pass on the head. Nine mutations of the fix round, each turned doc_check.test.py red: no .git refusal, no %00 catch, no "x" rule, no removed_target, a fence at any indent, a fence outliving its list item, no list items, no once-only dedup, no base read through the symlink. In scratch repos with --base: deleting or renaming the target, a chain of two symlinks, and the target's folder turned into a symlink out of the repository each fail as "this change removed ..."; a symlink dangling on the base and a symlink loop are not blamed; a .git target fails; %00 in a touched file is a broken link and in an untouched file is no traceback; 4-space and tab "fences" no longer hide later links; fences in "-" and "1)" items, nested items, a fence ending with its item, "x" (inline code, not a fence) and a "~~~" fence with a backtick in its info all read as CommonMark does; a symlink's WARN says "this change moved it past 90%"; a re-run gives the same output.
state: runs again: read-only, the same output (checked twice in a scratch repo). Stops half-way: nothing is written; exit 0 only after every file and the removed-link scan; the new hops (removed_target, base_doc) stop after 40, so a symlink loop ends and is not blamed. Old or empty state: a symlink already dangling on the base is not blamed; no deletion means no scan; with links off the scan still checks untouched symlinks; base_doc reads '' for a path git cannot see (a folder symlink on the base), which only changes a warning's wording.
round 1, fixed: Lint, the issue number in two test comments (Lint passes on the head)
round 1, fixed: an untouched symlink whose target is deleted, renamed, or whose folder becomes a symlink out of the repository now fails in --base
round 1, fixed: fences open with 0-3 spaces of indent from the list item's content, so an indented "" line no longer hides later links round 1, fixed: the symlink warning wording, the once-only report in --all, .git targets, %00 links, the action.yml description, CRLF in the body's strictness list blocker: none major: none minor: doc_check.py:350 (removed_target) follows the symlink's own text hop by hop but not a folder symlink inside that text. Base: docs/A.md, d -> docs, CLAUDE.md -> d/A.md; a change that deletes docs/A.md, or renames docs, exits 0 in --base although CLAUDE.md now points to no file (--all fails it). Main passes this too, and no symlink in this repository goes through a folder symlink. One fix, checked in a scratch copy: when removed_target finds nothing, also test the symlink's os.path.realpath (relative to the root) against gone; both cases then fail as "this change removed docs/A.md", and the dangling-on-base and loop cases still pass. minor: an untouched symlink is not checked again when its target changes. CLAUDE.md -> AGENTS.md, and a change grows AGENTS.md to 60 lines: --base exits 0, --all fails CLAUDE.md on its own 40-line budget. Not a regression (main never checked symlinks); worth a follow-up decision on which budget a symlink is judged by, since the common CLAUDE.md -> AGENTS.md pattern fails --all whenever AGENTS.md is over 40 lines. minor: link_lines (doc_check.py:193) has no indented code block: a link inside a list item's indented code (for example a "" line 4 spaces past the item's content) is now reported where main hid it after its any-indent fence. Fails closed and matches how top-level indented code was already read.

@scuttletoby
scuttletoby merged commit e3dd5e4 into main Nov 22, 2026
30 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

doc-check: four cases where it passes when it should fail fork-ci declared test formats: minors from the #784 reviews

1 participant