Skip to content

fix(vale): pin Vale 3.23.0's contract, and accept a rule's tests: key - #428

Merged
theCodeDrift merged 2 commits into
vendor/vale/upgradefrom
vendor/vale/3-23-0-contract
Sep 30, 2026
Merged

theCodeDrift merged 2 commits into
vendor/vale/upgradefrom
vendor/vale/3-23-0-contract

Conversation

@theCodeDrift

@theCodeDrift theCodeDrift commented Sep 30, 2026 •

Copy link
Copy Markdown
Member

Stack (root → tip):

#427 moves the six @taskless/vale-* pins to Vale 3.23.0. Its Validate never ran (every workflow is sitting at action_required), and run locally it is red: three contract failures. This PR makes the upgrade correct and pins what 3.23.0 changed. It merges down into vendor/vale/upgrade, so pin, contract, and guidance reach main together.

Why #427 is red

One cause, three failures. A standalone scope: doc(<leaf>) (doc(h2)) now reads the leaf's own text, where 3.21.0 and 3.22.0 matched nothing. scope/doc-leaf-standalone in the corpus recorded that as a divergence, and the recipe taught text & doc(h2) around it. This closes a trap rather than opening one; the test's own comment anticipated it.

  • The corpus row now expects accepted and carries no divergence.
  • The 3.21.0 vendor test keeps only the chained half, which is still true.
  • A new Vale 3.23.0 block pins the standalone form firing.

What changed that no test caught

Each case below was run against both binaries on the same fixture and differs between them:

3.22.0 3.23.0
.kt, .kts whole file is prose comments only
//nolint (Go), # noqa (Py), eslint-disable (JS) linted not read
docstring :param x: field linted not read
doc(h2) alone nothing fires

The capabilities table says a bump re-measures everything, and every existing row held, so the suite was green. The format Vale learned only shows up in the source: v3.22.0...v3.23.0 adds internal/lint/code/kt.go, and internal/core/format.go routes .kt/.kts to code. .kt and .kts are new comment rows. .toml also gained a grammar, but routes to data, which needs a View, so it still reads as plaintext here, and that is recorded in the table note too.

tests: was already a Vale field

verify rejected tests: as "an E201 that suppresses every other Vale rule". Measured: 3.22.0 and 3.23.0 both load it with any value and no diagnostic. The key was simply missing from the generator's hand-seeded FIELD_CANDIDATES, which that list's own header calls the worse failure. It is added, and the regenerated vocabulary records it as a common field Vale reads literally. There is a corpus row for it.

Guidance

  • create-vale-rule (topic v15): the leaf paragraph no longer calls the standalone form inert, and the comment-tier bullet says tool-directive comments and :param: fields are not read.
  • update (topic v12): the 0.12.0 ledger now says 3.21.0 → 3.23.0 (by way of 3.22.0) and gains three 3.23.0 entries.
  • The changeset on chore(vale): upgrade to Vale 3.23.0 #427 is extended, not duplicated, and stays patch.

Checks

pnpm lint, pnpm typecheck, and pnpm test (1905 passed) are green locally on this branch, built.

The PR that adds CI checks for this class of drift stacks on top of this one.

Refs #427

Contains #429

Stacks on #428, which stacks on #427. Two checks for the drift #427 slipped through. Neither one gates.

Validate never runs on the upgrade PR

vale-upgrade.yml left the contract suites to Validate: "left to Validate, it is a red check on a pull request that already carries upstream's notes." That red check never appears. A pull_request run triggered by github-actions[bot] waits at action_required until a maintainer approves it:

2026-09-30 vendor/vale/upgrade      action_required  trig=github-actions[bot]   (#427)
2026-09-15 vendor/vale/upgrade      action_required  trig=github-actions[bot]
2026-09-15 vendor/ast-grep/upgrade  action_required  trig=github-actions[bot]

Every earlier upgrade got its first Validate run when a human pushed. #427 read as a PR with no checks while three contract tests were failing.

Now: the job runs vale-schema-contract and vale-vendor-contract itself, with continue-on-error so a changed verdict still proposes the upgrade, as the header intends. vale-upgrade-report.cjs then renders the verdict into the PR body.

A format Vale learns can't be caught by a test

VALE_FORMAT_TIERS is re-probed row by row, which catches a format that moved. An extension with no row is never probed, so a format Vale learned only shows up in upstream's source. The table's header asked for that check by hand on every bump. On 3.23.0 nobody did it, and code/kt.go moved Kotlin to the comment tier with the suite green (#428 adds the rows).

Now: the report lists the files upstream added under internal/lint/ between the pinned tag and the new one, using the compare API. It names readers, not extensions, and says to read internal/core/format.go and probe each one before adding a row. That matters most for the dangerous case: a reader that shells out to a converter turns a missing row into a crash that takes down the whole Vale run.

What #427's body would have said

Rendered from the real compare API and #427's actual test report:

### Contract suites
**3 of 204 failed.** The new Vale changed a recorded behaviour. ...
  vale-schema-contract.test.ts > ... agrees with every recorded verdict
    AssertionError: scope/doc-leaf-standalone (scope: doc(h1)): recorded ignored, Vale 3.23.0 says accepted ...
  (2 more)

### Format readers upstream added
- internal/lint/code/doc.go
- internal/lint/code/kt.go
- internal/lint/code/toml.go
- internal/lint/quote.go
- internal/lint/txtdoc.go

Silence never reads as a pass

A missing test report renders Did not run. A failed fetch renders Could not check with the error. A comparison at the API's 300-file cap is flagged as possibly incomplete. If the report step fails outright, the body says Not measured. Each case is covered in vale-upgrade-report.test.cjs (11 tests).

Not in this PR

  • ast-grep-upgrade.yml has the same action_required exposure, as the table above shows. The contract half of this would port directly; the format-reader half is Vale-specific.
  • Any repo setting that would let bot-triggered runs start without approval. That is a security trade-off for a maintainer to decide, not something to change from a PR.

pnpm lint and pnpm test:scripts (500 passed) are green locally. The workflow parses, and the new step order is contract, then report, then propose.

Refs #427

Three contract failures on #427 had one cause: a standalone
`doc(<leaf>)` scope now reads the leaf's text, where 3.21.0 and 3.22.0
matched nothing. The corpus row agrees with the schema now and drops
its divergence; the 3.21.0 test keeps only the chained half, which is
still true; a new 3.23.0 block pins the standalone form firing.

What no test caught, measured against both binaries on the same
fixtures:

- `.kt` and `.kts` moved from the plaintext fallback to the comment
  tier (upstream adds internal/lint/code/kt.go; format.go routes both
  to `code`). Two new rows in VALE_FORMAT_TIERS. `.toml` did not move.
- A comment addressed to a tool (`//nolint`, `# noqa`,
  `eslint-disable`) and a docstring's `:param:` field are no longer
  read. Pinned in the 3.23.0 block.
- `tests:` is a field of every check, accepted by 3.22.0 as well, and
  was missing from the generator's candidates, so `verify` rejected it
  as an E201 Vale never raised. Added as a candidate and regenerated;
  it is common and read literally.

create-vale-rule (topic v15) and update (topic v12) say all of it, and
the changeset carries the release note.
…rs on the upgrade PR

The upgrade workflow left the contract suites to Validate, "a red check
on a pull request that already carries upstream's notes". Validate does
not run there: a pull_request run triggered by github-actions[bot]
waits at action_required until a maintainer approves it. Every
bot-opened upgrade, Vale and ast-grep alike, measured the same way, and
#427 read as a pull request with no checks while three contract tests
failed.

The job now runs vale-schema-contract and vale-vendor-contract itself,
continue-on-error so a changed verdict still proposes the upgrade, and
vale-upgrade-report.cjs renders the verdict into the body.

The same report lists the files upstream added under internal/lint/.
That is the source check capabilities.ts asked for by hand on every
bump, and the one that would have caught kt.go on 3.23.0: an extension
with no VALE_FORMAT_TIERS row is never probed, so no test can.

Neither half gates. A missing test report or a failed compare fetch is
said in the body, so "not checked" never reads as a pass.
@theCodeDrift
theCodeDrift merged commit 6fa93df into vendor/vale/upgrade Sep 30, 2026
4 checks passed
@theCodeDrift
theCodeDrift deleted the vendor/vale/3-23-0-contract branch September 30, 2026 17:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant