Skip to content

ci(vale): report the contract verdict and upstream's new format readers on the upgrade PR - #429

Merged
theCodeDrift merged 1 commit into
vendor/vale/3-23-0-contractfrom
vendor/vale/upgrade-report
Sep 30, 2026
Merged

theCodeDrift merged 1 commit into
vendor/vale/3-23-0-contractfrom
vendor/vale/upgrade-report

Conversation

@theCodeDrift

@theCodeDrift theCodeDrift commented Sep 30, 2026 •

Copy link
Copy Markdown
Member

Stack (root → tip):

Stacks on #428, which stacks on #427. Two checks for the drift #427 slipped through. Neither one gates.

Validate never runs on the upgrade PR

vale-upgrade.yml left the contract suites to Validate: "left to Validate, it is a red check on a pull request that already carries upstream's notes." That red check never appears. A pull_request run triggered by github-actions[bot] waits at action_required until a maintainer approves it:

2026-09-30 vendor/vale/upgrade      action_required  trig=github-actions[bot]   (#427)
2026-09-15 vendor/vale/upgrade      action_required  trig=github-actions[bot]
2026-09-15 vendor/ast-grep/upgrade  action_required  trig=github-actions[bot]

Every earlier upgrade got its first Validate run when a human pushed. #427 read as a PR with no checks while three contract tests were failing.

Now: the job runs vale-schema-contract and vale-vendor-contract itself, with continue-on-error so a changed verdict still proposes the upgrade, as the header intends. vale-upgrade-report.cjs then renders the verdict into the PR body.

A format Vale learns can't be caught by a test

VALE_FORMAT_TIERS is re-probed row by row, which catches a format that moved. An extension with no row is never probed, so a format Vale learned only shows up in upstream's source. The table's header asked for that check by hand on every bump. On 3.23.0 nobody did it, and code/kt.go moved Kotlin to the comment tier with the suite green (#428 adds the rows).

Now: the report lists the files upstream added under internal/lint/ between the pinned tag and the new one, using the compare API. It names readers, not extensions, and says to read internal/core/format.go and probe each one before adding a row. That matters most for the dangerous case: a reader that shells out to a converter turns a missing row into a crash that takes down the whole Vale run.

What #427's body would have said

Rendered from the real compare API and #427's actual test report:

### Contract suites
**3 of 204 failed.** The new Vale changed a recorded behaviour. ...
  vale-schema-contract.test.ts > ... agrees with every recorded verdict
    AssertionError: scope/doc-leaf-standalone (scope: doc(h1)): recorded ignored, Vale 3.23.0 says accepted ...
  (2 more)

### Format readers upstream added
- internal/lint/code/doc.go
- internal/lint/code/kt.go
- internal/lint/code/toml.go
- internal/lint/quote.go
- internal/lint/txtdoc.go

Silence never reads as a pass

A missing test report renders Did not run. A failed fetch renders Could not check with the error. A comparison at the API's 300-file cap is flagged as possibly incomplete. If the report step fails outright, the body says Not measured. Each case is covered in vale-upgrade-report.test.cjs (11 tests).

Not in this PR

  • ast-grep-upgrade.yml has the same action_required exposure, as the table above shows. The contract half of this would port directly; the format-reader half is Vale-specific.
  • Any repo setting that would let bot-triggered runs start without approval. That is a security trade-off for a maintainer to decide, not something to change from a PR.

pnpm lint and pnpm test:scripts (500 passed) are green locally. The workflow parses, and the new step order is contract, then report, then propose.

Refs #427

…rs on the upgrade PR

The upgrade workflow left the contract suites to Validate, "a red check
on a pull request that already carries upstream's notes". Validate does
not run there: a pull_request run triggered by github-actions[bot]
waits at action_required until a maintainer approves it. Every
bot-opened upgrade, Vale and ast-grep alike, measured the same way, and
#427 read as a pull request with no checks while three contract tests
failed.

The job now runs vale-schema-contract and vale-vendor-contract itself,
continue-on-error so a changed verdict still proposes the upgrade, and
vale-upgrade-report.cjs renders the verdict into the body.

The same report lists the files upstream added under internal/lint/.
That is the source check capabilities.ts asked for by hand on every
bump, and the one that would have caught kt.go on 3.23.0: an extension
with no VALE_FORMAT_TIERS row is never probed, so no test can.

Neither half gates. A missing test report or a failed compare fetch is
said in the body, so "not checked" never reads as a pass.
@theCodeDrift
theCodeDrift added this pull request to stack #430 September 30, 2026 16:03
@theCodeDrift
theCodeDrift removed this pull request from stack #430 September 30, 2026 17:16
@theCodeDrift
theCodeDrift merged commit d44d055 into vendor/vale/3-23-0-contract Sep 30, 2026
8 checks passed
@theCodeDrift
theCodeDrift deleted the vendor/vale/upgrade-report branch September 30, 2026 17:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant