Skip to content

ci: measure coverage and report to Coveralls - #11

Merged
dkijania merged 3 commits into
mainfrom
ci/add-coverage
Jul 22, 2026
Merged

dkijania merged 3 commits into
mainfrom
ci/add-coverage

Conversation

@dkijania

Copy link
Copy Markdown
Member

Why

mina-release-toolkit's codecov.yml lists deb-toolkit/ under ignore, with the comment that the submodules "report coverage from their own repos". This repo did not — there was no coverage job here, so the crate reported coverage nowhere. The parent has per-crate Codecov flags and badges for the three in-tree crates; deb-toolkit was the gap.

What

A cargo-llvm-cov job uploading to Codecov under a deb-toolkit flag, plus a codecov.yml mirroring the parent repo's conventions so the two report comparably.

Two things the coverage job has to copy from ci.yml rather than taking defaults:

  • ubuntu-22.04, not ubuntu-latest — same reason ci.yml pins it: the bundled GPG signing-key fixture under tests/res/ uses a digest algorithm gpg 2.4 rejects. Coverage has to run the same tests CI runs, so it needs the same runner.
  • Installs debsigs and debsig-verify — without them the integration tests skip themselves by design (see README) rather than fail. Everything they exercise — signer.rs, signature_verifier.rs, viewer.rs, and the packaging half of builder.rs — would then be reported as uncovered. That understates coverage instead of measuring it.

Baseline

Measured locally on main with the full integration toolchain present:

TOTAL    regions 77.28%    functions 72.73%    lines 79.99%

The spread across modules is wide:

Module Line cov
session/apply.rs 94.4%
session/control.rs 94.8%
session/compression.rs 91.9%
builder.rs 84.4%
session/open.rs 85.9%
signature_verifier.rs 74.6%
signer.rs 71.4%
content_verifier.rs 53.2%
misc.rs 31.6%

Because of that spread, thresholds start soft and advisory: project auto with 1% slack, patch target 70%, both if_ci_failed: ignore, and fail_ci_if_error: false on the upload so a Codecov outage cannot turn a PR red. Worth tightening once there is trend data showing each module's natural baseline.

Verification

cargo llvm-cov --all-features --workspace --lcov --output-path lcov.info — the exact command the workflow runs — was run locally and produces a valid lcov.info. Both YAML files parse. lcov.info is added to .gitignore so the artifact can't be committed by accident.

Note

Overlaps #10 only in README.md, and in a different hunk (badges at the top vs. the "Known limitation" section near the bottom), so the two merge cleanly in either order.

dkijania added 2 commits July 21, 2026 23:15
mina-release-toolkit's codecov.yml lists deb-toolkit/ under `ignore`,
on the stated basis that this repo reports coverage from its own CI.
It did not: there was no coverage job here, so the crate reported
nothing anywhere.

Adds a cargo-llvm-cov job uploading to Codecov under a deb-toolkit
flag, mirroring the parent repo's workflow so the two report
comparably.

Two things the job has to match from ci.yml rather than defaulting:
it pins ubuntu-22.04, because the tests/res GPG fixture uses a digest
gpg 2.4 rejects; and it installs debsigs and debsig-verify, without
which the integration tests skip themselves by design and everything
they cover (signer, signature_verifier, viewer, and the packaging half
of builder) would be reported uncovered rather than measured.

Thresholds start soft — project auto with 1% slack, patch 70%, both
advisory. The crate currently measures ~77% region / ~80% line, with a
wide per-module spread, so there is not yet a basis for a hard gate.
Codecov rejects tokenless uploads outright. With fail_ci_if_error
false, a missing CODECOV_TOKEN leaves the job green while publishing
nothing — which is the state the sibling crates' coverage jobs in
mina-release-toolkit have been in since May: green runs, badges, and
no data behind them.

Emit a warning annotation when the secret is absent so the condition
is visible on the PR.
@dkijania

Copy link
Copy Markdown
Member Author

Heads-up on a blocker this surfaced — the coverage job is green, but the upload is being rejected:

warning -- Branch `ci/add-coverage` is protected but no token was provided
error   -- Commit creating failed:  {"message":"Token required - not valid tokenless upload"}
error   -- Report creating failed:  {"message":"Token required - not valid tokenless upload"}
error   -- Upload queued for processing failed: {"message":"Token required - not valid tokenless upload"}

Codecov no longer accepts tokenless uploads. fail_ci_if_error: false keeps the job green anyway, so the report is generated and then discarded.

This is not specific to this PR. The same thing is happening in mina-release-toolkit, and has been since at least May 2026 — every one of the three crates' coverage jobs fails the same way (run 26185047893 shows it for buildkite-cache-manager and mina-bench-upload), while the workflow reports success and the README displays coverage badges. Neither repo has a CODECOV_TOKEN secret set.

To make this actually publish: add CODECOV_TOKEN to this repo's Actions secrets (Settings → Secrets and variables → Actions). Worth doing the same in mina-release-toolkit, otherwise its badges stay unbacked.

I pushed 639b346 so the condition is at least visible rather than silent: the job now emits a warning annotation when the secret is missing. I kept fail_ci_if_error: false — turning it on would make CI red until the token exists, which is the wrong default for an observability signal.

Codecov requires an upload token even for public repos — tokenless
uploads are accepted only for PRs from forks, and any other branch,
including a plain feature branch, is rejected with "Token required -
not valid tokenless upload".

Both ways of obtaining that token are gated on GitHub organization
ownership, and the people who need coverage here are org members with
repo admin, not org owners. Waiting on an owner to provision a secret
is what left mina-release-toolkit's three coverage jobs green and
empty since May.

Coveralls authenticates with the GITHUB_TOKEN that Actions injects
automatically — its docs are explicit that it must not be added to the
secrets store — and creates the repo on first upload. No secret, no
approval, nothing to rotate.

Drops codecov.yml; Coveralls keeps thresholds in its own UI rather
than in-repo. Swaps the README badge.
@dkijania dkijania changed the title ci: measure and upload coverage ci: measure coverage and report to Coveralls Jul 22, 2026
@dkijania

Copy link
Copy Markdown
Member Author

Switched this from Codecov to Coveralls in 49aa531.

Why: Codecov requires an upload token even for public repos — tokenless uploads are accepted only for PRs from forks, so an ordinary branch is rejected. Both the repo-level and global tokens are reachable only by a GitHub org owner, and the people who actually need coverage here (me included) are org member + repo admin. Codecov's own UI says it plainly: "Contact your admins to manage the upload token settings."

Waiting on an owner to provision a secret is exactly what left mina-release-toolkit's three coverage jobs green and empty since May.

Coveralls authenticates with the GITHUB_TOKEN that Actions injects automatically — its docs are explicit it must not be added to the secrets store — and it creates the repo on first upload. No secret, no org approval, nothing to rotate.

Changes: codecov.yml removed (Coveralls keeps thresholds in its own UI); README badge swapped; the missing-token warning step is gone, since there is no longer a token to miss.

Trade-offs, stated honestly:

  • Codecov is the more open of the two — its server is source-available under the FSL and converts to Apache 2.0 after two years, whereas Coveralls' service is fully closed. This change trades openness for the ability to actually ship coverage.
  • Per-crate flags in mina-release-toolkit map onto Coveralls' flag-name/parallel-build model, but I have not verified how cleanly. Not an issue for this repo, which is a single crate.

If an org owner would rather provision CODECOV_TOKEN org-wide, reverting 49aa531 restores the Codecov setup intact.

@coveralls

Copy link
Copy Markdown

Coverage Report for CI Build 29901343755

Warning

No base build found for commit efec997 on main.
Coverage changes can't be calculated without a base build.
If a base build is processing, this comment will update automatically when it completes.

Coverage: 81.818%

Details

  • Patch coverage: No coverable lines changed in this PR.

Uncovered Changes

No uncovered changes found.

Coverage Regressions

Requires a base build to compare against. How to fix this →


Coverage Stats

Coverage Status
Relevant Lines: 2079
Covered Lines: 1701
Line Coverage: 81.82%
Coverage Strength: 10.49 hits per line

💛 - Coveralls

@dkijania
dkijania merged commit 16a76d6 into main Jul 22, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants