Skip to content

CI: fix duplicate builds, add combined unit+system coverage reporting - #1222

Merged
mkarlesky merged 6 commits into
next_versionfrom
ci-dedup-and-coverage
Aug 21, 2026
Merged

CI: fix duplicate builds, add combined unit+system coverage reporting#1222
mkarlesky merged 6 commits into
next_versionfrom
ci-dedup-and-coverage

Conversation

@mkarlesky

Copy link
Copy Markdown
Member

Summary

Two independent CI improvements.

Fix duplicate CI builds on push + pull_request

All four Phase B PRs from the recent test_invoker refactor (#1218-#1221) each triggered two full CI runs. A prior fix (adf8b885) intended push and pull_request runs for the same commit to share one concurrency group so the second cancels the first, but its group key (ci-${{ github.event.pull_request.number || github.ref }}) evaluates to a different string per event type on the same commit -- the || fallback only ever fires for push, since pull_request always has github.event.pull_request.number set -- so the two runs never actually shared a group. github.head_ref || github.ref_name resolves to the identical bare branch name for both event types, fixing the dedup for real.

Combined unit + system test coverage reporting

Unit tests run in-process (SimpleCov in spec_helper.rb captures them directly). System tests never touch lib/ceedling in-process -- every example shells out to a real bundle exec ruby -S ceedling ... subprocess (SystemContext) against a throwaway, env-stripped Bundler context. Capturing that requires starting SimpleCov inside each child process instead, injected via RUBYOPT (new spec/support/system/simplecov_boot.rb) after Bundler's env-stripping, with SimpleCov.root pointed back at this repo since the child's own CWD is an ephemeral deployed project directory. Both sides accumulate into one shared coverage/.resultset.json; rake coverage:report merges and formats it once after both suites finish.

Two real bugs found and fixed during local verification (not just theorized -- caught by actually running the pipeline end-to-end):

  • GitHub Actions composite-action boolean inputs arrive as the string 'false' when disabled, which Ruby treats as truthy -- every gate compares against an exact expected string ('units'/'system') rather than bare presence/truthiness.
  • spec_helper.rb is also loaded by the system suite's own outer rspec process (which barely touches lib/ceedling directly -- the real work happens in each child subprocess). A bare on/off check there would make that process also claim SimpleCov's "units" resultset entry and silently overwrite the real unit-test coverage, since SimpleCov's multi-process merging replaces same-named entries rather than accumulating them. Fixed by having the env var's value itself say which suite is instrumenting.
  • (Also fixed in the same pass: the throwaway system-test Gemfile added simplecov unpinned, resolving a different version than the main Gemfile and triggering deprecation warnings on stderr that crashed an unrelated, pre-existing .strip-on-raw-stderr code path in compose_failure_report -- pinned to the same ~> 0.22 constraint.)

Everything is gated behind CEEDLING_TEST_COVERAGE, active only on CI's tests-linux Ruby-3.3 leg (reusing that job's existing tool installs rather than adding a separate job). simplecov is a Gemfile-only dependency, never in ceedling.gemspec, matching the existing diff-lcs precedent -- confirmed via an actual gem build that no simplecov dependency or coverage/ content ships in the built gem. coverage/ is gitignored.

Test plan

  • bundle exec rspec spec/units -- 2589 examples, 0 failures
  • Local coverage dry run: CEEDLING_TEST_COVERAGE=units rspec spec/units/..., then CEEDLING_TEST_COVERAGE=system rspec spec/system/..., then rake coverage:report -- produces one merged coverage/index.html with both "units" and multiple "system-<pid>-<timestamp>" entries in the resultset
  • Confirmed CEEDLING_TEST_COVERAGE unset (the normal/default path) produces zero behavior change and no coverage/ output
  • gem build ceedling.gemspec from a clean state -- succeeds, confirmed no simplecov dependency and no coverage output in the built gem's file list
  • CI: confirm the tests-linux Ruby-3.3 leg uploads a coverage-report artifact and that pushing + immediately opening a PR from the same branch results in one run completing (the other cancelled), not two

🤖 Generated with Claude Code

mkarlesky and others added 6 commits August 21, 2026 00:11
github.event.pull_request.number is always set on a pull_request event,
so the old ||-fallback to github.ref never actually triggered there --
push and pull_request runs for the same commit landed in different
groups and neither cancelled the other. head_ref || ref_name resolves
to the same bare branch name for both event types.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Unit tests run in-process, so SimpleCov started in spec_helper.rb
captures them directly. System tests never touch lib/ceedling
in-process at all -- every example shells out to a real `bundle exec
ruby -S ceedling ...` subprocess via SystemContext, running against a
throwaway Bundler environment that strips the outer process's env
clean. Capturing that requires starting SimpleCov inside each child
process instead, injected via RUBYOPT (spec/support/system/
simplecov_boot.rb) after Bundler's env-stripping, with root pointed
back at this repo since the child's own CWD is an ephemeral deployed
project directory. Both sides accumulate into one shared
coverage/.resultset.json, differentiated by SimpleCov's own
command_name, and `rake coverage:report` merges and formats it once
after both suites finish.

Everything is gated behind CEEDLING_TEST_COVERAGE, whose value (not
just its presence) matters: spec_helper.rb is also loaded by the
system suite's own outer rspec process, which barely touches
lib/ceedling directly, so distinguishing 'units' from 'system' keeps
that process from claiming (and silently overwriting) the real
unit-test resultset entry. Only enabled on CI's tests-linux Ruby-3.3
leg, reusing that job's existing tool installs rather than adding a
separate job. simplecov is a Gemfile-only dependency (never in
ceedling.gemspec, matching the existing diff-lcs precedent), and the
throwaway system-test Gemfile only gains it, pinned to the same
version constraint, when coverage mode is on.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…bugs

Adds `bin`, `lib`, and `plugins` group tabs to every report via add_group,
and generates units-only and system-only reports alongside the existing
combined one from the same resultset data (Rakefile coverage:report task).

Also fixes two pre-existing bugs in the coverage instrumentation from the
prior commit: track_files was being called multiple times and only the
last call took effect (SimpleCov overwrites rather than accumulates), so
lib/ tracking was silently dropped; consolidated into one brace-glob call.
Separately, the system-test child processes' backfill-untouched-files glob
resolves against the process's actual working directory rather than
SimpleCov.root, so it silently found nothing while CWD stayed in the
throwaway deployed test project -- simplecov_boot.rb now chdirs into the
repo for that call.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…b to run far longer than the other legs

Every system-test child process shared one coverage/.resultset.json.
SimpleCov's own store_result/merged_result pair reads, re-merges, and
rewrites that entire shared file on every single process's exit --
confirmed in the installed simplecov gem source -- so each new process
paid a cost proportional to every process that ran before it, and the
gap between the coverage job and the other CI legs widened the longer
a system-test run went.

Each system-test child process now gets its own small resultset file
under coverage/raw/ instead of sharing one growing file, so each
process's own write stays O(1). coverage:report merges all of those
files together at the end via SimpleCov::ResultMerger.merge_results,
the same one-file-at-a-time approach SimpleCov's own docs recommend
for large multi-process CI setups.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…ping stale results

coverage:report runs after both specs:units and specs:system finish, reading
back result files those already-finished steps wrote. SimpleCov's default
merge_results call filters out any file older than its 10-minute
merge_timeout -- meant for live in-process merges, not this after-the-fact
read-back -- so a system-test suite running longer than 10 minutes left the
early units file past the cutoff by the time this task ran, dropping it
entirely and crashing HTMLFormatter#format on a nil result. Passing
ignore_timeout: true matches what SimpleCov.collate's own API defaults to
for this same after-the-fact-merge use case.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…st, cppcheck cache permissions

Node 20 deprecation: actions/checkout@v4, actions/cache@v4, and
actions/upload-artifact@v4 (plus actions/download-artifact@v4, the same
family though not itself named in the warning) still resolve to releases
whose action.yml declares Node 20, which GitHub Actions runners now force
onto Node 24 anyway. Bumped every occurrence to each action's latest major
(checkout v7, cache v6, upload-artifact v7, download-artifact v8), all of
which declare Node 24 directly -- confirmed against each action's actual
released action.yml, not assumed.

Homebrew aws/tap trust warning (macOS job): aws/tap is pre-tapped on
GitHub's hosted macOS runner image and unrelated to anything this repo
uses; newer Homebrew warns about every untrusted tap present on any brew
install, not just ones involved in that install. Untapped before
installing cppcheck rather than trusting it or disabling tap-trust checks
outright.

cppcheck cache restore failure (Linux job): actions/cache's restore step
runs unprivileged and needs write access to recreate cached paths on a
fresh runner VM. /usr/local/bin is writable on this runner image but
/usr/local/share is not, so extracting the cached cppcheck subtree there
failed with "Cannot mkdir: Permission denied" on every cache hit. Added a
step to pre-create and chown the destination before the cache-restore
step runs, every run, since each job starts on a fresh ephemeral VM.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@mkarlesky
mkarlesky merged commit 9134f8d into next_version Aug 21, 2026
20 of 27 checks passed
@mkarlesky
mkarlesky deleted the ci-dedup-and-coverage branch August 21, 2026 18:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant