CI: fix duplicate builds, add combined unit+system coverage reporting - #1222
Merged
Conversation
github.event.pull_request.number is always set on a pull_request event, so the old ||-fallback to github.ref never actually triggered there -- push and pull_request runs for the same commit landed in different groups and neither cancelled the other. head_ref || ref_name resolves to the same bare branch name for both event types. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Unit tests run in-process, so SimpleCov started in spec_helper.rb captures them directly. System tests never touch lib/ceedling in-process at all -- every example shells out to a real `bundle exec ruby -S ceedling ...` subprocess via SystemContext, running against a throwaway Bundler environment that strips the outer process's env clean. Capturing that requires starting SimpleCov inside each child process instead, injected via RUBYOPT (spec/support/system/ simplecov_boot.rb) after Bundler's env-stripping, with root pointed back at this repo since the child's own CWD is an ephemeral deployed project directory. Both sides accumulate into one shared coverage/.resultset.json, differentiated by SimpleCov's own command_name, and `rake coverage:report` merges and formats it once after both suites finish. Everything is gated behind CEEDLING_TEST_COVERAGE, whose value (not just its presence) matters: spec_helper.rb is also loaded by the system suite's own outer rspec process, which barely touches lib/ceedling directly, so distinguishing 'units' from 'system' keeps that process from claiming (and silently overwriting) the real unit-test resultset entry. Only enabled on CI's tests-linux Ruby-3.3 leg, reusing that job's existing tool installs rather than adding a separate job. simplecov is a Gemfile-only dependency (never in ceedling.gemspec, matching the existing diff-lcs precedent), and the throwaway system-test Gemfile only gains it, pinned to the same version constraint, when coverage mode is on. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…bugs Adds `bin`, `lib`, and `plugins` group tabs to every report via add_group, and generates units-only and system-only reports alongside the existing combined one from the same resultset data (Rakefile coverage:report task). Also fixes two pre-existing bugs in the coverage instrumentation from the prior commit: track_files was being called multiple times and only the last call took effect (SimpleCov overwrites rather than accumulates), so lib/ tracking was silently dropped; consolidated into one brace-glob call. Separately, the system-test child processes' backfill-untouched-files glob resolves against the process's actual working directory rather than SimpleCov.root, so it silently found nothing while CWD stayed in the throwaway deployed test project -- simplecov_boot.rb now chdirs into the repo for that call. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…b to run far longer than the other legs Every system-test child process shared one coverage/.resultset.json. SimpleCov's own store_result/merged_result pair reads, re-merges, and rewrites that entire shared file on every single process's exit -- confirmed in the installed simplecov gem source -- so each new process paid a cost proportional to every process that ran before it, and the gap between the coverage job and the other CI legs widened the longer a system-test run went. Each system-test child process now gets its own small resultset file under coverage/raw/ instead of sharing one growing file, so each process's own write stays O(1). coverage:report merges all of those files together at the end via SimpleCov::ResultMerger.merge_results, the same one-file-at-a-time approach SimpleCov's own docs recommend for large multi-process CI setups. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…ping stale results coverage:report runs after both specs:units and specs:system finish, reading back result files those already-finished steps wrote. SimpleCov's default merge_results call filters out any file older than its 10-minute merge_timeout -- meant for live in-process merges, not this after-the-fact read-back -- so a system-test suite running longer than 10 minutes left the early units file past the cutoff by the time this task ran, dropping it entirely and crashing HTMLFormatter#format on a nil result. Passing ignore_timeout: true matches what SimpleCov.collate's own API defaults to for this same after-the-fact-merge use case. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…st, cppcheck cache permissions Node 20 deprecation: actions/checkout@v4, actions/cache@v4, and actions/upload-artifact@v4 (plus actions/download-artifact@v4, the same family though not itself named in the warning) still resolve to releases whose action.yml declares Node 20, which GitHub Actions runners now force onto Node 24 anyway. Bumped every occurrence to each action's latest major (checkout v7, cache v6, upload-artifact v7, download-artifact v8), all of which declare Node 24 directly -- confirmed against each action's actual released action.yml, not assumed. Homebrew aws/tap trust warning (macOS job): aws/tap is pre-tapped on GitHub's hosted macOS runner image and unrelated to anything this repo uses; newer Homebrew warns about every untrusted tap present on any brew install, not just ones involved in that install. Untapped before installing cppcheck rather than trusting it or disabling tap-trust checks outright. cppcheck cache restore failure (Linux job): actions/cache's restore step runs unprivileged and needs write access to recreate cached paths on a fresh runner VM. /usr/local/bin is writable on this runner image but /usr/local/share is not, so extracting the cached cppcheck subtree there failed with "Cannot mkdir: Permission denied" on every cache hit. Added a step to pre-create and chown the destination before the cache-restore step runs, every run, since each job starts on a fresh ephemeral VM. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two independent CI improvements.
Fix duplicate CI builds on push + pull_request
All four Phase B PRs from the recent
test_invokerrefactor (#1218-#1221) each triggered two full CI runs. A prior fix (adf8b885) intendedpushandpull_requestruns for the same commit to share one concurrency group so the second cancels the first, but its group key (ci-${{ github.event.pull_request.number || github.ref }}) evaluates to a different string per event type on the same commit -- the||fallback only ever fires forpush, sincepull_requestalways hasgithub.event.pull_request.numberset -- so the two runs never actually shared a group.github.head_ref || github.ref_nameresolves to the identical bare branch name for both event types, fixing the dedup for real.Combined unit + system test coverage reporting
Unit tests run in-process (SimpleCov in
spec_helper.rbcaptures them directly). System tests never touchlib/ceedlingin-process -- every example shells out to a realbundle exec ruby -S ceedling ...subprocess (SystemContext) against a throwaway, env-stripped Bundler context. Capturing that requires starting SimpleCov inside each child process instead, injected viaRUBYOPT(newspec/support/system/simplecov_boot.rb) after Bundler's env-stripping, withSimpleCov.rootpointed back at this repo since the child's own CWD is an ephemeral deployed project directory. Both sides accumulate into one sharedcoverage/.resultset.json;rake coverage:reportmerges and formats it once after both suites finish.Two real bugs found and fixed during local verification (not just theorized -- caught by actually running the pipeline end-to-end):
'false'when disabled, which Ruby treats as truthy -- every gate compares against an exact expected string ('units'/'system') rather than bare presence/truthiness.spec_helper.rbis also loaded by the system suite's own outer rspec process (which barely toucheslib/ceedlingdirectly -- the real work happens in each child subprocess). A bare on/off check there would make that process also claim SimpleCov's"units"resultset entry and silently overwrite the real unit-test coverage, since SimpleCov's multi-process merging replaces same-named entries rather than accumulating them. Fixed by having the env var's value itself say which suite is instrumenting.simplecovunpinned, resolving a different version than the main Gemfile and triggering deprecation warnings on stderr that crashed an unrelated, pre-existing.strip-on-raw-stderr code path incompose_failure_report-- pinned to the same~> 0.22constraint.)Everything is gated behind
CEEDLING_TEST_COVERAGE, active only on CI'stests-linuxRuby-3.3 leg (reusing that job's existing tool installs rather than adding a separate job).simplecovis aGemfile-only dependency, never inceedling.gemspec, matching the existingdiff-lcsprecedent -- confirmed via an actualgem buildthat nosimplecovdependency orcoverage/content ships in the built gem.coverage/is gitignored.Test plan
bundle exec rspec spec/units-- 2589 examples, 0 failuresCEEDLING_TEST_COVERAGE=units rspec spec/units/..., thenCEEDLING_TEST_COVERAGE=system rspec spec/system/..., thenrake coverage:report-- produces one mergedcoverage/index.htmlwith both"units"and multiple"system-<pid>-<timestamp>"entries in the resultsetCEEDLING_TEST_COVERAGEunset (the normal/default path) produces zero behavior change and nocoverage/outputgem build ceedling.gemspecfrom a clean state -- succeeds, confirmed nosimplecovdependency and no coverage output in the built gem's file listtests-linuxRuby-3.3 leg uploads acoverage-reportartifact and that pushing + immediately opening a PR from the same branch results in one run completing (the other cancelled), not two🤖 Generated with Claude Code