Runbook: external setup to enable the CodSpeed benchmark CI (org, OIDC, macro runners)

[FBumann]

TODO (human): one or two sentences on why we want this captured — e.g. "so we can stand the CodSpeed benchmark CI back up on upstream linopy without rediscovering the org/runner dance."

Note

The following content was generated by AI (drafted by Claude while wiring CodSpeed up on this fork). It's a setup runbook, not discussion.

What this covers

The benchmark workflows are committed (.github/workflows/codspeed.yml, codspeed-memory.yml, codspeed-macro.yml), but they only report once a chunk of out-of-band setup exists that the repo can't carry. This is that checklist — for re-creating the setup on another repo/org (e.g. upstream linopy) later.

Prerequisites

• The repo must live under a GitHub organization. Walltime/macro runners are org-only — a personal account cannot use them (CodSpeed greys the feature out with "this feature is only available for organization accounts").
• CodSpeedHQ/action@v4. v4 made mode a required input — every job must set mode: to one of simulation | walltime | memory. (This bit us: omitting it fails the step instantly.)

1. Connect CodSpeed

1. Install the CodSpeed GitHub App on the org (not a personal account): app.codspeed.io → import → select the org → add the repo.
2. Make sure the repo appears under the org workspace on CodSpeed, not your personal workspace. A repo transferred into the org keeps its old personal CodSpeed project; if it shows under personal, disconnect it there and re-add it under the org, or macro stays greyed out even though the app is on the org.

2. Auth — OIDC (no token secret)

• The workflows use OIDC, so there is no CODSPEED_TOKEN secret to manage. Each job needs:

  permissions:
    contents: read   # actions/checkout
    id-token: write  # OIDC auth with CodSpeed

• Once the app is connected (step 1), uploads authenticate automatically. Verified: the run logs show Performance data uploaded / Linked repository: <org>/<repo>.

3. Macro / walltime runners (org-only)

1. CodSpeed side: in the org workspace settings, enable macro runners. This registers CodSpeed's bare-metal runners to the GitHub org (they appear in the org's Default runner group).
2. GitHub side (public repos only): https://github.com/organizations/<org>/settings/actions/runner-groups → Default → tick "Allow public repositories". Without this the runs-on: codspeed-macro job sits queued forever (no eligible runner).
3. Verified: after both, a previously-stuck codspeed-macro job is picked up by a runner within seconds.

4. Security hardening (do this)

• Restrict the runner group to selected repositories (just this repo), not "all repositories" — limits blast radius on a public org.
• Optionally restrict the runner group to the macro workflow (.../codspeed-macro.yml@refs/heads/master) so nothing else can land jobs on the bare-metal runners.
• Deliberate design property: codspeed-macro.yml triggers on push: master + workflow_dispatch only — never pull_request — so untrusted fork PRs cannot execute on the self-hosted/macro runner (the main public-repo self-hosted-runner risk). Do not add a pull_request trigger to the macro workflow. (The PR-triggered simulation/memory jobs run on GitHub-hosted ubuntu-latest, not the macro runner.)

5. Workflows & cadence (already in the repo)

Workflow	mode	Runs on	Triggers	Cost
codspeed.yml	simulation	GitHub ubuntu-latest	push master + PR→master + dispatch	free
codspeed-memory.yml	memory	GitHub ubuntu-latest	push master + PR→master + dispatch	free
codspeed-macro.yml	walltime	codspeed-macro (org)	push master + dispatch	macro-minutes

All three are continue-on-error: true (non-gating) until baselines/noise are characterized.

6. Plan limits

• Free plan includes 600 macro-runner minutes/month — consumed by walltime only. simulation and memory run on GitHub-hosted runners and consume zero macro minutes (just free public-repo Actions minutes).
• OSS projects can request more macro minutes via contact@codspeed.io.

7. Establishing the baseline

The baseline is the first successful run on the default branch (master), per instrument. PRs-to-master (simulation/memory) and master pushes (all three) then compare against it.

Parked follow-up

Memory CI currently = CodSpeed memory only (per-test heap allocations). Open question: also gate the end-to-end pipeline OOM ceiling (python -m benchmarks memory --phase pipeline), which CodSpeed's per-test run structurally can't capture. Preferred shape: expose the pipeline phase as a CodSpeed-measured benchmark rather than a second workflow. (A self-contained memray --fail-over gate was prototyped but left unmerged.)

[FBumann]

Note

AI-generated (Claude) — CI cadence options and reasoning captured during setup. The decisions below are still the human's to make.

Per-instrument phase assignment

Three instruments, each fits different phases — assign per instrument instead of measuring everything on all three.

• Eligible set (all instruments): build, to_lp, to_solver. Excluded entirely: matrices (a subset of to_solver — both build the constraint matrix, proven by both warming the same label_index cache), netcdf (disk I/O, noisy under walltime), and the pypsa end-to-end test (~30s).
• simulation (cachegrind) trims further to build + to_lp, dropping to_solver (matrix-heavy; cachegrind is ~hundreds× native and under-weights the memory-bandwidth cost of dense-vs-sparse work). to_solver keeps coverage via memory (per-PR) and walltime (master).
• memory / walltime run the full eligible set.

Mechanism: conftest.CODSPEED_SETS + a --codspeed-set {full,simulation} pytest option. Each workflow passes its set (only the simulation job needs --codspeed-set simulation; the rest default to full). The definition lives in Python (testable, single source), not YAML. --codspeed (required by pytest-codspeed) is the signal that gates the narrowing, so smoke/sweep/local runs still execute the full grid.

Cadence (current)

Instrument	Trigger	~runtime	Role
memory	push master + PR + dispatch	~2 min	always-on per-PR sparsity/allocation check
simulation (cachegrind)	push master + PR + dispatch	~13 min	deterministic instruction-count regressions
walltime (macro)	push master + dispatch	~10 min	real wall-clock; on-demand for perf work
smoke (--benchmark-disable)	push master + PR	~3 min	"did a refactor break a spec"

All CodSpeed jobs are continue-on-error (non-gating).

Open decision — simulation cadence

Simulation is the only expensive instrument still on per-PR (~13 min, dominated by the heavy patterns merge_balance / kvl_cycles + pypsa_scigrid under cachegrind). Because it's non-gating, the 13 min never blocks a merge — it's a background check (free CI minutes on a public repo + a delayed dashboard result). Options:

Option	Per-PR cost	Per-PR CPU-regression signal
Leave as-is	~13 min (non-blocking)	full (incl. patterns)
Models-only on PR (patterns → master)	~3–4 min	model specs on PR; patterns on master
Flip to master + dispatch (match walltime)	0 (memory only)	on-demand only

Reasoning: memory already carries the per-PR signal that matters here (allocations); walltime is on-demand for perf work (workflow_dispatch). So per-PR simulation is optional. The trade for dropping it: no per-PR detection of pure-CPU regressions (same memory, slower code) — those would be caught on master or on-demand instead.

Cachegrind nuance (don't over-trim)

Cachegrind isn't useless on the patterns. It deterministically catches algorithmic / element-count regressions (an extra pass, an accidental dense materialization, Python overhead) — including on the sparsity specs. What it under-weights is specifically the memory-bandwidth / allocation cost: allocating a 132 MB array is a handful of instructions, and a cache-layout regression with the same element count is invisible to it. So the three instruments are complementary on patterns, not redundant — cachegrind = instructions, memory = allocations, walltime = bandwidth. Moving patterns off cachegrind is a latency/cost call, not a correctness one.

[FBumann]

Note

AI-generated (Claude) — how CodSpeed baselines behave when benchmarks/ changes.

What invalidates a CodSpeed baseline (and what doesn't)

CodSpeed keys each baseline by (uri, runnerMode), where uri = file::function[params] and runnerMode ∈ {Simulation, Memory, WallTime}. A new measurement is compared against the existing baseline as long as that URI is unchanged. So not every edit under benchmarks/ re-baselines — only ones that change the benchmark identity or the measured set.

Keeps the baseline (compares cleanly) — the normal perf-iteration case:

• Editing the code a benchmark measures — linopy itself, the phase functions (write_lp, …), or a model's build as long as its [params] id is unchanged. Same id, new number → you see the delta. This is the point.
• Refactoring non-test plumbing — conftest internals, cli/, plotting.py, memory.py, docs. URIs don't move.

Re-baselines (new / orphaned ids) — set-changing edits:

• Renaming a test file or function (test_lp_write → test_to_lp) — every URI changes.
• Changing [params]: spec names, sizes or severities (n=10→n=20, severity 5→3), or adding/removing a spec — each changed point is a new URI; the old one is orphaned.
• Narrowing the measured set (--codspeed-set, CODSPEED_MODULES) — dropped modules show as removed for that instrument.

Gotcha — changing what a stable id measures: rewriting e.g. build_merge_balance to construct a different model while keeping [merge_balance-severity=50] does not drop the baseline, but the comparison becomes apples-to-oranges — a one-off jump that's a model change, not a regression. If a spec materially changes what it builds, bump its name/size so the old baseline retires honestly.

Implication for growing coverage: increasing sizes or severity points re-baselines those points (new param ids) — expected; they get a fresh baseline after the next master run.

Practical note: the benchmark-set churn we saw on the dashboard ("🆕 new" / "⏩ skipped" with few "✅ untouched") was the one-time structural reorg (phase rename + severity trim + per-instrument --codspeed-set), not per-edit instability. It self-heals once the set stops changing and a clean master run lands per instrument.

[FBumann]

Note

AI-generated (Claude) — root cause + fix for memory benchmarks disappearing as "skipped".

Bug: separate per-instrument CI jobs overwrite each other

Symptom: memory benchmarks uploaded fine (79 benchmarked … Performance data uploaded) but then vanished from the report as "skipped", leaving only Simulation. Observed on PR #25.

Root cause: CodSpeed records one run per (commit, environment), and a second upload to the same environment supersedes the first — the first instrument's benchmarks then show as "skipped" (their baseline values are reused). We ran simulation and memory as separate jobs on the same ubuntu-latest environment for the same commit; simulation finished last (~15 min) and its upload overwrote memory's (~1.5 min). Whichever instrument uploads last "wins."

Two tempting non-fixes:

• Append to the benchmark keys — doesn't help. The overwrite is at the run level, not the key level; the later upload replaces the measured set regardless of keys (the earlier instrument's benchmarks are simply "not in the latest run").
• Different subsets per instrument (our old --codspeed-set) — unsupported. Instruments sharing a (commit, environment) must measure one consistent set in one upload. The docs only cover serial multi-instrument runs (mode: simulation,memory).

Fix: one upload per environment; trade runtime via cadence, not subsets

Workflow	Trigger	Mode	Env / runner
codspeed-memory.yml	PR (unlabelled)	memory	ubuntu — solo upload, nothing overwrites it
codspeed.yml	master push + PR trigger:benchmark + dispatch	simulation,memory	ubuntu — one combined upload
codspeed-macro.yml	master push + PR trigger:benchmark + dispatch	walltime	macro (separate environment)

Why it's overwrite-safe: every (commit, environment) gets exactly one upload — PR = solo memory; master/labelled-PR = one combined simulation,memory upload on ubuntu plus walltime on the macro runner (a different environment, so no conflict).

Cadence consequences:

• Per-PR is memory-only (fast, the always-on signal). The runtime win that the dropped --codspeed-set simulation-trim used to give now comes from cadence (simulation runs on master/label, not every PR), which is the supported shape.
• simulation + memory baselines are established together on master.

The trigger:benchmark label

workflow_dispatch runs detached from PR context, so CodSpeed can't post a PR comment. The trigger:benchmark label keeps PR context → full suite runs and CodSpeed comments. Mechanics:

• Job gate: if: github.event_name != 'pull_request' || contains(labels.*.name, 'trigger:benchmark'); trigger on pull_request: types: [labeled, synchronize].
• The label auto-removes after the run (a final actions/github-script step) so it can be cleanly re-added to re-trigger (otherwise the label sticks and won't re-fire on the next push).
• Security: codspeed-macro.yml runs on self-hosted bare-metal, so the label is a maintainer-controlled gate — only apply it to trusted (same-repo) PRs, never untrusted forks.

Implemented in e2276dc.