Skip to content

test(bandwidth): widen TestMeasureExcludesWarmup's margin to 2x - #3

Merged
Ryanmello07 merged 1 commit into
mainfrom
fix/bandwidth-warmup-test-margin
Aug 11, 2026
Merged

test(bandwidth): widen TestMeasureExcludesWarmup's margin to 2x#3
Ryanmello07 merged 1 commit into
mainfrom
fix/bandwidth-warmup-test-margin

Conversation

@Ryanmello07

@Ryanmello07 Ryanmello07 commented Aug 10, 2026

Copy link
Copy Markdown
Collaborator

Fixes the main CI failure in run 31435713876.

--- FAIL: TestMeasureExcludesWarmup (2.14s)
    bandwidth_test.go:214: BytesPerSecond = 7590822, want at least 3x
        the warmup-inclusive rate 2560304

2560304 × 3 = 7680912 against a measured 7590822 — 2.965×, short by 1.2%.

Not a regression, but it is ours

Nothing broke. The assertion is simply sitting on top of what the test's own shape produces.

The ratio being asserted is wall clock over steady window. Bytes flow only in the bulk phase — about 0.67s of a ~2.17s transfer — so an unloaded run lands at 3.19–3.27×. A 3× bar is asking for ~94% of the typical value, which ordinary runner jitter closes. (MaxSteadyInflation caps the ratio at 4× above that, but the shape binds first.)

To be clear about provenance, since I got this wrong on the first draft of this PR: this is fallout from #2. Three commits in that merge set the situation up —

commit change
d7fbf2f introduced the test with the 3*naive bar
2aa8fa4 made the steady window start at start+WarmupDuration rather than at the first read past it, widening the window and lowering the ratio
0dc0e3e replaced the absolute MinSteadyDuration floor with the MaxSteadyInflation ratio bound

Both 2aa8fa4 and 0dc0e3e were the right calls — the old absolute floor only rejected narrow tails and left a residual bound of ~9× at the default timeout. They just moved the ceiling down next to a bar written when there was headroom above it. (What did not cause this is the sharding work, which touched only cmd/egress-prober/ and ingest/; that was PR #2 on the fork, a different repo's numbering, and I conflated the two.)

Where the ratio actually goes

Measured on an 8-core host under cgroup CPU limits, -race, 4 runs per row:

CPU limit ratio
8 cores 3.19 – 3.25 pass pass
1.0 3.24 – 3.27 pass pass
0.5 3.17 – 3.23 pass (barely) pass
0.25 2.40 – 2.73 fail pass
0.15 1.65 – 1.68 fail fail

The mechanism is visible in the raw numbers: from 8 cores to 0.15 CPU the steady window stretches 0.67s → 2.32s while wall clock only goes 2.17s → 3.82s. The window grows faster than the total, so the ratio falls.

Why 2× still has teeth

The property under test is that warmup is excluded. If it stopped being excluded, BytesPerSecond would be the warmup-inclusive rate and the ratio would be 1.

Verified rather than assumed — I simulated exactly that regression by replacing steadyFrom = start.Add(WarmupDuration) with steadyFrom = start:

bandwidth_test.go:214: BytesPerSecond = 7744746, want at least 2x
    the warmup-inclusive rate 7736771

Ratio 1.001×, fails as it should. Then I restored the file.

What this does not fix

2× is a wider margin, not a guarantee. A runner below ~0.2 CPU still falls under it, as the table shows. No fixed multiplier can be robust here, because the ratio degrades continuously toward 1 as the machine slows — there is no clean separation between "slow runner" and "broken code" on this axis, only more or less headroom.

The structural fix is to derive the bound from sample.Elapsed (the window the implementation actually reports) instead of a hard multiple of the naive rate, which would make the assertion independent of machine speed. That is a larger change to someone else's test design, so I have left it out and noted it in the comment rather than doing it unilaterally. Happy to follow up if you'd prefer that shape.

Verification

  • go test -race -count=1 -timeout 20m ./... (the CI command): all 8 packages ok.
  • bandwidth alone: 6/6 consecutive -race runs, plus the CPU-limited sweep above.
  • gofmt -l bandwidth/ clean.

Test-only change; no non-test file is touched.

CI failed on main at 2.965x against a 3x bar. Nothing regressed: on this
host the ratio runs 3.19-3.27 unloaded, so 3x was asserting ~94% of what
this test's own shape produces, and ordinary runner jitter closes that gap.

The ratio is wall clock over steady window. Bytes flow only in the bulk
phase -- ~0.67s of a ~2.17s transfer -- and under contention that window
stretches faster than the total does, so the ratio falls as the runner
slows. Measured here under cgroup CPU limits, 4 runs each:

  8 cores    3.19 - 3.25
  1.0 cpu    3.24 - 3.27
  0.5 cpu    3.17 - 3.23
  0.25 cpu   2.40 - 2.73   <- 3x fails from here down
  0.15 cpu   1.65 - 1.68   <- 2x fails from here down

2x keeps the property the test exists for. If warmup stopped being
excluded, BytesPerSecond would BE the warmup-inclusive rate and the ratio
would be 1. Simulating exactly that regression (steadyFrom = start) gives
1.001x, which fails the 2x bar as it should.

2x is a wider margin, not a guarantee: a runner below ~0.2 CPU still falls
under it. No fixed multiplier can be robust here, because the ratio
degrades continuously toward 1 as the machine slows. Deriving the bound
from the measured window would be the structural fix; this change buys
about twice the contention headroom without one.

Full suite green under the CI command (go test -race -count=1 -timeout 20m
./...); bandwidth alone 6/6 consecutive -race runs.
@Ryanmello07
Ryanmello07 force-pushed the fix/bandwidth-warmup-test-margin branch from 0d5cc86 to 5328891 Compare August 10, 2026 23:20
@Ryanmello07 Ryanmello07 changed the title test(bandwidth): fix flaky TestMeasureExcludesWarmup — assertion sits at 90% of its ceiling test(bandwidth): widen TestMeasureExcludesWarmup's margin to 2x Aug 10, 2026
@Ryanmello07
Ryanmello07 merged commit 70b9e38 into main Aug 11, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant