test(bandwidth): widen TestMeasureExcludesWarmup's margin to 2x - #3
Merged
Conversation
CI failed on main at 2.965x against a 3x bar. Nothing regressed: on this host the ratio runs 3.19-3.27 unloaded, so 3x was asserting ~94% of what this test's own shape produces, and ordinary runner jitter closes that gap. The ratio is wall clock over steady window. Bytes flow only in the bulk phase -- ~0.67s of a ~2.17s transfer -- and under contention that window stretches faster than the total does, so the ratio falls as the runner slows. Measured here under cgroup CPU limits, 4 runs each: 8 cores 3.19 - 3.25 1.0 cpu 3.24 - 3.27 0.5 cpu 3.17 - 3.23 0.25 cpu 2.40 - 2.73 <- 3x fails from here down 0.15 cpu 1.65 - 1.68 <- 2x fails from here down 2x keeps the property the test exists for. If warmup stopped being excluded, BytesPerSecond would BE the warmup-inclusive rate and the ratio would be 1. Simulating exactly that regression (steadyFrom = start) gives 1.001x, which fails the 2x bar as it should. 2x is a wider margin, not a guarantee: a runner below ~0.2 CPU still falls under it. No fixed multiplier can be robust here, because the ratio degrades continuously toward 1 as the machine slows. Deriving the bound from the measured window would be the structural fix; this change buys about twice the contention headroom without one. Full suite green under the CI command (go test -race -count=1 -timeout 20m ./...); bandwidth alone 6/6 consecutive -race runs.
Ryanmello07
force-pushed
the
fix/bandwidth-warmup-test-margin
branch
from
August 10, 2026 23:20
0d5cc86 to
5328891
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the
mainCI failure in run 31435713876.2560304 × 3 = 7680912 against a measured 7590822 — 2.965×, short by 1.2%.
Not a regression, but it is ours
Nothing broke. The assertion is simply sitting on top of what the test's own shape produces.
The ratio being asserted is wall clock over steady window. Bytes flow only in the bulk phase — about 0.67s of a ~2.17s transfer — so an unloaded run lands at 3.19–3.27×. A 3× bar is asking for ~94% of the typical value, which ordinary runner jitter closes. (
MaxSteadyInflationcaps the ratio at 4× above that, but the shape binds first.)To be clear about provenance, since I got this wrong on the first draft of this PR: this is fallout from #2. Three commits in that merge set the situation up —
d7fbf2f3*naivebar2aa8fa4start+WarmupDurationrather than at the first read past it, widening the window and lowering the ratio0dc0e3eMinSteadyDurationfloor with theMaxSteadyInflationratio boundBoth
2aa8fa4and0dc0e3ewere the right calls — the old absolute floor only rejected narrow tails and left a residual bound of ~9× at the default timeout. They just moved the ceiling down next to a bar written when there was headroom above it. (What did not cause this is the sharding work, which touched onlycmd/egress-prober/andingest/; that was PR #2 on the fork, a different repo's numbering, and I conflated the two.)Where the ratio actually goes
Measured on an 8-core host under cgroup CPU limits,
-race, 4 runs per row:The mechanism is visible in the raw numbers: from 8 cores to 0.15 CPU the steady window stretches 0.67s → 2.32s while wall clock only goes 2.17s → 3.82s. The window grows faster than the total, so the ratio falls.
Why 2× still has teeth
The property under test is that warmup is excluded. If it stopped being excluded,
BytesPerSecondwould be the warmup-inclusive rate and the ratio would be 1.Verified rather than assumed — I simulated exactly that regression by replacing
steadyFrom = start.Add(WarmupDuration)withsteadyFrom = start:Ratio 1.001×, fails as it should. Then I restored the file.
What this does not fix
2× is a wider margin, not a guarantee. A runner below ~0.2 CPU still falls under it, as the table shows. No fixed multiplier can be robust here, because the ratio degrades continuously toward 1 as the machine slows — there is no clean separation between "slow runner" and "broken code" on this axis, only more or less headroom.
The structural fix is to derive the bound from
sample.Elapsed(the window the implementation actually reports) instead of a hard multiple of the naive rate, which would make the assertion independent of machine speed. That is a larger change to someone else's test design, so I have left it out and noted it in the comment rather than doing it unilaterally. Happy to follow up if you'd prefer that shape.Verification
go test -race -count=1 -timeout 20m ./...(the CI command): all 8 packages ok.bandwidthalone: 6/6 consecutive-raceruns, plus the CPU-limited sweep above.gofmt -l bandwidth/clean.Test-only change; no non-test file is touched.