Chart performance per release, and show it in the README - #89
Merged
Merged
Conversation
Adds a BenchmarkDotNet suite covering arithmetic, comparison, significance reduction, parsing and conversion, and a workflow that measures a fixed subset of it once per release, appends the numbers to a committed history file, and redraws the chart the README shows. Every point is measured in one job, because separate CI runs land on different hosts and that difference is larger than most releases are. Across jobs, BaselineBenchmarks ties them together: a fixed reference loop measured beside the benchmarks, which the chart divides the times by. Allocation needs none of that -- the same code allocates the same bytes anywhere -- so the chart reads allocation as exact and time as indicative. Backfills measure published packages through BenchmarkAgainstVersion, since no tag before this commit carries a benchmark project. It has to reach the build through the environment rather than a -p: switch: BenchmarkDotNet generates a project of its own per run, which a property on the outer command line never reaches. Releases from here on are measured from their own tag instead, which also keeps that path off the race against the NuGet push beside it. Two versions cannot be measured by this suite at all. 1.2.2 and 1.2.7 throw NotSupportedException from Parse, which is how every operand here is built, so they compile, run, and report a table of NA. Ingest refuses a run with no measurement in it rather than recording a release with nothing under it, and the backfill treats that as a skip. Significance reduction is spelled three ways across the published versions, so its file is left out of builds against anything older than 1.4.20 and the chart draws that panel with a gap, rather than losing six panels to a compile error. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017jrnV7N94UGL8fDRRE8Xt8
Six releases, 1.3.0 through 2.0.1, all measured in one pass against a single reference reading so that the times are comparable as they stand. Without this the README would show an empty chart until the next release, and a one-point chart after it. Also adds the benchmark project to the solution, which the first commit left out: `dotnet build` at the root built the library and the tests and never noticed the benchmarks were not compiling. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017jrnV7N94UGL8fDRRE8Xt8
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a BenchmarkDotNet suite, a workflow that measures a fixed subset of it once per release, and a chart in the README built from the accumulated numbers. Seeded with six releases so it says something the day it merges rather than after the next one.
What is measured, and how it is made comparable
Allocation and time are read differently, and the chart says so.
Allocation is exact. The same code allocates the same bytes on any machine, so a step in the top row is always a real change.
Time is not, because these run on shared CI hosts and the difference between an x86-64-v3 and a v4 runner is larger than most releases are.
BaselineBenchmarks.ReferenceWorkis a fixed integer loop measured beside the benchmarks in the same job; the chart divides every time by it. That cancels most of the host difference and leaves something indicative. Its body must never change — editing it rescales the whole history.A backfill measures every version in one job for the same reason: within a job the points are comparable as they stand.
Measuring a release
Backfills measure the published packages, through
BenchmarkAgainstVersion. No tag before this commit carries a benchmark project, so there is no older source to check out — and measuring packages is the better comparison anyway, since every version is then timed by identical benchmark code rather than by whatever each tag shipped.That variable has to reach the build through the environment, not
-p:. BenchmarkDotNet generates and builds a project of its own per run, which a property on the outer command line never reaches; it would build the benchmark assembly against the version asked for and the harness against the one pinned centrally, which does not compile when the type changed shape between them. MSBuild reads environment variables as properties in every project, so the environment form reaches both.Releases from here on are measured from their own tag, through a worktree. That keeps the release path off the race against the NuGet push happening beside it, which the package route would otherwise lose about as often as it won.
What the older packages cannot be asked
The history starts at 1.3.0, and one panel of it starts at 1.4.20.
Every operand is built by parsing text, which is the only construction route all the published versions share. 1.2.2 and 1.2.7 throw
NotSupportedExceptionfromParse, so they compile against these benchmarks, run, and report a table ofNA. That is not a build failure, so it got as far as being recorded as a release with nothing under it — a point on the axis reading zero, which is the opposite of the truth.ingestnow refuses a run with no measurement in it, the backfill treats that as a skip, and the two versions are off the default list.Significance reduction is spelled three ways across the versions: 1.2.x has no precision overload of
ToSignificantNumberat all, and 1.3 and 1.4.0 hang it off thePreciseNumberbase, where the receiver's type argument has to be written out. Carrying three spellings would measure the spellings, so the file is left out of builds against anything older than 1.4.20 and the chart draws that one panel with a gap. Everything else in those versions is still measured — the alternative was losing six panels to a compile error.The result
The 2.0 step is the type becoming a
readonly record structoverPreciseNumberinstead of deriving from it:AddCompareToreaches 0 B.Verification
dotnet build -c Release— clean, 0 warnings. The benchmark project was missing from the solution in the first commit, so the root build was not compiling it at all; that is fixed here.dotnet test -c Release— 95 passed, 0 failed.🤖 Generated with Claude Code
https://claude.ai/code/session_017jrnV7N94UGL8fDRRE8Xt8
Generated by Claude Code