Conversation
How to design and read an extension E2E benchmark run, how to report a result from several rounds, how per-iteration metrics flow from collector to reporter, and how Web Vitals are collected in benchmarks versus in production.
Where the browser delivers no `longtask` entries, the observer's counts stay at their initial 0, so a zero reads the same whether no task ran long or none could be seen.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Nothing in the repo says what a benchmark difference means.
performance-testingruns the benchmark. How many rounds, how much variance, and whether a 4% gap is a result are all unanswered. Without that a benchmark reports noise as a win, which is worse than not running one, because it produces a number people cite.Web Vitals numbers from a benchmark and from production routinely disagree, and the disagreement is a property of how each is collected rather than a regression.
Overview
Adds one skill and four knowledge files for E2E performance benchmarks. The profiling, render-delta and data-analysis skills are in #144.
benchmark-designcovers session hygiene, reporting across rounds, and artifact grouping.Knowledge:
benchmark-statistical-hygiene(one primary analysis fixed before the runs, isolating the fix vector, the artifact sort-order trap),metrics-pipeline-design(the collector, runner, statistics and reporter layers), and two on Web Vitals: runtime metrics, and why production numbers disagree with benchmark numbers.Showcase
No trial run yet. These are measurement instruments, and the honest evidence for one is a run that moved a decision. There is not one to cite.