[Nexthop] benchmark --filter/--filter_file accept the gtest filter grammar - #1434
Open
saty-nexthop wants to merge 1 commit into
Open
[Nexthop] benchmark --filter/--filter_file accept the gtest filter grammar#1434saty-nexthop wants to merge 1 commit into
saty-nexthop wants to merge 1 commit into
Conversation
saty-nexthop
force-pushed
the
benchmark-filter-gtest-grammar
branch
from
July 29, 2026 08:00
6ebda9d to
8ff42c9
Compare
Benchmarks could not be filtered the way every other test runner is.
--filter took a single regex and --filter_file took exact benchmark
names, so the wildcard patterns the runners write into a filter file
("*StatsCollection*") matched nothing.
Teach _get_benchmarks_to_run the gtest grammar, mirroring how
test_runner.py feeds --gtest_filter: join filter-file lines with ':',
split on the first '-' into includes and excludes, and expand '*'/'?'
wildcards. Wildcards already part of a regex are left alone so plain
regexes keep their meaning -- naively rewriting '*' turns ".*Route.*"
into "..*Route..*", which stops matching any name beginning with Route.
--filter_file wins when both are given, as on the gtest path.
Also report a run whose benchmarks are all known-bad or unsupported on
this platform as skipped rather than returning with no results file.
Callers decide "did this run produce results?" by looking for the CSV,
so the old early return looked like an infra failure instead of a
platform gap. This never triggered while a single test ran every
benchmark, but it does as soon as callers filter down to one family.
Test Plan: 275 unit tests pass (fboss_test_runner/unittests).
saty-nexthop
force-pushed
the
benchmark-filter-gtest-grammar
branch
from
July 29, 2026 08:11
8ff42c9 to
5dcf6cd
Compare
Contributor
|
@AarjunC has imported this pull request. If you are a Meta employee, you can view this in D114089843. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pre-submission checklist
pip install -r requirements-dev.txt && pre-commit installpre-commit runSummary
benchmark --filterand--filter_filedidn't take the filter syntax every other test runner takes.--filterwas a single regex and--filter_filematched exact benchmark names, so when a runner wrote*StatsCollection*into a filter file it matched nothing.Both now take the gtest grammar:
include:include:-exclude, with*and?wildcards. Same thingtest_runner.pydoes when it builds--gtest_filter. Plain regexes still work, because*only gets expanded where it isn't already part of a regex. Rewriting it blindly turns.*Route.*into..*Route..*, which then matches no name starting with Route. If both flags are given,--filter_filewins, same as on the gtest path.Second thing, while I was in here: a run where every benchmark is known-bad or unsupported on the platform used to return early and write no results CSV. Callers look for that CSV to decide whether the run produced anything, so this read as an infra failure instead of "this platform doesn't have these benchmarks". It now writes the CSV and reports the skips. It never came up while a single test ran every benchmark, but it does as soon as callers filter down to one family.
Test Plan
275 unit tests pass in
fboss/oss/scripts/run_scripts/fboss_test_runner/unittests, covering includes, excludes, multiple patterns,?, plain regexes, invalid patterns, missing and empty filter files, and the all-unsupported skip path.