Skip to content

Cap clock-resolution estimation iterations to avoid OOM #3196

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
22elix3r wants to merge 1 commit into from
Closed

Cap clock-resolution estimation iterations to avoid OOM #3196

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions docs/release-notes.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@

# Release notes
**Contents**<br>
[3.16.1](#3161)<br>
[3.16.0](#3160)<br>
[3.15.3](#3153)<br>
[3.15.2](#3152)<br>
Expand Down Expand Up @@ -80,6 +81,12 @@



## 3.16.1

### Fixes
* Fixed benchmark clock-resolution estimation potentially exhausting memory on fast or memory-constrained platforms. (#3180)


## 3.16.0

### Fixes
Expand Down
15 changes: 12 additions & 3 deletions src/catch2/benchmark/detail/catch_estimate_clock.hpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,20 +49,29 @@ namespace Catch {
constexpr auto minimum_ticks = 1000;
constexpr auto warmup_seed = 10000;
constexpr auto clock_resolution_estimation_time = std::chrono::milliseconds(500);
// resolution() allocates O(n) TimePoints; without a sample cap a
// cheap high-resolution clock can exhaust memory within 500ms. (#3180)
constexpr auto clock_resolution_estimation_iteration_limit = 100000;
constexpr auto clock_cost_estimation_time_limit = std::chrono::seconds(1);
constexpr auto clock_cost_estimation_tick_limit = 100000;
constexpr auto clock_cost_estimation_time = std::chrono::milliseconds(10);
constexpr auto clock_cost_estimation_iterations = 10000;

template <typename Clock>
int warmup() {
return run_for_at_least<Clock>(warmup_time, warmup_seed, &resolution<Clock>)
return run_for_at_least<Clock>(warmup_time,
warmup_seed,
&resolution<Clock>,
clock_resolution_estimation_iteration_limit)
.iterations;
}
template <typename Clock>
EnvironmentEstimate estimate_clock_resolution(int iterations) {
auto r = run_for_at_least<Clock>(clock_resolution_estimation_time, iterations, &resolution<Clock>)
.result;
auto r = run_for_at_least<Clock>(clock_resolution_estimation_time,
iterations,
&resolution<Clock>,
clock_resolution_estimation_iteration_limit)
.result;
return {
FDuration(mean(r.data(), r.data() + r.size())),
classify_outliers(r.data(), r.data() + r.size()),
Expand Down
15 changes: 12 additions & 3 deletions src/catch2/benchmark/detail/catch_run_for_at_least.hpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,19 +42,28 @@ namespace Catch {
[[noreturn]]
void throw_optimized_away_error();

constexpr auto run_for_at_least_max_iterations = 1 << 30;

template <typename Clock, typename Fun>
TimingOf<Fun, run_for_at_least_argument_t<Clock, Fun>>
run_for_at_least(IDuration how_long,
const int initial_iterations,
Fun&& fun) {
Fun&& fun,
int max_iterations = run_for_at_least_max_iterations) {
auto iters = initial_iterations;
while (iters < (1 << 30)) {
if (iters > max_iterations) {
iters = max_iterations;
}
while (iters < run_for_at_least_max_iterations) {
auto&& Timing = measure_one<Clock>(fun, iters, is_callable<Fun(Chronometer)>());

if (Timing.elapsed >= how_long) {
if (Timing.elapsed >= how_long || iters >= max_iterations) {
return { Timing.elapsed, CATCH_MOVE(Timing.result), iters };
}
iters *= 2;
if (iters > max_iterations) {
iters = max_iterations;
}
}
throw_optimized_away_error();
}
Expand Down
2 changes: 2 additions & 0 deletions tests/SelfTest/Baselines/automake.sw.approved.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -356,6 +356,7 @@ Message from section two
:test-result: PASS checkedIf
:test-result: FAIL checkedIf, failing
:test-result: PASS classify_outliers
:test-result: PASS clock resolution estimation is iteration-capped
:test-result: PASS comparisons between const int variables
:test-result: PASS comparisons between int variables
:test-result: PASS convertToBits
Expand Down Expand Up @@ -406,6 +407,7 @@ b1!
:test-result: PASS replaceInPlace
:test-result: PASS request an unknown %-starting stream fails
:test-result: PASS resolution
:test-result: PASS run_for_at_least respects max_iterations
:test-result: PASS run_for_at_least, chronometer
:test-result: PASS run_for_at_least, int
:test-result: FAIL second tag
Expand Down
2 changes: 2 additions & 0 deletions tests/SelfTest/Baselines/automake.sw.multi.approved.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -349,6 +349,7 @@
:test-result: PASS checkedIf
:test-result: FAIL checkedIf, failing
:test-result: PASS classify_outliers
:test-result: PASS clock resolution estimation is iteration-capped
:test-result: PASS comparisons between const int variables
:test-result: PASS comparisons between int variables
:test-result: PASS convertToBits
Expand Down Expand Up @@ -395,6 +396,7 @@
:test-result: PASS replaceInPlace
:test-result: PASS request an unknown %-starting stream fails
:test-result: PASS resolution
:test-result: PASS run_for_at_least respects max_iterations
:test-result: PASS run_for_at_least, chronometer
:test-result: PASS run_for_at_least, int
:test-result: FAIL second tag
Expand Down
17 changes: 13 additions & 4 deletions tests/SelfTest/Baselines/compact.sw.approved.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2647,6 +2647,12 @@ InternalBenchmark.tests.cpp:<line number>: passed: o.low_mild == lom for: 0 == 0
InternalBenchmark.tests.cpp:<line number>: passed: o.high_mild == him for: 1 == 1
InternalBenchmark.tests.cpp:<line number>: passed: o.high_severe == his for: 0 == 0
InternalBenchmark.tests.cpp:<line number>: passed: o.total() == los + lom + him + his for: 2 == 2
InternalBenchmark.tests.cpp:<line number>: passed: iterations <= Catch::Benchmark::Detail::clock_resolution_estimation_iteration_limit for: 100000 (0x<hex digits>) <= 100000 (0x<hex digits>)
InternalBenchmark.tests.cpp:<line number>: passed: iterations > 0 for: 100000 (0x<hex digits>) > 0
InternalBenchmark.tests.cpp:<line number>: passed: res.mean.count() == 1 for: 1.0 == 1
InternalBenchmark.tests.cpp:<line number>: passed: res.outliers.total() == 0 for: 0 == 0
InternalBenchmark.tests.cpp:<line number>: passed: res.outliers.samples_seen <= Catch::Benchmark::Detail::clock_resolution_estimation_iteration_limit for: 100000 (0x<hex digits>) <= 100000 (0x<hex digits>)
InternalBenchmark.tests.cpp:<line number>: passed: res.outliers.samples_seen > 0 for: 100000 (0x<hex digits>) > 0
Condition.tests.cpp:<line number>: passed: unsigned_char_var == 1 for: 1 == 1
Condition.tests.cpp:<line number>: passed: unsigned_short_var == 1 for: 1 == 1
Condition.tests.cpp:<line number>: passed: unsigned_int_var == 1 for: 1 == 1
Expand Down Expand Up @@ -2680,6 +2686,7 @@ InternalBenchmark.tests.cpp:<line number>: passed: erfc_inv(0.050000) == Approx(
Approx( 1.38590382434967796 )
InternalBenchmark.tests.cpp:<line number>: passed: res.mean.count() == rate for: 2000.0 == 2000 (0x<hex digits>)
InternalBenchmark.tests.cpp:<line number>: passed: res.outliers.total() == 0 for: 0 == 0
InternalBenchmark.tests.cpp:<line number>: passed: res.outliers.samples_seen <= Catch::Benchmark::Detail::clock_resolution_estimation_iteration_limit for: 100000 (0x<hex digits>) <= 100000 (0x<hex digits>)
Misc.tests.cpp:<line number>: passed:
Misc.tests.cpp:<line number>: passed:
Misc.tests.cpp:<line number>: passed:
Expand Down Expand Up @@ -2819,6 +2826,8 @@ InternalBenchmark.tests.cpp:<line number>: passed: res[i] == rate for: 1000.0 ==
InternalBenchmark.tests.cpp:<line number>: passed: res[i] == rate for: 1000.0 == 1000 (0x<hex digits>)
InternalBenchmark.tests.cpp:<line number>: passed: res[i] == rate for: 1000.0 == 1000 (0x<hex digits>)
InternalBenchmark.tests.cpp:<line number>: passed: res[i] == rate for: 1000.0 == 1000 (0x<hex digits>)
InternalBenchmark.tests.cpp:<line number>: passed: Timing.iterations == 16 for: 16 == 16
InternalBenchmark.tests.cpp:<line number>: passed: Timing.elapsed < time for: 1 ns < 1000000 ns
InternalBenchmark.tests.cpp:<line number>: passed: meter.runs() >= old_runs for: 1 >= 1
InternalBenchmark.tests.cpp:<line number>: passed: meter.runs() >= old_runs for: 2 >= 1
InternalBenchmark.tests.cpp:<line number>: passed: meter.runs() >= old_runs for: 4 >= 2
Expand Down Expand Up @@ -3008,14 +3017,14 @@ Misc.tests.cpp:<line number>: passed: v.size() == 5 for: 5 == 5
Misc.tests.cpp:<line number>: passed: v.capacity() >= 5 for: 5 >= 5
Misc.tests.cpp:<line number>: passed: v.size() == 5 for: 5 == 5
Misc.tests.cpp:<line number>: passed: v.capacity() >= 5 for: 5 >= 5
InternalBenchmark.tests.cpp:<line number>: passed: (iterations * rate) > Catch::Benchmark::Detail::warmup_time.count() for: 160000000 (0x<hex digits>) > 100
InternalBenchmark.tests.cpp:<line number>: passed: (end - start) > Catch::Benchmark::Detail::warmup_time for: 310016000 ns > 100 ms
InternalBenchmark.tests.cpp:<line number>: passed: (iterations * rate) > Catch::Benchmark::Detail::warmup_time.count() for: 100000000 (0x<hex digits>) > 100
InternalBenchmark.tests.cpp:<line number>: passed: (end - start) > Catch::Benchmark::Detail::warmup_time for: 250016000 ns > 100 ms
InternalBenchmark.tests.cpp:<line number>: passed: q1 == 14.5 for: 14.5 == 14.5
InternalBenchmark.tests.cpp:<line number>: passed: med == 18. for: 18.0 == 18.0
InternalBenchmark.tests.cpp:<line number>: passed: q3 == 23. for: 23.0 == 23.0
Misc.tests.cpp:<line number>: passed:
Misc.tests.cpp:<line number>: passed:
test cases: 454 | 334 passed | 96 failed | 6 skipped | 18 failed as expected
assertions: 2423 | 2222 passed | 158 failed | 43 failed as expected
test cases: 456 | 336 passed | 96 failed | 6 skipped | 18 failed as expected
assertions: 2432 | 2231 passed | 158 failed | 43 failed as expected


17 changes: 13 additions & 4 deletions tests/SelfTest/Baselines/compact.sw.multi.approved.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2640,6 +2640,12 @@ InternalBenchmark.tests.cpp:<line number>: passed: o.low_mild == lom for: 0 == 0
InternalBenchmark.tests.cpp:<line number>: passed: o.high_mild == him for: 1 == 1
InternalBenchmark.tests.cpp:<line number>: passed: o.high_severe == his for: 0 == 0
InternalBenchmark.tests.cpp:<line number>: passed: o.total() == los + lom + him + his for: 2 == 2
InternalBenchmark.tests.cpp:<line number>: passed: iterations <= Catch::Benchmark::Detail::clock_resolution_estimation_iteration_limit for: 100000 (0x<hex digits>) <= 100000 (0x<hex digits>)
InternalBenchmark.tests.cpp:<line number>: passed: iterations > 0 for: 100000 (0x<hex digits>) > 0
InternalBenchmark.tests.cpp:<line number>: passed: res.mean.count() == 1 for: 1.0 == 1
InternalBenchmark.tests.cpp:<line number>: passed: res.outliers.total() == 0 for: 0 == 0
InternalBenchmark.tests.cpp:<line number>: passed: res.outliers.samples_seen <= Catch::Benchmark::Detail::clock_resolution_estimation_iteration_limit for: 100000 (0x<hex digits>) <= 100000 (0x<hex digits>)
InternalBenchmark.tests.cpp:<line number>: passed: res.outliers.samples_seen > 0 for: 100000 (0x<hex digits>) > 0
Condition.tests.cpp:<line number>: passed: unsigned_char_var == 1 for: 1 == 1
Condition.tests.cpp:<line number>: passed: unsigned_short_var == 1 for: 1 == 1
Condition.tests.cpp:<line number>: passed: unsigned_int_var == 1 for: 1 == 1
Expand Down Expand Up @@ -2673,6 +2679,7 @@ InternalBenchmark.tests.cpp:<line number>: passed: erfc_inv(0.050000) == Approx(
Approx( 1.38590382434967796 )
InternalBenchmark.tests.cpp:<line number>: passed: res.mean.count() == rate for: 2000.0 == 2000 (0x<hex digits>)
InternalBenchmark.tests.cpp:<line number>: passed: res.outliers.total() == 0 for: 0 == 0
InternalBenchmark.tests.cpp:<line number>: passed: res.outliers.samples_seen <= Catch::Benchmark::Detail::clock_resolution_estimation_iteration_limit for: 100000 (0x<hex digits>) <= 100000 (0x<hex digits>)
Misc.tests.cpp:<line number>: passed:
Misc.tests.cpp:<line number>: passed:
Misc.tests.cpp:<line number>: passed:
Expand Down Expand Up @@ -2808,6 +2815,8 @@ InternalBenchmark.tests.cpp:<line number>: passed: res[i] == rate for: 1000.0 ==
InternalBenchmark.tests.cpp:<line number>: passed: res[i] == rate for: 1000.0 == 1000 (0x<hex digits>)
InternalBenchmark.tests.cpp:<line number>: passed: res[i] == rate for: 1000.0 == 1000 (0x<hex digits>)
InternalBenchmark.tests.cpp:<line number>: passed: res[i] == rate for: 1000.0 == 1000 (0x<hex digits>)
InternalBenchmark.tests.cpp:<line number>: passed: Timing.iterations == 16 for: 16 == 16
InternalBenchmark.tests.cpp:<line number>: passed: Timing.elapsed < time for: 1 ns < 1000000 ns
InternalBenchmark.tests.cpp:<line number>: passed: meter.runs() >= old_runs for: 1 >= 1
InternalBenchmark.tests.cpp:<line number>: passed: meter.runs() >= old_runs for: 2 >= 1
InternalBenchmark.tests.cpp:<line number>: passed: meter.runs() >= old_runs for: 4 >= 2
Expand Down Expand Up @@ -2997,14 +3006,14 @@ Misc.tests.cpp:<line number>: passed: v.size() == 5 for: 5 == 5
Misc.tests.cpp:<line number>: passed: v.capacity() >= 5 for: 5 >= 5
Misc.tests.cpp:<line number>: passed: v.size() == 5 for: 5 == 5
Misc.tests.cpp:<line number>: passed: v.capacity() >= 5 for: 5 >= 5
InternalBenchmark.tests.cpp:<line number>: passed: (iterations * rate) > Catch::Benchmark::Detail::warmup_time.count() for: 160000000 (0x<hex digits>) > 100
InternalBenchmark.tests.cpp:<line number>: passed: (end - start) > Catch::Benchmark::Detail::warmup_time for: 310016000 ns > 100 ms
InternalBenchmark.tests.cpp:<line number>: passed: (iterations * rate) > Catch::Benchmark::Detail::warmup_time.count() for: 100000000 (0x<hex digits>) > 100
InternalBenchmark.tests.cpp:<line number>: passed: (end - start) > Catch::Benchmark::Detail::warmup_time for: 250016000 ns > 100 ms
InternalBenchmark.tests.cpp:<line number>: passed: q1 == 14.5 for: 14.5 == 14.5
InternalBenchmark.tests.cpp:<line number>: passed: med == 18. for: 18.0 == 18.0
InternalBenchmark.tests.cpp:<line number>: passed: q3 == 23. for: 23.0 == 23.0
Misc.tests.cpp:<line number>: passed:
Misc.tests.cpp:<line number>: passed:
test cases: 454 | 334 passed | 96 failed | 6 skipped | 18 failed as expected
assertions: 2423 | 2222 passed | 158 failed | 43 failed as expected
test cases: 456 | 336 passed | 96 failed | 6 skipped | 18 failed as expected
assertions: 2432 | 2231 passed | 158 failed | 43 failed as expected


4 changes: 2 additions & 2 deletions tests/SelfTest/Baselines/console.std.approved.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1743,6 +1743,6 @@ due to unexpected exception with message:
Why would you throw a std::string?

===============================================================================
test cases: 454 | 352 passed | 76 failed | 7 skipped | 19 failed as expected
assertions: 2401 | 2222 passed | 136 failed | 43 failed as expected
test cases: 456 | 354 passed | 76 failed | 7 skipped | 19 failed as expected
assertions: 2410 | 2231 passed | 136 failed | 43 failed as expected

73 changes: 69 additions & 4 deletions tests/SelfTest/Baselines/console.sw.approved.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17580,6 +17580,50 @@ InternalBenchmark.tests.cpp:<line number>: PASSED:
with expansion:
2 == 2

-------------------------------------------------------------------------------
clock resolution estimation is iteration-capped
warmup
-------------------------------------------------------------------------------
InternalBenchmark.tests.cpp:<line number>
...............................................................................

InternalBenchmark.tests.cpp:<line number>: PASSED:
REQUIRE( iterations <= Catch::Benchmark::Detail::clock_resolution_estimation_iteration_limit )
with expansion:
100000 (0x<hex digits>) <= 100000 (0x<hex digits>)

InternalBenchmark.tests.cpp:<line number>: PASSED:
REQUIRE( iterations > 0 )
with expansion:
100000 (0x<hex digits>) > 0

-------------------------------------------------------------------------------
clock resolution estimation is iteration-capped
estimate_clock_resolution
-------------------------------------------------------------------------------
InternalBenchmark.tests.cpp:<line number>
...............................................................................

InternalBenchmark.tests.cpp:<line number>: PASSED:
REQUIRE( res.mean.count() == 1 )
with expansion:
1.0 == 1

InternalBenchmark.tests.cpp:<line number>: PASSED:
REQUIRE( res.outliers.total() == 0 )
with expansion:
0 == 0

InternalBenchmark.tests.cpp:<line number>: PASSED:
REQUIRE( res.outliers.samples_seen <= Catch::Benchmark::Detail::clock_resolution_estimation_iteration_limit )
with expansion:
100000 (0x<hex digits>) <= 100000 (0x<hex digits>)

InternalBenchmark.tests.cpp:<line number>: PASSED:
REQUIRE( res.outliers.samples_seen > 0 )
with expansion:
100000 (0x<hex digits>) > 0

-------------------------------------------------------------------------------
comparisons between const int variables
-------------------------------------------------------------------------------
Expand Down Expand Up @@ -17752,6 +17796,11 @@ InternalBenchmark.tests.cpp:<line number>: PASSED:
with expansion:
0 == 0

InternalBenchmark.tests.cpp:<line number>: PASSED:
REQUIRE( res.outliers.samples_seen <= Catch::Benchmark::Detail::clock_resolution_estimation_iteration_limit )
with expansion:
100000 (0x<hex digits>) <= 100000 (0x<hex digits>)

-------------------------------------------------------------------------------
even more nested SECTION tests
c
Expand Down Expand Up @@ -18853,6 +18902,22 @@ InternalBenchmark.tests.cpp:<line number>: PASSED:
with expansion:
1000.0 == 1000 (0x<hex digits>)

-------------------------------------------------------------------------------
run_for_at_least respects max_iterations
-------------------------------------------------------------------------------
InternalBenchmark.tests.cpp:<line number>
...............................................................................

InternalBenchmark.tests.cpp:<line number>: PASSED:
REQUIRE( Timing.iterations == 16 )
with expansion:
16 == 16

InternalBenchmark.tests.cpp:<line number>: PASSED:
REQUIRE( Timing.elapsed < time )
with expansion:
1 ns < 1000000 ns

-------------------------------------------------------------------------------
run_for_at_least, chronometer
-------------------------------------------------------------------------------
Expand Down Expand Up @@ -20177,12 +20242,12 @@ InternalBenchmark.tests.cpp:<line number>
InternalBenchmark.tests.cpp:<line number>: PASSED:
REQUIRE( (iterations * rate) > Catch::Benchmark::Detail::warmup_time.count() )
with expansion:
160000000 (0x<hex digits>) > 100
100000000 (0x<hex digits>) > 100

InternalBenchmark.tests.cpp:<line number>: PASSED:
REQUIRE( (end - start) > Catch::Benchmark::Detail::warmup_time )
with expansion:
310016000 ns > 100 ms
250016000 ns > 100 ms

-------------------------------------------------------------------------------
weighted_average_quantile
Expand Down Expand Up @@ -20226,6 +20291,6 @@ Misc.tests.cpp:<line number>
Misc.tests.cpp:<line number>: PASSED:

===============================================================================
test cases: 454 | 334 passed | 96 failed | 6 skipped | 18 failed as expected
assertions: 2423 | 2222 passed | 158 failed | 43 failed as expected
test cases: 456 | 336 passed | 96 failed | 6 skipped | 18 failed as expected
assertions: 2432 | 2231 passed | 158 failed | 43 failed as expected

Loading