benchmarks: add a benchmarks/ directory with the warm-TT and recorded-workload benchmarks

[ThorvaldAagaard]

What

Creates benchmarks/ and puts two benchmarks in it. No library or API changes.

target	measures	runs in CI
//benchmarks:warm_tt_benchmark	transposition-table reuse in isolation, on a synthetic opening-lead workload	yes
//benchmarks:replay_test	smoke test over a 6-call recording, so the replay path stays correct	yes
//benchmarks:dds_replay	a real client's entire DDS workload, replayed and verified	no (manual)

This started as the warm-TT test alone, in library/tests/solve_board/. Per @tameware's review it moved to a benchmark directory, and since I was building a recorded-workload benchmark anyway, both now live here. The two measure the same effect from opposite ends: one isolates TT reuse synthetically, the other shows what it is worth inside a real engine's traffic.

---

1. Warm-transposition-table benchmark

api/dds_c_api.h hands out an opaque solver-context handle that owns the transposition table, and the docs note a context can be reused across solves — but nothing in the tree quantified the benefit. This puts a number on it, on the workload where it matters most: opening-lead evaluation.

Why this workload. Two ways to value the opening leads of a deal:

• A — one solve with solutions=3, which scores all 13 leads at once.
• B — prune to the top-K leads (an engine would use a NN shortlist; the benchmark just takes the K best from A), play each, and solve the resulting position with solutions=1.

B only competes if the TT survives between the K post-lead positions. Those differ by only a couple of cards, so they pass the solver's similar-deal gate and the TT is not reset — but that only helps if the caller keeps the same context. So B is measured both ways: B_cold (a fresh context per lead) and B_warm (one context for all K).

Results — 100 deterministic deals × {3N, 4S}, -c opt, declarer South / leader West:

K	B_cold	B_warm	warm/cold
4	0.93× A	0.54× A	0.58
6	1.34× A	0.71× A	0.53

Context reuse roughly halves the post-lead solve cost. At K=6 it flips the top-K approach from a 1.34× loss against solutions=3 into a 0.71× win.

What is asserted. Correctness only: every solve returns RETURN_NO_FAULT and at least one card, and warm and cold agree on every lead's score across all deals — so TT reuse provably never changes the answer. That check is arguably the more valuable half: it is a regression guard on context reuse. Timings are printed, not asserted, so it cannot flake under CI load; the one timing check is a loose sanity bound, not a performance gate.

Deals come from a fixed-seed mt19937, so every run on every machine measures the same 100 deals. Binds only to the exported C ABI (//library/src/api:dds_c_api), not testable_dds.

2. Recorded-workload benchmark

dds_replay re-issues every DDS call a real client made, in the order it made them, times them, and checks the answers still match what was recorded.

Most solver benchmarks pick a set of deals and solve them — which measures the deals someone chose, not the work a client asks for. A bridge engine's traffic is dominated by things a hand-picked deal list does not reproduce: batches of sampled hands rather than single deals, a distribution of trick depths (most calls happen mid-play, not at trick 1), a mix of solutions=1 and solutions=3, and repeated calls on nearly identical positions that hit a warm TT. Replaying a recording keeps all of it.

Because the recording stores the result of every call, the replay also verifies them — so this is a regression test as well as a benchmark, over 154,370 solved deals.

               calls    boards   seconds    ms/call   ms/board  rec ms/call
---------------------------------------------------------------------------
bid              962     50335    99.062     102.97      1.968       128.62
play            1226     73653    22.720      18.53      0.308        22.54
lead             128     17680    13.858     108.27      0.784       117.91
par               32        32     3.760     117.49    117.491       320.39
claimcheck       392     12702     1.135       2.90      0.089         4.20
---------------------------------------------------------------------------
TOTAL           2740    154402   140.534      51.29      0.910        65.09

VERIFY: all 2740 calls returned the recorded result

The committed workload — please read this bit. testdata/dds-camrose-1-32.jsonl is 11.7 MB, the largest thing in this PR by far. It is every DDS call made while BEN bid and played the 32 boards of the Camrose Trophy 2024, all four seats. The deals are from the tournament; the play is the engine's.

It is committed rather than generated because it cannot be regenerated from this repository — producing it needs BEN and its trained neural models. Committing it is what makes the benchmark runnable and comparable across machines and over time, but I appreciate 11.7 MB is a real cost and I am happy to drop it and have dds_replay require a recording the user supplies, if you would rather keep it out of the tree.

testdata/sample-recording.jsonl is a 6-call excerpt backing //benchmarks:replay_test, so CI exercises the replay path (JSON parsing, PBN decoding, batching, canonicalisation, verification) without spending two minutes on the full workload.

---

Review fixes in this PR

• @zzcgumn — make_deal() is only as deterministic as the generator handed to it; the explanation of why the seed is fixed moved to the seeding site.
• Copilot — solver contexts are now held by a ScopedContext RAII guard, so a failing ASSERT_* (which expands to a bare return) cannot leak the handle. The guard keeps an explicit destroy() so the cost of destruction still lands inside the timed region.
• Copilot — a zero-card result is now a failure rather than a skipped deal; skipping silently dropped it from the warm/cold comparison and skewed the reported deal count. Every FutureTricks is value-initialised and cards > 0 is asserted before reading score[0].
• Copilot — depends only on //library/src/api:dds_c_api, which already pulls in //library/src:dds. Same redundancy dropped from replay_lib.

On @tameware's questions

What does failure mean? If pass and fail are meaningful in a CI context I'd like to see it run there.

They are, so it stays a cc_test rather than becoming a manual target. warm_tt_benchmark fails if warm and cold contexts ever disagree on a lead's score; replay_test fails if the replay path mis-parses or mis-canonicalises a result. Linux and Windows CI both run bazel test //..., so both are picked up automatically. Only dds_replay is tagged manual — it is a measuring instrument, not a test.

On the ~45s runtime: warm_tt_benchmark is sized medium, and replay_test is small and takes ~3s. Still happy to cut the deal count or drop to a single contract if 45s is too much for the default cycle — say the word.

Verification

bazel test //benchmarks:all //library/tests/solve_board:all passes on Windows/MSVC (4/4).

[zzcgumn]

This generator is not necessarily deterministic. What makes the test deterministic is the fixed seed inside the actual test case. I would prefer to have a comment explaining why the seed is fixed inside the test case.

[zzcgumn]

This makes sense to me. Would prefer to comment about how deals are generated deterministically to be moved.

[ThorvaldAagaard]

Thanks for the review — and good catch, you're right.

make_deal() isn't deterministic in itself; it's only as reproducible as the generator handed to it. The guarantee actually comes from the fixed seed in the test case, so I've moved the explanation there (6607c1c):

• make_deal() now just says it's as reproducible as the caller's rng, and points at the seeding site.
• The seed itself now explains why it's fixed: std::mt19937 is specified by the standard, so a given seed yields the same sequence on every platform and library version — every run measures the same 100 deals. That matters because these numbers are meant to be compared across runs and across solver changes; resampling each run would move the timings for reasons unrelated to the code under test, and would make any failure of the warm/cold agreement check impossible to reproduce from the failure output alone.

Rebuilt and re-ran after the change: still passes.

Happy to also address the sizing point from the description if you'd like — it runs ~45s, so if that's too slow for the default cycle I can cut the deal count, drop to a single contract, or tag it manual. Just say which you'd prefer.

[tameware]

I agree that 45 seconds is too long for the default test cycle. It's not exactly a test, in that it doesn't pass or fail. We could add a benchmark directory and move it there, along with benchmark.py and its wrapper, benchmark.sh.
XXX
Scratch that, I see it passes, in 19 seconds on my Mac. What does failure mean? If pass and fail are meaningful in a CI context I'd like to see it run there.

[Copilot]

Pull request overview

Adds a new GoogleTest benchmark/regression test to quantify (and guard) the performance benefit of reusing a DDS_C_SOLVER_CTX so the transposition table stays warm across successive similar solves, specifically on an opening-lead evaluation workload via the public C ABI.

Changes:

• Introduces a new benchmark-style test that compares solutions=3 vs “top-K then solve after lead” in cold-context vs warm-context modes, while asserting warm/cold correctness equivalence.
• Registers the new test as a Bazel cc_test target (sized medium) under library/tests/solve_board.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

File	Description
library/tests/solve_board/context_warm_tt_benchmark.cpp	New benchmark/regression test measuring warm-TT context reuse benefit and asserting warm/cold result equality.
library/tests/solve_board/BUILD.bazel	Adds a cc_test target for the new benchmark.

Comments suppressed due to low confidence (1)

library/tests/solve_board/context_warm_tt_benchmark.cpp:153

• B_cold reads fut.score[0] without checking that the solver returned at least one card. If FutureTricks::cards ever ends up 0 (e.g., due to an unexpected solver condition), this becomes an out-of-bounds read. Assert cards>0 (and value-initialize FutureTricks) before indexing.

      FutureTricks fut;
      const int rc = dds_c_solve_board(ctx, &pos, -1, 1, 0, &fut);
      dds_c_destroy_solvercontext(ctx);
      ASSERT_EQ(RETURN_NO_FAULT, rc) << label << " deal " << d << ": B_cold solve failed";
      cold_scores.push_back(fut.score[0]);

[ThorvaldAagaard]

I am creating a benchmark test for robot bidding and play, so I can move this to a new benchmark folder

[tameware]

How does this technique relate to the internal context reuse in PR #262?

[ThorvaldAagaard]

My new benchmark is a full capture of all DDS-calls from BEN from a 32 boards play. It will use the principles from PR #262

[tameware]

How does this technique relate to the internal context reuse in PR #262?

To partially answer my own question, I created a branch with both PRs and ran context_warm_tt_benchmark multiple times, averaging the results. There was no difference greater than a half a percent. Sometimes this PR's branch was faster, sometimes the combined branch was. I conclude that the two approaches are compatible, and that there's no need to add an option for callers to disable internal context reuse.

[tameware]

For one way to keep a test in the CI but out of bazel test //..., see --test_tag_filters=-e2e at the top of .bazelrc.