Port benchmark to python by tameware · Pull Request #255 · dds-bridge/dds

tameware · 2026-07-22T02:29:22Z

No description provided.

Replace the shell implementation with python/tests/benchmark.py, keep ./benchmark.sh as a thin wrapper, and move the dirty-tree git-prep checks into //python:benchmark_test. Co-authored-by: Cursor <cursoragent@cursor.com>

…e/dds into port-benchmark-to-python

Copilot

Pull request overview

This PR ports the repository’s dtest benchmarking workflow from a large Bash script to a Python implementation, and updates Bazel targets accordingly (including replacing the prior shell-based git-prep test with a Python unit test suite).

Changes:

Added a Python benchmark runner (python/tests/benchmark.py) that supports building/checking out branches, running dtest, and producing summary tables.
Added stdlib unittest coverage for argument parsing, output parsing, summary formatting, and git branch “dirty tree” gating (python/tests/benchmark_test.py).
Updated Bazel setup to expose the benchmark as a py_binary/py_test and removed the previous rules_shell dependency and sh_test.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
python/tests/benchmark.py	New Python benchmark implementation for running and comparing `dtest` binaries.
python/tests/benchmark_test.py	New unit tests covering benchmark logic and git-prep behavior.
python/BUILD.bazel	Adds `benchmark_lib`, `benchmark` binary, and `benchmark_test` Bazel targets.
MODULE.bazel	Removes direct `rules_shell` dependency no longer needed after migrating tests.
BUILD.bazel	Removes the old `sh_test` target for benchmark git-prep validation.
benchmark.sh	Converts to a thin wrapper invoking the Python benchmark.
benchmark_git_prep_test.sh	Removes the old shell-based git-prep test script.

tameware · 2026-07-22T03:10:26Z

I'll address Copilot's open suggestions in a future PR. This one is intended to be a strict port, though in a moment of weakness I did add one divide-by-zero guard.

Branch builds overwrite bazel-bin dtest, so that mix could benchmark the wrong executable after restore. Co-authored-by: Cursor <cursoragent@cursor.com>

Cleanup and main were hardcoding sys.stdout, so a redirected or custom output stream could miss the terminal teardown. Co-authored-by: Cursor <cursoragent@cursor.com>

run_build was hardcoding sys.stderr, which broke the runner constructor contract for redirected or custom error output. Co-authored-by: Cursor <cursoragent@cursor.com>

Copilot

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.

Dropping the whole solver/file row hid partial results; keep the line and omit ratio/note until both sides are available. Co-authored-by: Cursor <cursoragent@cursor.com>

Copilot

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.

Copilot

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 2 comments.

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Bazel emits dtest.exe on nt, so the hardcoded dtest path broke the default baseline and checkout-binary overlap check on Windows. Co-authored-by: Cursor <cursoragent@cursor.com>

Copilot

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (1)

python/tests/benchmark.py:719

detect_git_branch() calls out to git unconditionally via _git(...). If git is not installed (or not on PATH), this raises FileNotFoundError and crashes even when the user is not using --branch (previous benchmark.sh tolerated missing git and just printed git branch as "unknown"). Catch OSError here and fall back to "unknown".

    def detect_git_branch(self) -> None:
        probe = _git(self.root, "rev-parse", "--is-inside-work-tree", check=False)
        if probe.returncode != 0:
            self.git_branch = "unknown"
            return

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated no new comments.

tameware and others added 2 commits July 21, 2026 21:23

Port dtest benchmark.sh to Python with unit tests.

77f78b7

Replace the shell implementation with python/tests/benchmark.py, keep ./benchmark.sh as a thin wrapper, and move the dirty-tree git-prep checks into //python:benchmark_test. Co-authored-by: Cursor <cursoragent@cursor.com>

Merge branch 'port-benchmark-to-python' of https://github.com/tamewar…

3fe02eb

…e/dds into port-benchmark-to-python

tameware requested a review from Copilot July 22, 2026 02:29

Copilot started reviewing on behalf of tameware July 22, 2026 02:29 View session

Copilot AI reviewed Jul 22, 2026

View reviewed changes

Comment thread python/tests/benchmark.py

Comment thread python/tests/benchmark.py

Comment thread python/tests/benchmark.py

tameware self-assigned this Jul 22, 2026

tameware marked this pull request as ready for review July 22, 2026 03:09

tameware and others added 3 commits July 23, 2026 00:13

Reject --binary at checkout dtest when combined with --branch.

33aeebe

Branch builds overwrite bazel-bin dtest, so that mix could benchmark the wrong executable after restore. Co-authored-by: Cursor <cursoragent@cursor.com>

Write alt-screen escapes to the injected out stream.

687bfd5

Cleanup and main were hardcoding sys.stdout, so a redirected or custom output stream could miss the terminal teardown. Co-authored-by: Cursor <cursoragent@cursor.com>

Write failed build logs to the injected err stream.

7b95821

run_build was hardcoding sys.stderr, which broke the runner constructor contract for redirected or custom error output. Co-authored-by: Cursor <cursoragent@cursor.com>

tameware requested a review from Copilot July 23, 2026 06:04

Copilot started reviewing on behalf of tameware July 23, 2026 06:05 View session

Copilot AI reviewed Jul 23, 2026

View reviewed changes

Comment thread python/tests/benchmark.py Outdated

Show NA in summary when a binary lacks avg_user.

b68a2ec

Dropping the whole solver/file row hid partial results; keep the line and omit ratio/note until both sides are available. Co-authored-by: Cursor <cursoragent@cursor.com>

tameware requested a review from Copilot July 23, 2026 06:25

Copilot started reviewing on behalf of tameware July 23, 2026 06:25 View session

Copilot AI reviewed Jul 23, 2026

View reviewed changes

Comment thread MODULE.bazel

Merge branch 'dds-bridge:develop' into port-benchmark-to-python

285814f

tameware requested a review from Copilot July 23, 2026 20:08

Copilot started reviewing on behalf of tameware July 23, 2026 20:09 View session

Copilot AI reviewed Jul 23, 2026

View reviewed changes

Comment thread python/tests/benchmark.py

Comment thread python/tests/benchmark.py Outdated

tameware requested a review from zzcgumn July 23, 2026 20:24

tameware and others added 2 commits July 23, 2026 21:25

Potential fix for pull request finding

4470ae8

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Use dtest.exe path on Windows for Bazel binary lookup.

8903b27

Bazel emits dtest.exe on nt, so the hardcoded dtest path broke the default baseline and checkout-binary overlap check on Windows. Co-authored-by: Cursor <cursoragent@cursor.com>

tameware requested a review from Copilot July 23, 2026 20:31

Copilot started reviewing on behalf of tameware July 23, 2026 20:31 View session

Copilot AI reviewed Jul 23, 2026

View reviewed changes

Comment thread python/tests/benchmark.py Outdated

Potential fix for pull request finding

16f05eb

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

tameware requested a review from Copilot July 23, 2026 20:38

Copilot started reviewing on behalf of tameware July 23, 2026 20:38 View session

Copilot AI reviewed Jul 23, 2026

View reviewed changes

zzcgumn approved these changes Jul 23, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Port benchmark to python#255

Port benchmark to python#255
tameware wants to merge 10 commits into
dds-bridge:developfrom
tameware:port-benchmark-to-python

tameware commented Jul 22, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tameware commented Jul 22, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

tameware commented Jul 22, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tameware commented Jul 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

tameware commented Jul 22, 2026 •

edited

Loading