feat(delphi): Clojure timing probe (Spec C)#2643
Draft
jucor wants to merge 1 commit into
Draft
Conversation
Add scripts/clj_timing_probe.py — a click CLI that runs the Clojure Mode A replay driver (math/dev/replay.clj) at increasing vote-count sizes, records wall-clock seconds + final-blob success per size, fits runtime ~ a + b*N^k (log-log least squares, a estimated from the smallest run as the fixed JVM-startup cost), and recommends the largest N that stays within a wall-clock budget. TDD: tests/replay_harness/test_timing_probe.py mocks the clojure subprocess for all unit tests (CSV truncation, schedule shape, fit recovery on synthetic power-law data, recommendation math, stdout line budget, CLI end-to-end) plus one real-subprocess integration test gated on `clojure` being on PATH and RUN_CLJ_INTEGRATION=1. commit-id:96c1f19e
This was referenced Jul 22, 2026
Draft
Draft
docs(delphi): R1-parity goal docs, journal (2026-07-18 → 07-24 s5), quirks + divergence ledger
#2626
Draft
jucor
marked this pull request as draft
July 22, 2026 00:51
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a Delphi-side CLI tool to empirically benchmark the Clojure Mode A replay driver (math/dev/replay.clj) across increasing vote-count sizes, fit a simple power-law runtime model, and emit a recommendation for a max vote cutoff within a wall-clock budget—along with a thorough mocked test suite and one gated real-subprocess integration test.
Changes:
- Added
delphi/scripts/clj_timing_probe.py: Click CLI to run Clojure replays on truncated vote CSVs, time them, fitt ~ a + b*N^k, and recommend anNwithin--budget-min. - Added
delphi/tests/replay_harness/test_timing_probe.py: unit tests with fully mocked subprocess plus a gated integration test that runs realclojure -M:replay.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| delphi/scripts/clj_timing_probe.py | Implements the timing probe CLI, subprocess invocation, runtime fitting, and report generation. |
| delphi/tests/replay_harness/test_timing_probe.py | Adds unit + gated integration coverage for the timing probe behavior. |
Comments suppressed due to low confidence (1)
delphi/scripts/clj_timing_probe.py:437
- probe() doesn’t handle an empty votes file (n_max==0) and may write schedules / run probes at size=0. Also, recommend_max_votes() can return a value larger than the dataset size; the CLI should clamp the recommendation to n_max so the output is actionable.
n_max = count_data_rows(votes_path)
size_list = parse_sizes(sizes, n_max)
if not size_list:
raise click.UsageError("no sizes to probe")
results = run_all(
size_list, votes_path, dataset=dataset, math_dir=MATH_DIR, timeout=timeout_sec,
)
ok_sizes = [r.size for r in results if r.ok]
ok_seconds = [r.seconds for r in results if r.ok]
if ok_sizes:
fit = fit_power_law(ok_sizes, ok_seconds)
else:
fit = FitResult(a_est=None, b=None, k=None, n_fit_points=0)
recommended = recommend_max_votes(fit, budget_min)
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+106
to
+117
| def parse_sizes(spec: str, n_max: int) -> list[int]: | ||
| """Parse a comma-separated size list, capped at ``n_max``, deduped, ascending.""" | ||
| sizes: set[int] = set() | ||
| for part in spec.split(","): | ||
| part = part.strip() | ||
| if not part: | ||
| continue | ||
| n = int(part) | ||
| if n <= 0: | ||
| raise ValueError(f"size must be positive, got {n}") | ||
| sizes.add(min(n, n_max)) | ||
| return sorted(sizes) |
Comment on lines
+339
to
+343
| lines.append(f"recommended_max_votes~={recommended} for budget={budget_min:g}min") | ||
| else: | ||
| lines.append(f"recommended_max_votes=NA for budget={budget_min:g}min (insufficient data)") | ||
|
|
||
| return lines |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add scripts/clj_timing_probe.py — a click CLI that runs the Clojure
Mode A replay driver (math/dev/replay.clj) at increasing vote-count
sizes, records wall-clock seconds + final-blob success per size, fits
runtime ~ a + b*N^k (log-log least squares, a estimated from the
smallest run as the fixed JVM-startup cost), and recommends the
largest N that stays within a wall-clock budget.
TDD: tests/replay_harness/test_timing_probe.py mocks the clojure
subprocess for all unit tests (CSV truncation, schedule shape, fit
recovery on synthetic power-law data, recommendation math, stdout
line budget, CLI end-to-end) plus one real-subprocess integration
test gated on
clojurebeing on PATH and RUN_CLJ_INTEGRATION=1.commit-id:96c1f19e
Stack: