fix: unbreak benchmark pipeline (ty errors + Python selection) - #9
Conversation
The weekly run has failed since late December 2025. `run.sh` uses
`set -euo pipefail`, so the non-zero exit from `ty check .` aborted the
suite before any benchmark ran. A ty release started flagging four real
issues in 05_comparison.py:
- `DataFrame.to_markdown()` is typed `str | None`, so `+ "\n\n"` was
rejected; use an f-string instead.
- `DataFrame.itertuples()` is typed `tuple[Any, ...]`, so `row.framework`
and friends were unresolved attributes; iterate `to_dict("records")`.
ty stays unpinned on purpose: the repo benchmarks latest versions, and
picking up new diagnostics is the point.
Also replace the ~80-line Python-version detection step. It tried the
newest interpreter and downgraded up to twice, doing a full `uv pip sync`
per attempt just to test compatibility. Locally it already needed all
three attempts (3.15.0b4 -> 3.14 -> 3.13) and would have hard-failed on
the next CPython minor. Since fireducks is the framework that lags newest
CPython, read the highest cp3XX Linux wheel tag from its latest release
and use that, then confirm the remaining dependencies resolve there.
Selects Python 3.13 today and adopts newer versions automatically.
Verified end to end with `act`: full workflow green.
|
Warning Review limit reached
Next review available in: 53 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: ASSERTIVE Plan: Pro Plus Run ID: 📒 Files selected for processing (1)
Summary by CodeRabbit
WalkthroughThe workflow now selects and validates the Python version from FireDucks wheel availability. Benchmark expressions and comparison output use revised formatting without changing their behavior. The README documents workflow version selection and scheduled-workflow re-enablement. ChangesBenchmark maintenance
Estimated code review effort: 3 (Moderate) | ~20 minutes 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/workflows/benchmark.yml:
- Around line 32-36: Update the PyPI request in the minor-version discovery
pipeline to add an explicit short connect timeout, maximum request duration, and
limited retry count for transient failures. Preserve the existing curl failure
behavior and jq-based version extraction.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 8c1b5b88-df05-4bd4-8c1a-6cf1241b3511
📒 Files selected for processing (4)
.github/workflows/benchmark.yml02_benchmark.py05_comparison.pyREADME.md
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
1 participant
Problem
The weekly benchmark has failed on every run since late December 2025 (last success: 2025-12-21).
run.shusesset -euo pipefail, so the non-zero exit fromty check .at line 72 aborted the whole suite before a single benchmark ran, which matches the ~25s failure duration. Atyrelease in that window started emitting four diagnostics:05_comparison.py:413:DataFrame.to_markdown()is typedstr | None, so+ "\n\n"is an unsupported operator05_comparison.py:433-435:DataFrame.itertuples()is typedtuple[Any, ...], sorow.framework/row.cache_status/row.meanare unresolved attributesReproduced locally with
actagainst the unmodified tree, which failed at exactly this step.Two further issues surfaced while diagnosing:
disabled_inactivity). GitHub auto-disables scheduled workflows after 60 days without repo activity, which is why the last run was 2026-03-22. Already re-enabled; it now reportsactive.uv pip syncper attempt just to test compatibility. The localactrun consumed all three attempts (3.15.0b4→3.14→3.13), so the next minor would have hitNo compatible Python version found after 3 attempts. It also evaluated pre-release betas as candidates, and theUsing Python $PYTHON_VERSIONecho was always empty (GITHUB_ENVonly applies to later steps).Changes
Fix the
tydiagnostics: f-string instead of+concatenation, and iterateto_dict("records")instead ofitertuples().tydeliberately stays unpinned and a hard gate: this repo exists to benchmark latest versions, so picking up new diagnostics is the point, even at the cost of an occasional break.Replace ~80 lines of version-detection bash with a direct query. FireDucks is the framework that lags newest CPython, so its published wheels decide the version: take the highest
cp3XXmanylinux wheel tag from the latest FireDucks release, then confirm the remaining dependencies resolve there before committing to it.Selects Python 3.13 today (FireDucks 1.4.4 ships
cp39–cp313,requires_python: <3.14,>=3.9) and correctly rejects 3.14. It adopts 3.14+ automatically the day FireDucks ships those wheels: no annual maintenance, no retry cap.Housekeeping:
ruff format .applied (02_benchmark.pywas being silently reformatted by CI on every run), and a README note about the 60-day auto-disable.