Move pure filesystem, project, and scoring helpers into bcbench-core - #917
Open
Sun Haoran (haoranpb) wants to merge 2 commits into
Open
Sun Haoran (haoranpb) wants to merge 2 commits into
Sun Haoran (haoranpb) wants to merge 2 commits into
Conversation
First code to move into the library. These helpers have no dependency on BC-Bench configuration or policy, so they move with minimal changes and the app imports them directly from bcbench_core (no re-export shims). - bcbench_core.filesystem: remove_tree, clear_directory, prepare_run_dir - bcbench_core.projects: categorize_projects; the AL test-project identifiers become DEFAULT_TEST_PROJECT_IDENTIFIERS instead of a constant app config field, and collection uses the same constant - bcbench_core.scoring: precision/recall, F-beta, pass@k, pass^k; bootstrap_ci stays in the app because it needs numpy and scipy - Core tests live in packages/bcbench-core/tests with their own pytest configuration (importlib import mode, strict) and run as a separate CI step Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 3fa47b3c-f5d6-4c36-9be3-c4fdf2bebe47
…roject
- Run app and core tests in one pytest invocation from the root config;
uv workspaces manage environments, not test runners, and a single run
yields one coverage report for both packages
- Move bootstrap_ci into bcbench_core.stats; core now depends on numpy and
scipy, kept out of bcbench_core.scoring so scoring stays dependency-free
- Make is_test_project public and inline the ("test", "tests") default;
collection keeps its own substring-based identifiers in app config
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 3fa47b3c-f5d6-4c36-9be3-c4fdf2bebe47
Sun Haoran (haoranpb)
marked this pull request as ready for review
October 2, 2026 13:03
Sun Haoran (haoranpb)
added this pull request to stack #916
October 2, 2026 13:04
Contributor
There was a problem hiding this comment.
Copilot review overview
🔵 Needs a closer look
The public project classifier does not satisfy its documented case-insensitive component-prefix behavior.
Review effort: Balanced
Findings: None
What changed in this PR
Extracts reusable filesystem, project categorization, scoring, and statistical helpers into bcbench-core, updating application consumers and test infrastructure.
Changes:
- Moves reusable helpers into four core modules.
- Updates application and notebook imports without compatibility shims.
- Relocates and expands core tests and CI coverage.
| File | Description |
|---|---|
uv.lock |
Locks core runtime and test dependencies. |
tests/test_filesystem_operations.py |
Removes superseded application-level tests. |
src/bcbench/results/leaderboard.py |
Imports scoring and statistics from core. |
src/bcbench/results/codereview.py |
Imports scoring helpers from core. |
src/bcbench/results/__init__.py |
Removes obsolete metric re-exports. |
src/bcbench/operations/git_operations.py |
Uses core filesystem helper. |
src/bcbench/operations/bc_operations.py |
Uses core filesystem helper. |
src/bcbench/operations/__init__.py |
Removes moved helper exports. |
src/bcbench/evaluate/testgeneration.py |
Uses core project categorization. |
src/bcbench/evaluate/nl2al.py |
Uses core tree removal. |
src/bcbench/evaluate/dataquery.py |
Uses core directory clearing. |
src/bcbench/evaluate/codereview_judge_calibration.py |
Uses core precision and recall. |
src/bcbench/evaluate/bugfix.py |
Uses core project categorization. |
src/bcbench/commands/evaluate.py |
Uses core run-directory preparation. |
src/bcbench/commands/contamination.py |
Uses core run-directory preparation. |
pyproject.toml |
Adds core tests to pytest discovery. |
packages/bcbench-core/tests/test_stats.py |
Tests bootstrap confidence intervals. |
packages/bcbench-core/tests/test_scoring.py |
Tests extracted scoring helpers. |
packages/bcbench-core/tests/test_projects.py |
Tests project categorization. |
packages/bcbench-core/tests/test_filesystem.py |
Tests filesystem helpers. |
packages/bcbench-core/src/bcbench_core/stats.py |
Adds bootstrap statistics. |
packages/bcbench-core/src/bcbench_core/scoring.py |
Houses dependency-light scoring functions. |
packages/bcbench-core/src/bcbench_core/projects.py |
Adds configurable project categorization. |
packages/bcbench-core/src/bcbench_core/filesystem.py |
Houses reusable filesystem helpers. |
packages/bcbench-core/README.md |
Documents the core test command. |
packages/bcbench-core/pyproject.toml |
Declares dependencies and test linting. |
notebooks/utils.py |
Updates scoring imports. |
notebooks/test-generation/altest-comparison.ipynb |
Updates core helper imports. |
notebooks/bug-fix/claude-vs-copilot.ipynb |
Updates statistics import. |
notebooks/bug-fix/altool-comparison.ipynb |
Updates core helper imports. |
notebooks/archive/multi-run.ipynb |
Updates scoring imports. |
CONTRIBUTING.md |
Documents combined coverage invocation. |
.github/workflows/CI.yml |
Includes core in coverage collection. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stack 2 of the
bcbench-coreextraction, stacked on #914. Moves the first code into the library: helpers that are free of BC-Bench configuration, logging setup, and benchmark policy. The app imports them directly frombcbench_core; there are no re-export shims.bcbench_core.filesystemoperations/filesystem_operations.py(rename)remove_tree,clear_directory,prepare_run_dirbcbench_core.projectsoperations/project_operations.py(rename)is_test_project,categorize_projectsbcbench_core.scoringresults/metrics.py(split)precision_recall,f_beta_score,f1_score,pass_at_k,pass_hat_kbcbench_core.statsresults/metrics.py(rename)bootstrap_ciBehavior-preserving changes needed to move:
categorize_projectsandis_test_projecttake test identifiers as a parameter defaulting to("test", "tests")instead of reading app config. Patch collection keeps the config field because it matches identifiers as substrings, not path-component prefixes.projectslogs through a standardlogging.getLogger(__name__)logger. Its only log is an error, which still reaches the app's root handlers (including GitHub annotations).bootstrap_ci. They are imported only bybcbench_core.stats, sobcbench_core.scoringstays dependency-free at import time.Tests: moved tests live in
packages/bcbench-core/testsand run in the sameuv run pytestinvocation as the app (testpathscovers both), producing one coverage report. I added direct tests forprecision_recall, F-beta,remove_tree, andprepare_run_dir, which previously had only indirect coverage; core is at 100% line coverage. Core declares pytest in its owndevgroup so ty'smissing-direct-dependencyaccepts test imports.Deferred: git operations need a logging decision first. They log at INFO, but the app only raises the
bcbenchlogger above the root's WARNING level, so INFO logs frombcbench_core.*would be dropped.