test: restore CI test runtime after dependency-policy regression - #3830
Conversation
Castiron custom code✅ No new custom-code files detected. 36 mixed files remain; 0 existing customizations changed. Compared 36 existing customizations unchanged
A changed generated baseline means this report cannot reliably identify which handwritten lines changed. Inspect the custom-code diffDownload the exact patch produced by this run (requires repository access): gh run download 34386625874 --repo openai/openai-python \
--name castiron-custom-code-34386625874-1 --dir /tmp/castiron-custom-code-34386625874-1
git apply --stat /tmp/castiron-custom-code-34386625874-1/custom-code.patch
cat /tmp/castiron-custom-code-34386625874-1/custom-code.patchOr reproduce it from an SDK checkout containing the vendored reporter: git fetch --no-tags origin 41f0a2317759e8796ccfbde75536bd42e4aca7a2 4d81eb7f3dcd2e5a6e1a0cf1c74beea6cf724c92
python3 scripts/castiron/custom_code_report.py report \
--base 41f0a2317759e8796ccfbde75536bd42e4aca7a2 \
--head 4d81eb7f3dcd2e5a6e1a0cf1c74beea6cf724c92 --fetch --require-head-hash --public \
--out /tmp/castiron-custom-code-4d81eb7f3dcd
cat /tmp/castiron-custom-code-4d81eb7f3dcd/custom-code.patchThis is the current full custom patch for mixed files, not an attribution of only the handwritten lines changed by this PR. |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6dc4ed05c0
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
jbeckwith-oai
left a comment
jbeckwith-oai
left a comment
There was a problem hiding this comment.
Reviewed exact head 4d81eb7. The test-only refactor preserves fresh per-case globals while caching compilation, and MonkeyPatch restores cwd, environment, subprocess.run, sys.argv, and the Python 3.10 tomllib alias. Real subprocess parity and trusted-base coverage remain in place, production policy code is unchanged, and exact-head Python 3.10/3.14 plus the rest of CI are green.
markstuart-oai
left a comment
markstuart-oai
left a comment
There was a problem hiding this comment.
PR Review Boi: reviewed 4d81eb7 in response to the sdk-reviews request.
Two independent review passes found no actionable correctness or structural issues. Caching compilation while executing fresh checker globals removes the repeated process overhead without duplicating the policy implementation. Both execution paths use the same Git stub, the scoped patches restore process state, and the real CLI parity and trusted-base/stdlib-shadow tests remain in place. The added harness is proportionate to the runtime fix; no production policy or budget changes are introduced.
Validation: 1,141 dependency-security tests passed on Python 3.12.14 (pytest tests/test_uv_workflows.py -k security -n 0); git diff --check passed. The Python 3.10 alias-restoration branch was reviewed statically, not executed locally. Rechecked that the PR remained open and at this head before submitting.
3 participants
Problem and root cause
Test jobs jumped on August 27 at #3641 (
f3f96317e), which added the dependency-security checker and its policy regression matrix. Comparing adjacent main runs:Before run · After run · This PR
The suites account for 269 of the 276 additional seconds. Large-payload tests were already present while jobs remained around 2m10s; the Steady migration happened afterward. Setup is not the primary cause.
The current security-floor matrix has 1,760 cases. Each previously launched a Python checker and up to four Python Git stubs. This process and compilation overhead repeats under both Pydantic versions in all three test jobs. A representative local profile spent 85ms of 94ms in subprocess calls, with little time in the policy algorithm. A 20-iteration harness comparison measured median 180ms per subprocess case versus 0.9ms in-process.
Change
Compile the same security checker once per pytest worker and execute each policy case with fresh globals, its existing file fixtures, and the same Git stub. Restore patched process state after each execution. Preserve real subprocess execution for CLI parity cases and the existing trusted-base, isolation, and workflow integration tests.
All existing policy cases, SDK coverage, Pydantic lanes, Python versions, large-payload sizes, and production enforcement remain unchanged. Add slow-test reporting to pytest configuration so future regressions are visible.