Skip to content

Reject nonpositive chunksizes in pool imap methods - #481

Open
JackyHsu-X wants to merge 1 commit into
celery:mainfrom
JackyHsu-X:fix/imap-chunksize-validation
Open

JackyHsu-X wants to merge 1 commit into
celery:mainfrom
JackyHsu-X:fix/imap-chunksize-validation

Conversation

@JackyHsu-X

Copy link
Copy Markdown

What does this PR do?

Reject non-positive chunksize with ValueError in Pool.imap() and Pool.imap_unordered(), before consuming the iterable or queuing tasks. This replaces assertions that disappear under python -O.

With the current code, list(pool.imap(abs, [-1, -2], chunksize=0)) returns [] under python -O; imap_unordered() does the same. The nonempty input is silently discarded. The explicit checks keep validation active in optimized Python, matching the standard library's behavior.

Adds regressions for zero and negative chunk sizes, empty and nonempty iterables, unchanged cache/queue state on rejection, and valid chunk sizes after rejection. Includes a changelog entry.

Verification

  • Windows, Python 3.12.8: all 12 new cases pass in ordinary and optimized mode. Against the original source, all 8 rejection cases fail under -O.
  • Linux, Python 3.12: full t/unit run: 97 passed, 51 skipped, 2 failed. A pristine checkout at 3a806fb has the same two failures (test_exception_info_log_before_pickle and test_exception_info_log_after_pickle), with 85 passed and 51 skipped.
  • Real Linux two-worker pool: reproduced silent empty results on the original code under -O. Both methods reject 0/-1 and subsequently execute valid batches with chunk sizes 1/2 on the fix, with and without -O.
  • Official pre-commit codespell hook passes. Flake8 has no new diagnostics compared with the original touched files; existing diagnostics remain.

The complete Windows pool suite was not completed: a real spawn-pool check stalled and was stopped. Other Windows unit tests have two failures also reproduced on the pristine checkout. No cross-version or free-threaded matrix was run.

No existing issue is claimed as fixed. Open PRs #473 and #475 address timeout and worker-exit handling, respectively.

Codex assisted with investigation, implementation, test execution, and this PR description.

Signed-off-by: JackyHsu-X <xv1070733106@gmail.com>
@auvipy
auvipy self-requested a review October 24, 2026 04:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant