Skip to content

fix(dataset): detect failed streaming packing workers - #10083

Open
MrCapricornLiu wants to merge 2 commits into
modelscope:mainfrom
MrCapricornLiu:lch/fix-streaming-packing-worker-errors
Open

fix(dataset): detect failed streaming packing workers#10083
MrCapricornLiu wants to merge 2 commits into
modelscope:mainfrom
MrCapricornLiu:lch/fix-streaming-packing-worker-errors

Conversation

@MrCapricornLiu

Copy link
Copy Markdown
Contributor

PR type

  • Bug Fix
  • New Feature
  • Document Updates
  • More Models or Datasets Support

PR information

With streaming packing and strict=True, an encoding error other than
MaxLengthError terminates a packing worker. The iterator still waits for one
queue result per submitted sample, so it blocks indefinitely. An unexpected
worker exit has the same effect.

Poll the output queue and check the workers' process sentinels when no result
arrives. Raise a RuntimeError pointing to the worker logs if a worker has
exited. Sentinels also work when a forked DataLoader process inherits the
packing workers; checking exitcode there cannot detect processes owned by
its parent. Live workers may take longer than the polling interval, and
non-strict errors and MaxLengthError retain their existing skip behavior.

Experiment results

CUDA_VISIBLE_DEVICES='' OMP_NUM_THREADS=1 TOKENIZERS_PARALLELISM=false \
  HF_HUB_OFFLINE=1 python -m pytest \
  tests/general/test_packing_multiprocessing_context.py -q

All 40 tests pass, including seven regressions for strict errors, abrupt exit,
skipped samples, slow workers, and failure/live-worker behavior through a
forked DataLoader consumer. The two original worker-failure regressions fail
without the fix. Changed-file pre-commit and diff checks pass.

Tested on Linux with Python 3.12 and real spawned packing workers. Full model
training, distributed training and Windows/macOS were not run. Workers that
remain alive but are stuck inside an encoder are outside this change.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant