Skip to content

fix: key pending commit work by consumer and partition - #99

Merged
lesnik512 merged 1 commit into
mainfrom
fix/pending-keyed-by-consumer
Oct 4, 2026
Merged

lesnik512 merged 1 commit into
mainfrom
fix/pending-keyed-by-consumer

Conversation

@lesnik512

Copy link
Copy Markdown
Member

Closes #96.

Problem

PendingCommits kept one offset-sorted list per TopicPartition. When one concurrent handler serves subscribers in different consumer groups that read the same topic, both groups' user tasks for a partition shared that list, so the ready prefix was computed across both. An in-flight task in group A held back group B's finished offsets on that partition until A's task finished. Commits stayed safe, but one group's slow handler stalled the other group's commits and their share of max_uncommitted_tasks.

Change

  • Pending lists are keyed by (id(consumer), partition), the same owner identity the cancellation watermarks already use. The new OwnerKey alias names it once for both.
  • extract_ready_prefixes is unchanged apart from its key type. The public surface of PendingCommits (absorb, hold, take_ready, __len__, clear_watermarks) and the shape of ReadyCommit (still one per consumer) are unchanged.
  • The Ready prefix glossary entry and the README's committer paragraph now say "per consumer and partition".

Tests

Regression tests written first. These fail on main:

  • test_another_consumers_unfinished_task_does_not_hold_back_a_ready_prefix (both orders)
  • test_unfinished_task_still_holds_back_its_own_consumers_later_tasks
  • test_another_consumer_groups_in_flight_task_does_not_delay_a_commit (committer-level, the reproduction from the issue)

test_cancelled_task_sets_a_watermark_only_for_its_own_consumer passes on main and guards watermark scoping. The four extract_ready_prefixes tests now build pending state keyed by OwnerKey.

just lint passes. just test --cov=. --cov-branch gives 232 passed, 100% branch coverage.

Pending lists were keyed by TopicPartition alone, so subscribers in different
consumer groups reading the same topic shared one list per partition, and an
unfinished task in one group held back the other group's ready prefix. Key
them by (id(consumer), partition), the identity cancellation watermarks
already use.

Closes #96
@lesnik512
lesnik512 merged commit bebbc6e into main Oct 4, 2026
21 of 22 checks passed
@lesnik512
lesnik512 deleted the fix/pending-keyed-by-consumer branch October 4, 2026 09:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

One consumer group's in-flight task holds back another group's ready prefix on a shared partition

1 participant