Skip to content

fix(kv-cache): drain finite replay requests - #840

Draft
jay-tau wants to merge 3 commits into
mlcommons:mainfrom
jay-tau:codex/fix-finite-replay-drain
Draft

fix(kv-cache): drain finite replay requests#840
jay-tau wants to merge 3 commits into
mlcommons:mainfrom
jay-tau:codex/fix-finite-replay-drain

Conversation

@jay-tau

@jay-tau jay-tau commented Aug 3, 2026

Copy link
Copy Markdown
Member

Summary

  • distinguish finite replay completion from hard cancellation
  • drain all generated BurstGPT and ShareGPT requests before shutdown
  • preserve non-draining behavior for duration, request-limit, autoscaler, failure, and explicit cancellation stops
  • cover graceful and hard-stop paths with replay regression tests

Tests

  • focused replay shutdown tests: 21 passed
  • complete KV-cache test suite: 250 passed, 2 deselected
  • git diff --check
  • compileall

Fixes #839

@jay-tau
jay-tau requested a review from a team August 3, 2026 11:58
@github-actions

github-actions Bot commented Aug 3, 2026

Copy link
Copy Markdown

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

@jay-tau
jay-tau marked this pull request as draft August 3, 2026 11:59
@jay-tau
jay-tau marked this pull request as ready for review August 3, 2026 12:06
@jay-tau
jay-tau marked this pull request as draft August 3, 2026 12:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Finite replay cycles drop queued requests

2 participants