Discrete-event simulation benchmark for decode preemption policies in LLM serving: when to interrupt a running decode to admit an urgent prefill, and whether checkpoint or recompute is cheaper.
performance-engineering benchmark simulation latency scheduling inference checkpoint systems decode slo fairness preemption discrete-event-simulation prefill kv-cache llm llm-serving machine-learning-infrastructure recompute serving-systems
-
Updated
Jul 26, 2026 - Python