server: preempt a slot instead of ending every conversation when the KV pool fills - #184

Open

danielhanchen wants to merge 29 commits into

feat/server-side-preemption

Commits on Sep 5, 2026

server: preempt a slot instead of ending every conversation when the KV pool fills

Daniel Han
authored and
danielhanchen
committed
6009998
server: test that preemption keeps the output and finishes every slot
danielhanchen
committed
41cbff4
server: park prompt-processing slots too, restore whatever fits first, and bound the parked state with --preempt-ram
danielhanchen
committed
32c0a77
server: report preemption through /metrics and /slots
danielhanchen
committed
63cdac4
Batch fragmented sequence restores through bounded host staging
danielhanchen
committed
75944db

Commits on Sep 6, 2026

Commits on Sep 7, 2026