Skip to content

server: preempt a slot instead of ending every conversation when the KV pool fills - #184

Open
danielhanchen wants to merge 4 commits into
masterfrom
feat/server-side-preemption
Open

server: preempt a slot instead of ending every conversation when the KV pool fills#184
danielhanchen wants to merge 4 commits into
masterfrom
feat/server-side-preemption

server: report preemption through /metrics and /slots

63cdac4
Select commit
Loading
Failed to load commit list.

Workflow runs completed with no jobs