[Protocol]: Add lifecycle-safe disaggregated KV session orchestration

[connorcarpenter15]

Existing proposals

• I searched the existing issues and did not find this proposal.

Intended consumers

Distributed frameworks.

Problem and use case

OpenEngine currently models P/D handoff as a fixed sequence: call prefill Generate, wait for terminal PrefillReady, then call decode Generate with the returned KvSessionRef. The engine owns lifetime, but the contract exposes no allocation, observation, renewal, or release operation.

This cannot portably express context-first, generation-first, or concurrent startup; decoder-created rendezvous; conditional prefill bypass; rejection after KV is pinned; lease renewal while queued; deterministic cleanup after cancellation/crash/timeout; or idempotent orchestration retries. Leaked blocks reduce capacity, while early reclamation causes failed transfers.

Evidence from engine protocols

• SGLang Model Gateway dispatches prefill and decode concurrently in its gRPC request execution.
• TensorRT-LLM exposes context_first and generation_first scheduling in trtllm-serve.
• vLLM NIXL usage requires explicit transfer ownership and failure handling.
• llm-d requires scheduler-directed disaggregation and runtime latency/throughput tradeoffs.

Proposed contract

enum KvScheduleStyle {
  KV_SCHEDULE_STYLE_UNSPECIFIED = 0;
  KV_SCHEDULE_STYLE_CONTEXT_FIRST = 1;
  KV_SCHEDULE_STYLE_GENERATION_FIRST = 2;
  KV_SCHEDULE_STYLE_CONCURRENT = 3;
}

enum KvTransferDisposition {
  KV_TRANSFER_DISPOSITION_UNSPECIFIED = 0;
  KV_TRANSFER_DISPOSITION_REQUIRED = 1;
  KV_TRANSFER_DISPOSITION_BYPASS = 2;
}

enum KvSessionState {
  KV_SESSION_STATE_UNSPECIFIED = 0;
  KV_SESSION_STATE_ALLOCATED = 1;
  KV_SESSION_STATE_READY = 2;
  KV_SESSION_STATE_ATTACHED = 3;
  KV_SESSION_STATE_COMPLETE = 4;
  KV_SESSION_STATE_BYPASSED = 5;
  KV_SESSION_STATE_REJECTED = 6;
  KV_SESSION_STATE_CANCELLED = 7;
  KV_SESSION_STATE_FAILED = 8;
  KV_SESSION_STATE_RELEASED = 9;
  KV_SESSION_STATE_EXPIRED = 10;
}

service OpenEngineKv {
  rpc AllocateKvSession(AllocateKvSessionRequest) returns (KvSession);
  rpc GetKvSession(GetKvSessionRequest) returns (KvSession);
  rpc RenewKvSession(RenewKvSessionRequest) returns (KvSession);
  rpc ReleaseKvSession(ReleaseKvSessionRequest)
      returns (ReleaseKvSessionResponse);
}

Allocation is possible before either Generate. Consumer allocation can return the rendezvous for generation-first/concurrent flows; producer-first implementations may allocate on the producer. The same reference correlates both participants. KvOptions gains an explicit required/bypass disposition. The normative state machine remains portable and does not expose scheduler internals.

Presence, defaults, and validation

• Request ID, model, and idempotency key are required for allocation.
• Identical idempotency retries return the session; conflicting reuse fails.
• Unspecified schedule style requests engine/framework default without asserting support. Unsupported explicit styles fail before allocation.
• Absent lease selects an advertised default and the response reports absolute expiry. Zero/negative is invalid and absence never means immortal.
• Session ID is required for get/renew/release. Release is idempotent; tombstones remain queryable for safe retries.
• A decode request without a session is valid only under explicit bypass or an advertised local-prefill mode.
• Released, expired, rejected, cancelled, failed, complete, and bypassed are terminal and cannot be resurrected.
• Incompatible transfer metadata is rejected before mutation where possible.

Request and response lifecycle

Normative flows cover:

1. Context-first: producer computes, session becomes ready, consumer attaches, transfer completes, release follows.
2. Generation-first: consumer allocates a rendezvous and may queue; producer later fills the same session.
3. Concurrent: both calls start with one allocated reference; temporary not-ready state is not failure.
4. Conditional bypass: no producer/session is required; the consumer uses local state or fails cleanly.
5. Rejection/failure: either participant can trigger abort and idempotent release; leases reclaim orphaned state.

Abort(kv_session) stops active work; ReleaseKvSession relinquishes transfer resources. Both orders are safe. Cancellation cleans partial allocations. A queued consumer may renew; missing renewal expires and reclaims producer state. Drain counts allocated sessions and leaves no live session after completion.

Discovery and capability impact

KvConnectorInfo advertises supported schedule styles, caller allocation, renewable leases, explicit release, conditional bypass, session lookup, and relevant limits. Clients reject incompatible deployments before traffic.

Alternatives considered

• Keep only PrefillReady: cannot decode first/concurrently or own cleanup.
• Encode lifecycle in Struct: not portable or independently testable.
• Rely only on Abort: does not distinguish stopping computation from releasing completed transfer state and provides no leases/allocation idempotency.
• Bidirectional Generate: does not solve independent lifecycle, retries, or crash cleanup.

Out of scope

Worker selection, cache-hit policy, block placement, transport choice, or application-level multi-turn sessions.

Proposal checklist

• The behavior is portable and does not expose one engine's internal scheduler model as the common API.
• I have identified the corresponding documentation changes.
• Add allocation/get/renew/release RPCs and a portable state machine.
• Define context-first, generation-first, concurrent, and bypass flows.
• Define abort/release ordering, leases, idempotency, rejection, and drain cleanup.
• Add failure/cancellation/peer-loss conformance tests proving no pinned resource remains.
• Document lossless mappings for vLLM, SGLang, and TensorRT-LLM.
• Add sequence diagrams and status/error semantics to docs/api.md.