Skip to content

feat: stream HTTP responses (SSE/chunked) and generalise scheduled push to core - #102

Draft
outofcoffee wants to merge 4 commits into
mainfrom
feat/streaming-responses
Draft

outofcoffee wants to merge 4 commits into
mainfrom
feat/streaming-responses

Conversation

@outofcoffee

@outofcoffee outofcoffee commented Jul 26, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds incremental HTTP streaming to the rest and openapi plugins via stream: true, and generalises the websocket plugin's multiple-response and scheduled-frame machinery into a protocol-agnostic core (internal/emit) that HTTP and websocket now share.

Two capabilities on a streamed resource:

  • Fixed sequence — a responses list is written and flushed one chunk at a time, paced by each block's delay. Ideal for an OpenAI-style token stream ending in [DONE].
  • Open-ended push — a schedule on a streamed resource pushes further responses over the open connection until its limit is reached or the client disconnects (SSE keepalive, progress, live data).
plugin: rest
resources:
  - path: /v1/chat/completions
    method: POST
    stream: true
    responses:
      - content: "data: {...}\n\n"
        headers: { Content-Type: text/event-stream }
        delay: { exact: 50 }
      - content: "data: [DONE]\n\n"

The generalisation (DRY)

New core package internal/emit:

  • Sink — abstracts "send one processed response body" (a websocket text frame, or an HTTP flushed chunk).
  • EmitResponses — the shared process-then-emit loop (per-block delay/file/content/template).
  • ScheduleHost — runs connection- or request-scoped schedules through the existing internal/scheduler, emitting via a Sink.

The websocket plugin is refactored onto these (a frameSink + ScheduleHost), deleting its bespoke send/schedule code. OpenAPI inherits streaming for free because it already serves responses through the REST pipeline. The top-level engine-lifetime scheduler is unchanged (no client sink; avoids an import cycle).

Behaviour & safety

  • Chunked (Transfer-Encoding: chunked); status/headers taken from the first chunk. SSE framing is authored in the content, so the same mechanism serves NDJSON or any chunked format.
  • Client disconnect stops request-scoped schedules promptly (via r.Context()).
  • Non-flushable connections (AWS Lambda adapter): chunks are concatenated into one buffered response and schedules are skipped with a warning.
  • Validation: multiple responses or a schedule on an HTTP resource require stream: true; stream: true with nothing to stream is rejected; schedule-entry checks are shared between websocket and HTTP.

Tests & docs

  • Unit tests for internal/emit (emit loop, buffered fallback, schedule limit + context-cancel), streaming validation tests. Full suite: 30 packages pass; go test -race clean on the concurrency-sensitive packages.
  • New examples/rest/sse-streaming example and docs/streaming.md.

Verified end-to-end on the native engine: 50 ms SSE pacing to [DONE]; scheduled push honouring limit and stopping on client disconnect; and OpenAPI streaming through the shared pipeline.

Add `stream: true` on rest/openapi resources to deliver output incrementally
(HTTP chunked / Server-Sent Events) instead of buffering a single body:

- a `responses` list is flushed one chunk at a time, paced by each block's
  `delay` (e.g. an OpenAI-style token stream ending in [DONE]);
- a `schedule` on a streamed resource pushes further responses over the open
  connection until its limit is hit or the client disconnects (SSE keepalive,
  progress, live data).

The sending and scheduling machinery that was specific to the websocket plugin
is generalised into a new core package, `internal/emit`:

- `Sink` abstracts "send one processed body" (a websocket frame, or an HTTP
  flushed chunk);
- `EmitResponses` is the shared process-then-emit loop;
- `ScheduleHost` runs connection- or request-scoped schedules via the existing
  `internal/scheduler`, emitting through a Sink.

The websocket plugin is refactored onto these primitives (a `frameSink` plus
`ScheduleHost`), removing its bespoke send/schedule code. OpenAPI inherits
streaming for free because it already serves responses through the REST
pipeline. When the connection cannot flush (AWS Lambda adapter), the chunks are
concatenated into one buffered response and schedules are skipped.

Validation: multiple `responses`, or a `schedule`, on an HTTP resource now
require `stream: true`; `stream: true` with nothing to stream is rejected; the
schedule-entry checks are shared between websocket and HTTP resources.

Adds unit tests for the emit package (including the buffered fallback and
schedule limit/cancel), streaming validation tests, an `examples/rest/
sse-streaming` example and `docs/streaming.md`. Verified end to end on the
native engine: incremental SSE pacing, scheduled push with limit and
client-disconnect, and OpenAPI streaming.

Claude-Session: https://claude.ai/code/session_01NGFQaVnKN4wv322agMownn
@outofcoffee outofcoffee added the enhancement New feature or request label Jul 26, 2026
A websocket connection is inherently a streaming, multi-frame transport, so it
is implicitly 'stream: true'. The real misconfiguration is an explicit
'stream: false', which asserts something the plugin cannot honour — but a plain
bool cannot tell 'false' from 'unset', so the previous code instead warned on
the harmless-and-consistent 'stream: true'.

Make BaseResource.Stream a *bool (tri-state) with a StreamEnabled() helper:

- websocket: 'stream: true'/unset are accepted silently; explicit 'stream:
  false' is now a startup error.
- rest/openapi: unset and 'stream: false' both mean the default (buffered);
  'stream: true' opts into incremental streaming, as before.
- other plugins: only an explicit 'stream: true' is rejected as unsupported.

Adds validation tests for the websocket true/false cases and updates the
streaming doc. Full suite passes.

Claude-Session: https://claude.ai/code/session_01NGFQaVnKN4wv322agMownn
Use a plain progress-style SSE stream in the fixed-sequence example instead of
OpenAI chat-completion chunks, and drop the "OpenAI compatible" aside, so the
guide teaches the feature generically. The concrete OpenAI-shaped case stays in
the examples/rest/sse-streaming example.

Claude-Session: https://claude.ai/code/session_01NGFQaVnKN4wv322agMownn
Reword doc comments that narrated how the code came to be (e.g. "the
generalisation of the websocket plugin's connection-scoped schedules") to
describe what the code does now, and drop speculative "leaves room for later"
rationale.

Claude-Session: https://claude.ai/code/session_01NGFQaVnKN4wv322agMownn
@outofcoffee outofcoffee self-assigned this Jul 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request refactor

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant