Skip to content

[PureGo] Add ingestion core: buffer, encoder, ack model, and supervisor#531

Draft
zlata-stefanovic-db wants to merge 3 commits into
purego-oauth-providerfrom
purego-core
Draft

[PureGo] Add ingestion core: buffer, encoder, ack model, and supervisor#531
zlata-stefanovic-db wants to merge 3 commits into
purego-oauth-providerfrom
purego-core

Conversation

@zlata-stefanovic-db

@zlata-stefanovic-db zlata-stefanovic-db commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

Summary

Fourth in the pure-Go SDK stack. Stacked on #496 (base branch purego-oauth-provider), though it can also target main once #495/#496 merge.

Adds internal/stream — the generic protocol-agnostic ingestion core that sits between the auth and transport layers and the (not yet built) public API:

  • buffer.go — bounded, offset-assigning queue with semaphore-based backpressure. enqueue and next are both context.Context-cancellable: enqueue blocks until a slot opens (or ctx fires); next blocks until an item is queued (or its per-stream context fires), so each runOnce can stop its sender goroutine cleanly without cancelling the supervisor's outer context. requeue moves all in-flight items back to the front of the queue so recovery re-sends them in order. A doneCh unblocks callers parked on the semaphore when the buffer is closed or drained, so a blocked Ingest returns instead of hanging forever.

  • encoder.goencoder interface plus four implementations: protoEncoder, jsonEncoder, protoBatchEncoder, jsonBatchEncoder. Encoding is eager at Ingest time so the buffer holds []byte, never live user objects. Matches the Rust SDK's pattern (LandingZone holds already-encoded batches).

  • ackmodel.goackModel interface plus offsetAckModel for proto/JSON (DurabilityAckUpToOffset maps directly to the logical offset). Arrow's record-count ack model slots in here without touching the core.

  • core.go — three goroutines (sender, receiver, supervisor), monotonic ack watermark, Flush/WaitForOffset/Close/GetUnacked. The sender uses a per-runOnce derived context so a stream teardown unblocks buf.next() without cancelling the supervisor context (which would signal Close). Offset assignment is serialized with enqueue under ingestMu so nextOffset only advances for records that actually make it into the buffer — a failed Ingest (encode error or ctx-cancelled backpressure) no longer consumes an offset that Flush would wait on forever. Flush targets the highest successfully-enqueued offset, not nextOffset-1. Ingest on a cleanly closed stream now returns an error instead of (0, nil).

  • supervisor.gocreate → run → recover loop. Server-side EOF is treated as a retryable disconnect. Defaults match the Rust SDK: RecoveryRetries=4, RecoveryBackoff=2s, FlushTimeout=5m, LackOfAckTimeout=60s. On terminal failure it drains the buffer and fires OnError per unacked item when an AckCallback is registered; otherwise it closes the buffer and leaves items retrievable via GetUnacked (the two are mutually exclusive).

Also adds transport.StreamRPC (exported interface) and transport.NewStreamFromRPC so stream-package tests can inject in-process fake RPCs without a live gRPC server.

Stack

  1. [PureGo] Add gRPC transport and stream layer #463 — gRPC transport + stream plumbing
  2. [PureGo] Add GracefulClose to transport Stream #489 — GracefulClose
  3. [PureGo] Add custom auth-header hook to transport #494 — auth header hook on transport
  4. [PureGo] Add auth provider interfaces and per-table token cache #495 — auth HeadersProvider + token cache
  5. [PureGo] Add UC OAuth token provider with per-table caching #496 — UC OAuth provider
  6. this PR — ingestion core

Test plan

  • go build ./... and go test ./internal/... -race pass (94 tests, 34 in internal/stream).
  • Recovery test (TestCoreStreamRecoveryRequeuesUnacked) verifies items in flight when a stream dies are re-sent on the recovery stream and acknowledged correctly.
  • Added regression tests: Ingest on a closed stream errors, a failed Ingest doesn't advance the Flush target (no permanent gap), and a Close unblocks an Ingest parked at capacity.

What's not yet here

  • Public zerobus package (SDK builder, Stream interface, typed ingest methods)
  • Arrow/Flight wire path and record-count ack model

@zlata-stefanovic-db zlata-stefanovic-db self-assigned this Jul 17, 2026
Adds internal/stream with the generic protocol-agnostic ingestion core:

- buffer.go: bounded offset-assigning queue with semaphore backpressure
  and ctx-cancellable next/enqueue so goroutines stop cleanly per stream
  rather than per supervisor lifetime.
- encoder.go: encoder interface + proto/JSON single-record and batch
  implementations; encoding is eager at Ingest time so the buffer holds
  []byte, never live user objects.
- ackmodel.go: ackModel interface + offsetAckModel for proto/JSON
  (DurabilityAckUpToOffset maps directly to the logical offset).
- core.go: sender/receiver/supervisor goroutines, ack watermark,
  Flush/WaitForOffset/Close/GetUnacked.
- supervisor.go: create->run->recover loop; RecoveryRetries=4 and
  RecoveryBackoff=2s matching the Rust SDK defaults; server-side EOF
  treated as retryable disconnect.

Also adds transport.NewStreamFromRPC and transport.StreamRPC so the
stream package tests can inject fake in-process RPCs without a live
gRPC server.

Signed-off-by: Zlata Stefanovic <zlata.stefanovic@databricks.com>
Signed-off-by: Zlata Stefanovic <zlata.stefanovic@databricks.com>
Serialize offset assignment with enqueue so a failed Ingest (encode error
or ctx-cancelled backpressure) no longer consumes an offset that Flush would
wait on forever. Add a doneCh to the buffer so callers blocked on the inflight
semaphore unblock when the stream closes, and have the supervisor drain-or-close
the buffer based on whether an AckCallback is registered. Ingest on a cleanly
closed stream now returns an error instead of (0, nil).

Signed-off-by: Zlata Stefanovic <zlata.stefanovic@databricks.com>
@zlata-stefanovic-db
zlata-stefanovic-db changed the base branch from main to purego-oauth-provider July 20, 2026 09:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant