[PureGo] Add ingestion core: buffer, encoder, ack model, and supervisor#531
Draft
zlata-stefanovic-db wants to merge 4 commits into
Draft
[PureGo] Add ingestion core: buffer, encoder, ack model, and supervisor#531zlata-stefanovic-db wants to merge 4 commits into
zlata-stefanovic-db wants to merge 4 commits into
Conversation
Adds internal/stream with the generic protocol-agnostic ingestion core: - buffer.go: bounded offset-assigning queue with semaphore backpressure and ctx-cancellable next/enqueue so goroutines stop cleanly per stream rather than per supervisor lifetime. - encoder.go: encoder interface + proto/JSON single-record and batch implementations; encoding is eager at Ingest time so the buffer holds []byte, never live user objects. - ackmodel.go: ackModel interface + offsetAckModel for proto/JSON (DurabilityAckUpToOffset maps directly to the logical offset). - core.go: sender/receiver/supervisor goroutines, ack watermark, Flush/WaitForOffset/Close/GetUnacked. - supervisor.go: create->run->recover loop; RecoveryRetries=4 and RecoveryBackoff=2s matching the Rust SDK defaults; server-side EOF treated as retryable disconnect. Also adds transport.NewStreamFromRPC and transport.StreamRPC so the stream package tests can inject fake in-process RPCs without a live gRPC server. Signed-off-by: Zlata Stefanovic <zlata.stefanovic@databricks.com>
Signed-off-by: Zlata Stefanovic <zlata.stefanovic@databricks.com>
Serialize offset assignment with enqueue so a failed Ingest (encode error or ctx-cancelled backpressure) no longer consumes an offset that Flush would wait on forever. Add a doneCh to the buffer so callers blocked on the inflight semaphore unblock when the stream closes, and have the supervisor drain-or-close the buffer based on whether an AckCallback is registered. Ingest on a cleanly closed stream now returns an error instead of (0, nil). Signed-off-by: Zlata Stefanovic <zlata.stefanovic@databricks.com>
zlata-stefanovic-db
force-pushed
the
purego-core
branch
from
July 20, 2026 09:26
bb63d7a to
83700f8
Compare
On a clean Close the ingestion core now tears the connection down in an orderly CloseSend -> drain-acks-to-EOF -> release sequence so the server observes an END_STREAM rather than an abrupt reset. Because the transport's GracefulClose needs exclusive Recv ownership and the receiver goroutine owns Recv, the sequence runs cooperatively in gracefulTeardown with the receiver as the drain loop, bounded by a new DrainTimeout config knob (default 500ms). The hard abort is retained for failure/recovery paths where the stream is already broken. Signed-off-by: Zlata Stefanovic <zlata.stefanovic@databricks.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fourth in the pure-Go SDK stack. Stacked on #496 (base branch
purego-oauth-provider), though it can also targetmainonce #495/#496 merge.Adds
internal/stream— the generic protocol-agnostic ingestion core that sits between the auth and transport layers and the (not yet built) public API:buffer.go— bounded, offset-assigning queue with semaphore-based backpressure.enqueueandnextare bothcontext.Context-cancellable:enqueueblocks until a slot opens (or ctx fires);nextblocks until an item is queued (or its per-stream context fires), so eachrunOncecan stop its sender goroutine cleanly without cancelling the supervisor's outer context.requeuemoves all in-flight items back to the front of the queue so recovery re-sends them in order. AdoneChunblocks callers parked on the semaphore when the buffer is closed or drained, so a blockedIngestreturns instead of hanging forever.encoder.go—encoderinterface plus four implementations:protoEncoder,jsonEncoder,protoBatchEncoder,jsonBatchEncoder. Encoding is eager atIngesttime so the buffer holds[]byte, never live user objects. Matches the Rust SDK's pattern (LandingZoneholds already-encoded batches).ackmodel.go—ackModelinterface plusoffsetAckModelfor proto/JSON (DurabilityAckUpToOffsetmaps directly to the logical offset). Arrow's record-count ack model slots in here without touching the core.core.go— three goroutines (sender, receiver, supervisor), monotonic ack watermark,Flush/WaitForOffset/Close/GetUnacked. The sender uses a per-runOncederived context so a stream teardown unblocksbuf.next()without cancelling the supervisor context (which would signalClose). Offset assignment is serialized withenqueueunderingestMusonextOffsetonly advances for records that actually make it into the buffer — a failedIngest(encode error or ctx-cancelled backpressure) no longer consumes an offset thatFlushwould wait on forever.Flushtargets the highest successfully-enqueued offset, notnextOffset-1.Ingeston a cleanly closed stream now returns an error instead of(0, nil).supervisor.go—create → run → recoverloop. Server-side EOF is treated as a retryable disconnect. Defaults match the Rust SDK:RecoveryRetries=4,RecoveryBackoff=2s,FlushTimeout=5m,LackOfAckTimeout=60s. On terminal failure it drains the buffer and firesOnErrorper unacked item when anAckCallbackis registered; otherwise it closes the buffer and leaves items retrievable viaGetUnacked(the two are mutually exclusive).Also adds
transport.StreamRPC(exported interface) andtransport.NewStreamFromRPCso stream-package tests can inject in-process fake RPCs without a live gRPC server.Stack
Test plan
go build ./...andgo test ./internal/... -racepass (94 tests, 34 ininternal/stream).TestCoreStreamRecoveryRequeuesUnacked) verifies items in flight when a stream dies are re-sent on the recovery stream and acknowledged correctly.Ingeston a closed stream errors, a failedIngestdoesn't advance theFlushtarget (no permanent gap), and aCloseunblocks anIngestparked at capacity.What's not yet here
zerobuspackage (SDK builder, Stream interface, typed ingest methods)