Skip to content

End GetRecords iterators after closed shard exhaustion - #372

Draft
forhappy wants to merge 1 commit into
ExtendDB:mainfrom
crabbuild:codex/stream-completion
Draft

forhappy wants to merge 1 commit into
ExtendDB:mainfrom
crabbuild:codex/stream-completion

Conversation

@forhappy

Copy link
Copy Markdown

Summary

  • Give StreamEngine::get_stream_records an explicit continuation state so an empty open shard remains pollable and an exhausted closed shard ends its iterator.
  • Update the engine and SQLite, PostgreSQL, and MongoDB producers together. Each producer reads one lookahead record and returns the final page's records before ending the iterator.
  • Document closed-shard behavior and the 1,000-record page limit.

This replaces draft PR #371 with the same commit on the crabbuild-owned fork branch. BeyondDB pins this commit from the crabbuild fork so its release cadence is independent of upstream merge timing.

Verification

  • SQLite open/closed, final-page, and account-isolation unit tests passed.
  • Engine iterator continuation and sequence-padding unit tests passed.
  • PostgreSQL and MongoDB adapters compiled against the changed trait.
  • Strict Clippy for storage, engine, SQLite, PostgreSQL, and MongoDB, plus format and diff checks, passed.
  • BeyondDB's signed process smoke exercises GetShardIterator/GetRecords through the DynamoDB SDK and AWS CLI across a hard restart.

PostgreSQL and MongoDB runtime integration tests remain to be run in their database environments. This changes the storage trait consumed by external backends, so contract review is especially useful.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant