Skip to content

[python] Limit scan#613

Merged
fresh-borzoni merged 1 commit into
apache:mainfrom
fresh-borzoni:feat/python-batch-scanner
Jun 10, 2026
Merged

[python] Limit scan#613
fresh-borzoni merged 1 commit into
apache:mainfrom
fresh-borzoni:feat/python-batch-scanner

Conversation

@fresh-borzoni

@fresh-borzoni fresh-borzoni commented Jun 9, 2026

Copy link
Copy Markdown
Member

closes #612

Python binding for the limit scan from #515. table.new_scan().limit(n).create_bucket_batch_scanner(bucket) returns a BatchScanner with next_batch(), collect_all_batches(), to_arrow() and to_pandas() - a thin wrapper over the core LimitBatchScanner.

Porting projection turned up a core bug: decode_log_batch used projection pushdown, which assumes the server already projected, but limit scans don't project server side so it misread the buffers. Now decodes the full batch and projects after, like the KV path. Added the projection test #515 was missing.

@fresh-borzoni fresh-borzoni force-pushed the feat/python-batch-scanner branch from 3bc9fd3 to 0f38f85 Compare June 9, 2026 17:42
@fresh-borzoni fresh-borzoni marked this pull request as draft June 9, 2026 18:00
@fresh-borzoni fresh-borzoni force-pushed the feat/python-batch-scanner branch 2 times, most recently from c0ae7d3 to da65235 Compare June 9, 2026 22:36
@fresh-borzoni fresh-borzoni marked this pull request as ready for review June 9, 2026 22:37
@fresh-borzoni

Copy link
Copy Markdown
Member Author

@charlesdong1991 charlesdong1991 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice finding!! 👍 thanks!

Comment thread bindings/python/src/table.rs
@fresh-borzoni fresh-borzoni force-pushed the feat/python-batch-scanner branch from da65235 to 7c36bdb Compare June 10, 2026 22:28
@fresh-borzoni fresh-borzoni merged commit d071c90 into apache:main Jun 10, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[python] Implement BatchScanner with limit-based scan

2 participants