feat: S3 Tables backend#3666
Draft
bblaszkow06 wants to merge 12 commits into
Draft
Conversation
Adds a Rustler NIF with iceberg-rust library. Working connection test. No-op pipeline based on S3
Fixes the NIFs to be truly async and return right after scheduling work for Tokio runtime. The response is delivered via message from worker thread.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a backend adaptor that writes ingested logs, metrics, and traces to AWS S3 Tables (Apache Iceberg).
Implementation
native/s3_tables_ex) wrapping theiceberg-rustlibrary andiceberg-catalog-s3tables— handles catalog init, table provisioning, and Iceberg snapshot commits. Rows arrive as NDJSON and are decoded into ArrowRecordBatches, then written as Parquet data files.s3_tables_adaptor/pipeline.ex) — one pipeline per backend across all sources. Events are batched by{event_type, day_bucket}so each append targets a single type-specific Iceberg table (otel_logs/otel_metrics/otel_traces) and, typically, one day partition. Failed batches are requeued with a bounded retry counter and dropped once exhausted.CatalogManager— a per-backend GenServer that provisions tables on boot and caches the catalog resource in theRegistryvalue for lock-free hot-path reads (with column-drift warnings for existing tables).Logflare.Mapper.PostProcess, now used by both.Non-blocking NIFs
The Iceberg operations (catalog load, Parquet upload, snapshot commit with retries) are executed on a shared tokio runtime. However, to avoid blocking any scheduler, since Rustler doesn't support yielding NIFs, each NIF returns
:okimmediately. The result is sent via a message from the tokio runner thread to a caller