diff --git a/.claude/todos/00-roadmap.md b/.claude/todos/00-roadmap.md index 5369e949..ce68487e 100644 --- a/.claude/todos/00-roadmap.md +++ b/.claude/todos/00-roadmap.md @@ -114,10 +114,17 @@ here when the ticket is deleted. Flink's converters exactly (divergences/21). The residual tail lives in ticket 32 (CSV/JSON *file* sources and the smaller CDC follow-ups). - **Nexmark matrix vs Flink — running.** The full q0–q22 suite (q6 excluded, wontdos/39) runs - native-substituted vs stock Flink across four source rungs (generator, Kafka JSON/Avro/Protobuf); + native-substituted vs stock Flink across the source rungs (generator, Parquet, Kafka + JSON/Avro/Protobuf, and the opt-in Fluss rung below); per-query routed-fraction/fallback reasons via `NexmarkExplainTest`, throughput matrix in the readme. It is the standing prioritization + regression gate; the coverage/perf gaps it surfaces feed the backlog below. +- **Native Fluss log-table source — done** (was ticket 36). Fluss (columnar streaming storage) is the + opt-in fourth source rung of the Nexmark matrix (`SF_MATRIX_FLUSS=true`): the native fluss-rs + log-table reader vs stock Flink-on-Fluss, same steelman perimeter — the columnar-on-the-wire source + the perimeter-transpose hypothesis called for. Requires the `fluss` cargo feature (a pinned + arrow-aligned fork of apache/fluss-rust until the bump is upstreamed — `native/Cargo.toml`). + Method + build line in `docs/benchmarks.md`; numbers pending a full run. ## Next, roughly in order (re-prioritized 2026-07-04 after the operator profiling round) @@ -171,7 +178,8 @@ here when the ticket is deleted. the exchange split — OVER +121–162%, retracting Top-N +228%, exchange +208% on Criterion; session `update` batches gap-connected runs — dense shape 20× vs per-row; the `RowData → Arrow` transpose was made row-major + pre-sized, ~25% faster; a native decoder was investigated and - rejected on benchmark grounds — wontdos/28.) + rejected on benchmark grounds — wontdos/28.) Native Fluss follow-up: coalesce small fluss-rs + scanner batches before JNI export if profiles show per-batch overhead inside columnar pipelines. ## Production-readiness (not yet load-bearing) - **Memory accounting**: shipped for every stateful native operator (mini-batch local pre-aggregate @@ -204,15 +212,6 @@ here when the ticket is deleted. decode; raw consume 1.21x the Java client). Remaining before the `kafkaSource` gate can default on: per-partition watermarks/idleness, specific-offsets / topic-pattern startup, key.format, SASL/SSL build features, Linux `mimalloc` link-alias verification, multi-broker measurement. -- **Nexmark with Apache Fluss as the source** (ticket 36): add Fluss (columnar streaming storage) as a - fourth source in the Nexmark matrix; its columnar format may let the native island ingest Arrow with - little/no row transpose — the perimeter transpose is a visible share of the remaining stateful-query - cost, so Fluss is the source most likely to show the engine's largest end-to-end margin. -- **Native Fluss log source** (ticket 44): the Kafka pattern mirrored onto Fluss — reuse its Flink - enumerator (dynamic partition discovery included) verbatim, swap the split reader for a JNI reader - over fluss-rust's Arrow batch scanner. Fluss's log is Arrow on the wire, so ingest can be - zero-decode/zero-transpose; feasibility confirmed 2026-07-05 - (`.claude/research/fluss-native-source-findings.md`). Enables ticket 36's measurement. - **Native lookup join** (ticket 40): **coverage DONE** (2026-07-03). The native operators drive Flink's own generated lookup runners over each Arrow probe batch, so a processing-time lookup join (Nexmark q13) — INNER + LEFT, sync + async, field-ref *and constant* keys, pre-filter, dim-side diff --git a/.claude/todos/11-arroyo-operator-coverage.md b/.claude/todos/11-arroyo-operator-coverage.md index a9478d78..a695b2bd 100644 --- a/.claude/todos/11-arroyo-operator-coverage.md +++ b/.claude/todos/11-arroyo-operator-coverage.md @@ -86,8 +86,8 @@ is picked up. Operators are in `~/data/arroyo/crates/arroyo-worker/src/arrow/`. ## Connectors (later) - [x] Parquet source + sink (local `file:`), exactly-once sink. - [ ] Remote/columnar sources and sinks (Iceberg, `hdfs:`/`s3:`) — ticket 24. -- [ ] Native columnar sources (Fluss / Iceberg-CDC) via Flink Source API + - availability futures — ticket 01. +- [ ] Native columnar sources beyond Fluss (Iceberg-CDC) via Flink Source API + + availability futures — ticket 01. (The native Fluss log-table source is done.) ## Cross-cutting (do alongside, not after) - Acceleration config (allowIncompatible / master / per-operator flags) — done (`NativeConfig`) diff --git a/.claude/todos/36-nexmark-fluss-source.md b/.claude/todos/36-nexmark-fluss-source.md deleted file mode 100644 index 6ac89c97..00000000 --- a/.claude/todos/36-nexmark-fluss-source.md +++ /dev/null @@ -1,32 +0,0 @@ -# Run the Nexmark matrix with Apache Fluss as the source - -**Status:** TODO (not started). Requested 2026-06-30. - -Add **Apache Fluss** as a fourth source in `NexmarkMatrixBenchmark`, alongside the generator and the -three Kafka formats (json/avro/protobuf). Fluss is a streaming-native storage layer (Apache incubating, -ex-Alibaba/Ververica) with a first-class Flink connector and a **columnar** log/at-rest format. - -**Why it's the interesting source for this engine.** Every source we benchmark today hands Flink -rowwise `RowData`, so the native island pays a `RowData → Arrow` transpose at ingest (and `Arrow → -RowData` at the sink). The differential profiles of the stateful queries show that perimeter transpose -is now a real share of the remaining cost once the operator bodies are columnar (e.g. q20 ~34%). Because -Fluss is columnar on the wire / at rest, it's the source most likely to let the native island ingest -**Arrow batches with little or no row transpose** — potentially StreamFusion's largest end-to-end margin -vs stock Flink. Worth measuring whether the Fluss connector can be read straight into Arrow (a native -decode/scan, like the Parquet/ORC file sources) rather than through `RowData`. - -**Shape of the work.** -- A `fluss`-connector source table in the matrix harness (a Fluss test cluster/container, mirroring the - Kafka Testcontainers setup), producing the same wide Nexmark event row. -- Compare native vs **stock Flink-on-Fluss** as the baseline, same steelman perimeter (rowwise sink, - object reuse on both engines) so the comparison is honest. -- Investigate a native columnar read path for Fluss (skip the `RowData` transpose at ingest) — that is - the point of the exercise; if it still has to go through `RowData`, note that and measure anyway. - **Update 2026-07-05: confirmed feasible and designed — ticket 44 builds that native source** (Fluss's - log is Arrow on the wire; fluss-rust exposes it as arrow-rs batches). Benchmark this ticket once 44 - lands, comparing the native source rung against stock Flink-on-Fluss. -- Report alongside the generator + Kafka rungs in the same table. - -Relates to: the Nexmark matrix (a fourth source rung; see roadmap "Where we are"), ticket 32 (native -decode to Arrow at ingest), ticket 24 (columnar endpoints beyond local Parquet), ticket 34 (columnar -sinks / nested boundary types). diff --git a/.claude/todos/37-disaggregated-state-store.md b/.claude/todos/37-disaggregated-state-store.md index 3d93be30..8f880dfd 100644 --- a/.claude/todos/37-disaggregated-state-store.md +++ b/.claude/todos/37-disaggregated-state-store.md @@ -20,7 +20,8 @@ Flink-native client. An operator's keyed state maps cleanly onto a Fluss PK tabl `(operator, key)`. Crucially, the state we now hold is **already arrow-row bytes** (memcomparable key + value-encoded payload — from the Top-N/join/dedup/group-by refactors): that byte layout is exactly what a remote KV store wants on the wire, so the encode/decode is mostly already paid. Pairs naturally with -ticket 36 (Fluss as a *source*) — one Fluss dependency serving both ingest and state. +the native Fluss source (shipped — the Nexmark matrix's Fluss rung) — one Fluss dependency serving both +ingest and state. **Sketch of the work (to be designed, not prescribed):** - A state-access abstraction over the operators' maps (get/put/delete/scan by key) so the in-memory map @@ -35,5 +36,5 @@ ticket 36 (Fluss as a *source*) — one Fluss dependency serving both ingest and object-store + LSM (Hummock/ForSt-style) the better primitive with Fluss only for the changelog? Benchmark state-heavy Nexmark (q4/q9/q16 — large keyspaces) under disaggregated vs in-memory before committing. -Relates to: ticket 36 (Fluss source), memory accounting (shipped — divergences/16), ticket 01 (mailbox/async), +Relates to: the native Fluss source (shipped), memory accounting (shipped — divergences/16), ticket 01 (mailbox/async), and the changelog-operator byte-state refactors (the on-the-wire format is ready). diff --git a/.claude/todos/44-native-fluss-log-source.md b/.claude/todos/44-native-fluss-log-source.md deleted file mode 100644 index ae875366..00000000 --- a/.claude/todos/44-native-fluss-log-source.md +++ /dev/null @@ -1,78 +0,0 @@ -# Native Fluss log source (Rust scanner → Arrow, zero-transpose ingest) - -**Status:** designed, not started. Feasibility investigated 2026-07-05 — -`.claude/research/fluss-native-source-findings.md` holds the full findings (class/crate -references, discovery mechanics, capability matrix). Requested as "a Fluss log source that -works in practice very similar to Kafka". - -## Why - -Fluss log tables are Arrow on the wire. The stock connector decodes to `ColumnarRow` and then -converts per record to `RowData`; our native reader can hand the batch across the C Data -Interface untouched — the first source with a ~zero ingest perimeter (no decode, no -RowData→Arrow transpose). Ticket 36 (Nexmark-on-Fluss) is the measurement of exactly this. - -## Shape: the Kafka pattern, one-to-one - -Mirror `NativeKafkaSource`: reuse Fluss's JobManager-side coordination verbatim, swap only the -per-subtask reader. - -- **Reuse verbatim** (all public, directly instantiable): `FlinkSourceEnumerator`, - `SourceSplitSerializer` (null LakeSource), `SourceEnumeratorState` + its serializer, the - split types. Our `Source` impl delegates `createEnumerator`/serializers exactly like - `NativeKafkaSource` does with `KafkaSourceEnumerator`. -- **Replace** `FlinkSourceSplitReader` with a native split reader: JNI handle over a - fluss-rust `RecordBatchLogScanner`; `SplitsAddition` → `subscribe_partition(partitionId, - bucket, startingOffset)` / `subscribe(bucket, offset)`; poll returns per-bucket Arrow - batches → per-split `ArrowBatch` records (split state advances per bucket, as in - `NativeKafkaSplitReader.fetch()`). -- **Dynamic partition discovery comes free**: the reused enumerator polls - `listPartitionInfos` every `scan.partition.discovery.interval` (default 1 min) and assigns - new-partition splits (FLIP-288: post-first-round partitions start EARLIEST). fluss-rust - subscriptions are an RwLock map separate from the poll path, so incremental mid-poll - subscribe works — same reconcile shape as the Kafka reader, except Fluss's subscribe is - already incremental (no full-assignment replace, no offset-tracking union needed natively). -- **Partition removal (no Kafka analog)**: dropped partitions arrive as - `PartitionsRemovedEvent`; the reader must unsubscribe those buckets and ack with - `PartitionBucketsUnsubscribedEvent` (the fetcher manager's `removePartitions` path). The - native reader needs `unsubscribe` plumbing plus split-state cleanup so checkpoints stop - carrying removed splits. - -## Scope (first cut) - -- **Append-only log tables, ARROW log format** (the only format fluss-rust reads), streaming - mode. Startup modes: full/earliest, latest, timestamp — all map to subscribe offsets or the - admin `list_offsets(Earliest|Latest|Timestamp)` the enumerator resolves anyway. -- Projection pushdown: wire `SupportsProjectionPushDown` → `TableScan::project` (server-side - pruning on the local path; remote/tiered reads decode full schema — prune client-side, note - in coverage doc). -- Watermarks: same per-split regeneration we built for Kafka (per-bucket batch-max rowtime). -- **Fall back** on: PK tables (`HybridSnapshotLogSplit` — fluss-rust has no snapshot-bootstrap - scanner; changelog-only reads exist in record mode but the Arrow surface rejects PK), lake - union reads, bounded/batch mode, non-PLAIN auth / TLS, non-default ZSTD level, `INDEXED` - log format, any Fluss type our boundary schema can't carry. Matcher must mirror the - connector's schema `sanityCheck`. - -## Known frictions - -- **arrow-rs 57 (fluss-rust) vs 58 (ours)**: try bumping fluss-rust to 58 (upstream if it - compiles clean); otherwise bridge zero-copy via Arrow C FFI inside the native lib - (fluss-rust ships a sync `RecordBatchReader` FFI adapter built for this). -- fluss-rust spawns untracked fetch tasks (no cancel-on-drop) — verify clean task shutdown. -- Parity referee: harness comparing native output vs the stock Fluss connector on the same - table (Testcontainers Fluss cluster), including a partition created mid-job (discovery) and - a partition dropped mid-job (removal round trip), changelog kinds (+A only for log tables), - and every startup mode. - -## Sequencing - -1. Bump/bridge arrow; JNI binding over `RecordBatchLogScanner` (model: the cpp binding's - C ABI + Arrow C Data surface). -2. Source + split reader + planner matcher (`connector = 'fluss'`), fall-back gates, coverage - doc section. -3. Parity harness incl. dynamic discovery/removal scenarios. -4. Ticket 36: Nexmark matrix rung on Fluss — the payoff measurement (native vs stock - Flink-on-Fluss, honest perimeter). - -Relates to: ticket 36 (the benchmark this enables), ticket 33 (the Kafka pattern being -mirrored), ticket 24 (columnar endpoints), ticket 37 (Fluss PK KV as a state-store candidate). diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index ca13521c..470df046 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -64,3 +64,44 @@ jobs: # hosted runners' preinstalled Docker daemon serves directly. - name: Run Java test suite run: mvn -B -ntp test + + # The jobs above build without `--features fluss`, so Native.flussFeatureBuilt() is false and + # every @EnabledIf-gated Fluss test silently skips there. This leg builds the native library + # with the fluss feature and actually runs them: the Rust unit tests under the feature, then + # the Fluss JUnit classes against a fluss-feature debug build (the Fluss harness uses an + # in-process test cluster, no Docker needed). + fluss-tests: + name: Fluss feature (Rust + Java tests) + runs-on: ubuntu-24.04 + timeout-minutes: 150 + steps: + - uses: actions/checkout@v6 + + # Same reason as java-tests: a cold debug build of the native library plus the + # Maven repository outgrows the runner's default free space. + - name: Free disk space + run: | + sudo rm -rf /usr/share/dotnet /usr/local/lib/android /opt/ghc /usr/local/.ghcup + df -h / + + - uses: actions/setup-java@v4 + with: + distribution: temurin + java-version: "17" + cache: maven + + - uses: dtolnay/rust-toolchain@stable + + - name: Install protoc + run: sudo apt-get update && sudo apt-get install -y protobuf-compiler + + - uses: Swatinem/rust-cache@v2 + with: + workspaces: native + + - name: Run native test suite (fluss feature) + working-directory: native + run: cargo test --features fluss + + - name: Run Fluss Java tests (fluss-feature native build) + run: mvn -B -ntp -Dnative.cargo.args="build --features fluss" -Dtest='*Fluss*' test diff --git a/docs/benchmarks.md b/docs/benchmarks.md index 5227639a..bc25caed 100644 --- a/docs/benchmarks.md +++ b/docs/benchmarks.md @@ -316,7 +316,8 @@ across the ladder — all vs stock Flink, same steelmanned perimeter. 500K event `SF_BENCHMARK=true mvn test -Pbench -Dnative.cargo.args="build --release --features mimalloc" -Dtest=NexmarkMatrixBenchmark` (Testcontainers Kafka; the `kafka` feature is a build default, and `mimalloc` — the recommended build — rebinds the library's allocator, divergences/19). Column -toggles: `SF_MATRIX_GENERATOR` / `SF_MATRIX_PARQUET` / `SF_MATRIX_KAFKA` (`false` skips one). +toggles: `SF_MATRIX_GENERATOR` / `SF_MATRIX_PARQUET` / `SF_MATRIX_KAFKA` (`false` skips one), plus +`SF_MATRIX_FLUSS` (`true` *adds* the opt-in Fluss rung — off by default; see below). The matrix runs with the native managed-memory cap **in force**: the shared test cluster declares a deployment-like managed-memory size (flink-test-utils' default gave each slot ~10 MB, which the @@ -490,6 +491,19 @@ q3/q14/q18/q21, whose JSON rows were below 1× on their old best rung and now si of the table is q16 and the changelog-bound q3/q19 — operator-bound queries where the consume saving is diluted, not reversed. +**Fluss** — the opt-in fourth source rung (`SF_MATRIX_FLUSS=true`), the columnar-on-the-wire +source: the same wide event row is preloaded into a local Fluss test cluster and read back by +both engines in the identical default streaming runtime — stock Flink-on-Fluss vs the native +fluss-rs log-table reader. Boundedness comes from a count-N-then-cancel collecting sink, so each +cell measures time-to-Nth-row at `SF_ROWS` scale. The native reader requires the `fluss` cargo +feature in the build, added alongside the recommended `mimalloc`: `SF_BENCHMARK=true +SF_MATRIX_FLUSS=true mvn test -Pbench -Dnative.cargo.args="build --release --features +mimalloc,fluss" -Dtest=NexmarkMatrixBenchmark`. Building the `fluss` feature currently needs +`protoc` (`protobuf-compiler`) because fluss-rs generates its RPC protos at build time. + +_Fluss numbers are pending a full run; the rung's column will be reported here alongside the +tables above once it lands._ + ### The tuned (mini-batch) matrix — the full suite Production Flink deployments routinely enable mini-batch for stateful queries, so the matrix has a diff --git a/docs/coverage-and-fallbacks.md b/docs/coverage-and-fallbacks.md index f4a02561..307bec4f 100644 --- a/docs/coverage-and-fallbacks.md +++ b/docs/coverage-and-fallbacks.md @@ -142,7 +142,8 @@ array`, is **not** here: Flink rejects it too, so we're at parity.) batch's I/O. Still falls back: an **upsert-materialized** (keyed-state) lookup. - **Sources/sink** — local `file:` path only (Parquet/ORC source, Parquet sink); Kafka decode limited (see below); CDC covers the four JSON dialects (Debezium/OGG/Maxwell/Canal — the latter two for - flat scalar schemas). + flat scalar schemas); Fluss log tables via the native fluss-rs scanner (ARROW log format, a + verified scalar-type whitelist — see §5). - **Proctime** support, by operator: - **Deduplication** and **`OVER`** (running / bounded-ROWS) — native; they emit eagerly in arrival order, no wall-clock timer needed. @@ -190,7 +191,8 @@ array`, is **not** here: Flink rejects it too, so we're at parity.) `localGroupAggregate`, `miniBatchAssigner`, `lookupJoin`, …). The two-phase global half reuses the `groupAggregate` switch. All default on, `kafkaSource` included — the planner probes whether the loaded native library carries the (default) `kafka` cargo feature and falls back to the decode - path on an opt-out build. + path on an opt-out build. `flussSource` probes the same way for the **opt-in** `fluss` cargo + feature and falls back to the stock Fluss connector when the library was built without it. ### 2. Per-operator matcher declines (exact conditions) - **OVER** — a frame not of the form `… PRECEDING .. CURRENT ROW` (a `ROWS`/`RANGE` lower bound that @@ -435,3 +437,41 @@ array`, is **not** here: Flink rejects it too, so we're at parity.) columns** (findValue's recursive search could false-match a column name inside another field — flat scalar schemas only, up to 128 columns); Canal's `database.include`/`table.include` regex filters; and `debezium-avro-confluent` (Avro-bodied envelope). +- **Fluss** — the native source (fluss-rs' `RecordBatchLogScanner`) replaces a Fluss **log-table** + scan behind a two-level gate: the `flussSource` operator switch + (`-Dstreamfusion.operator.flussSource.enabled`, default on) **and** a native library built with + the **opt-in** `fluss` cargo feature (`-Dnative.cargo.args="build --features fluss"` — the planner + probes `flussFeatureBuilt` and records `the native library was built without the fluss feature` + on a build without it). Coordination stays on the JVM by design — split assignment, startup-offset + resolution, partition discovery/acknowledgement, snapshot leases, and checkpointing run in the + Flink-side enumerator, so `scan.startup.mode`/`scan.startup.timestamp`, + `scan.partition.discovery.interval`, and `scan.kv.snapshot.lease.*` are honored there and never + translated into the native config (streaming and bounded scans, static and dynamically discovered + partitions, and column projection are all native). Still falling back, each recorded as + `fluss source: `: + - **Primary-key tables** — they read a changelog the native log scanner does not carry + (append-only log tables only). + - **Datalake-enabled tables** — a `lakeSource` reads through the lake, not the log. + - **Pushdown the native reader can't honor** — a pushed single-row filter, a modification scan + type, a row-count scan, a pushed-down `LIMIT`, pushed partition filters, an empty projection; + plus metadata/computed columns (not produced natively). + - **A `WATERMARK` clause** — like Kafka, Flink pushes it into the scan, and the native Fluss + source does not regenerate watermarks. + - **A column type outside the verified whitelist** — BOOLEAN, TINYINT, SMALLINT, INT, BIGINT, + FLOAT, DOUBLE, CHAR, VARCHAR, DECIMAL, DATE, TIME, plain TIMESTAMP, VARBINARY, and nested ROWs + whose leaves are also whitelisted: the intersection of fluss-rs' Arrow export and what the + vendored `ArrowConversion` readers accept, parity-pinned by `NativeFlussTypeParityTest`. + Notable exclusions: **TIMESTAMP_LTZ** (fluss-rs exports a zoned timestamp vector, which + `ArrowConversion` currently rejects), **BINARY** (`ArrowConversion.toArrowSchema` has no + BINARY mapping), and **nested ARRAY/MAP** (unverified across the boundary). + - **`table.log.format` other than `ARROW`** — fluss-rs' scan validation errors on any other log + format. + - **Client config the native client can't mirror** — an unrecognized `client.*` option; a known + option with no fluss-rs `Config` field (`client.id`, `client.request-timeout`, + `client.scanner.log.check-crc`, `client.scanner.io.tmpdir`, `client.metrics.enabled`, + `client.security.sasl.jaas.config`); a `client.security.protocol` other than PLAINTEXT or SASL; + a SASL mechanism other than PLAIN. `client.writer.*`/`client.lookup.*` options are **ignored**, + not fallbacks — a read-only source never runs the write or lookup paths. + + Every decline above surfaces through the usual channel — `NativePlanner.explain(...)` / + `-Dstreamfusion.logFallbackReasons=true`. diff --git a/native/Cargo.lock b/native/Cargo.lock index b104c0c1..bcc41ed4 100644 --- a/native/Cargo.lock +++ b/native/Cargo.lock @@ -67,12 +67,56 @@ version = "0.1.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "4b46cbb362ab8752921c97e041f5e366ee6297bd428a31275b9fcf1e380f7299" +[[package]] +name = "anstream" +version = "1.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "824a212faf96e9acacdbd09febd34438f8f711fb84e09a8916013cd7815ca28d" +dependencies = [ + "anstyle", + "anstyle-parse", + "anstyle-query", + "anstyle-wincon", + "colorchoice", + "is_terminal_polyfill", + "utf8parse", +] + [[package]] name = "anstyle" version = "1.0.14" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "940b3a0ca603d1eade50a4846a2afffd5ef57a9feac2c0e2ec2e14f9ead76000" +[[package]] +name = "anstyle-parse" +version = "1.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "52ce7f38b242319f7cabaa6813055467063ecdc9d355bbb4ce0c68908cd8130e" +dependencies = [ + "utf8parse", +] + +[[package]] +name = "anstyle-query" +version = "1.1.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "40c48f72fd53cd289104fc64099abca73db4166ad86ea0b4341abe65af83dadc" +dependencies = [ + "windows-sys 0.61.2", +] + +[[package]] +name = "anstyle-wincon" +version = "3.0.11" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "291e6a250ff86cd4a820112fb8898808a366d8f9f58ce16d1f538353ad55747d" +dependencies = [ + "anstyle", + "once_cell_polyfill", + "windows-sys 0.61.2", +] + [[package]] name = "anyhow" version = "1.0.103" @@ -169,11 +213,11 @@ dependencies = [ "flate2", "indexmap", "liblzma", - "rand", + "rand 0.9.4", "serde", "serde_json", "snap", - "strum_macros", + "strum_macros 0.28.0", "uuid", "zstd", ] @@ -313,7 +357,7 @@ version = "58.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f633dbfdf39c039ada1bf9e34c694816eb71fbb7dc78f613993b7245e078a1ed" dependencies = [ - "bitflags", + "bitflags 2.13.0", "serde_core", "serde_json", ] @@ -381,12 +425,29 @@ dependencies = [ "num-traits", ] +[[package]] +name = "atomic-waker" +version = "1.1.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1505bd5d3d116872e7271a6d4e16d81d0c8570876c8de68093a09ac269d8aac0" + [[package]] name = "autocfg" version = "1.5.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f2032f911046de80f0a198e0901378627c33f59ea0ac00e363d481118bd70a53" +[[package]] +name = "backon" +version = "1.6.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cffb0e931875b666fc4fcb20fee52e9bbd1ef836fd9e9e04ec21555f9f85f7ef" +dependencies = [ + "fastrand", + "gloo-timers", + "tokio", +] + [[package]] name = "base64" version = "0.22.1" @@ -404,14 +465,33 @@ dependencies = [ "num-bigint", "num-integer", "num-traits", + "serde", ] +[[package]] +name = "bitflags" +version = "1.3.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bef38d45163c2f1dde094a7dfd33ccf595c92905c8f8f4fdc18d06fb1037718a" + [[package]] name = "bitflags" version = "2.13.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b4388bee8683e3d04af747c73422af53102d2bd24d9eadb6cbc100baef4b43f8" +[[package]] +name = "bitvec" +version = "1.1.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ddcec3d12c579d40898fe0a9a358a803c23e9c52ca3c425707f81c9436211837" +dependencies = [ + "funty", + "radium", + "tap", + "wyz", +] + [[package]] name = "blake2" version = "0.10.6" @@ -528,6 +608,23 @@ version = "1.0.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9330f8b2ff13f34540b44e946ef35111825727b38d33286ef986142615121801" +[[package]] +name = "cfg_aliases" +version = "0.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "613afe47fcd5fac7ccf1db93babcb082c5994d996f20b8b159f2ad1658eb5724" + +[[package]] +name = "chacha20" +version = "0.10.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d524456ba66e72eb8b115ff89e01e497f8e6d11d78b70b1aa13c0fbd97540a81" +dependencies = [ + "cfg-if", + "cpufeatures 0.3.0", + "rand_core 0.10.1", +] + [[package]] name = "chrono" version = "0.4.45" @@ -585,6 +682,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1ddb117e43bbf7dacf0a4190fef4d345b9bad68dfc649cb349e7d17d28428e51" dependencies = [ "clap_builder", + "clap_derive", ] [[package]] @@ -593,8 +691,22 @@ version = "4.6.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "714a53001bf66416adb0e2ef5ac857140e7dc3a0c48fb28b2f10762fc4b5069f" dependencies = [ + "anstream", "anstyle", "clap_lex", + "strsim", +] + +[[package]] +name = "clap_derive" +version = "4.6.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f2ce8604710f6733aa641a2b3731eaa1e8b3d9973d5e3565da11800813f997a9" +dependencies = [ + "heck", + "proc-macro2", + "quote", + "syn", ] [[package]] @@ -603,6 +715,12 @@ version = "1.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "c8d4a3bb8b1e0c1050499d1815f5ab16d04f0959b233085fb31653fbfc9d98f9" +[[package]] +name = "colorchoice" +version = "1.0.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1d07550c9036bf2ae0c684c4297d503f838287c83c53686d05370d0e139ae570" + [[package]] name = "combine" version = "4.6.7" @@ -709,6 +827,15 @@ version = "2.5.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "217698eaf96b4a3f0bc4f3662aaa55bdf913cd54d7204591faa790070c6d0853" +[[package]] +name = "crc32c" +version = "0.6.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3a47af21622d091a8f0fb295b88bc886ac74efcc613efc19f5d0b21de5c89e47" +dependencies = [ + "rustc_version", +] + [[package]] name = "crc32fast" version = "1.5.0" @@ -875,7 +1002,7 @@ dependencies = [ "object_store", "parking_lot", "parquet", - "rand", + "rand 0.9.4", "regex", "sqlparser", "tempfile", @@ -997,7 +1124,7 @@ dependencies = [ "liblzma", "log", "object_store", - "rand", + "rand 0.9.4", "tokio", "tokio-util", "url", @@ -1129,7 +1256,7 @@ dependencies = [ "log", "object_store", "parking_lot", - "rand", + "rand 0.9.4", "tempfile", "url", ] @@ -1195,7 +1322,7 @@ dependencies = [ "md-5", "memchr", "num-traits", - "rand", + "rand 0.9.4", "regex", "sha2", "unicode-segmentation", @@ -1509,6 +1636,48 @@ dependencies = [ "sqlparser", ] +[[package]] +name = "defmt" +version = "1.1.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e2953bfe4f93bbd20cc71198842756f77d161884c99ebbabc41d80231ded88d1" +dependencies = [ + "bitflags 1.3.2", + "defmt-macros", +] + +[[package]] +name = "defmt-macros" +version = "1.1.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bad9c72e7ca2137e0dc3813245a0d282fd6daad32fd800af018306a9169b5fe8" +dependencies = [ + "defmt-parser", + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "defmt-parser" +version = "1.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "10d60334b3b2e7c9d91ef8150abfb6fa4c1c39ebbcf4a81c2e346aad939fee3e" +dependencies = [ + "thiserror 2.0.18", +] + +[[package]] +name = "delegate" +version = "0.13.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "780eb241654bf097afb00fc5f054a09b687dad862e485fdcf8399bb056565370" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + [[package]] name = "digest" version = "0.10.7" @@ -1543,6 +1712,17 @@ version = "1.0.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "877a4ace8713b0bcf2a4e7eec82529c029f1d0619886d18145fea96c3ffe5c0f" +[[package]] +name = "erased-serde" +version = "0.4.10" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d2add8a07dd6a8d93ff627029c51de145e12686fbc36ecb298ac22e74cf02dec" +dependencies = [ + "serde", + "serde_core", + "typeid", +] + [[package]] name = "errno" version = "0.3.14" @@ -1583,7 +1763,7 @@ version = "25.12.19" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "35f6839d7b3b98adde531effaf34f0c2badc6f4735d26fe74709d8e513a96ef3" dependencies = [ - "bitflags", + "bitflags 2.13.0", "rustc_version", ] @@ -1607,6 +1787,46 @@ dependencies = [ "num-traits", ] +[[package]] +name = "fluss-rs" +version = "0.2.0" +source = "git+https://github.com/michaelmitchell-bit/fluss-rust.git?rev=7182719652d5f6227f15611f888430d74f2eb26b#7182719652d5f6227f15611f888430d74f2eb26b" +dependencies = [ + "arrow", + "arrow-schema", + "bigdecimal", + "bitvec", + "byteorder", + "bytes", + "clap", + "crc32c", + "dashmap", + "delegate", + "futures", + "jiff", + "linked-hash-map", + "log", + "metrics", + "opendal", + "ordered-float 5.3.0", + "parking_lot", + "parse-display", + "prost 0.14.4", + "prost-build", + "rand 0.9.4", + "scopeguard", + "serde", + "serde_json", + "snafu", + "strum", + "strum_macros 0.26.4", + "tempfile", + "thiserror 1.0.69", + "tokio", + "url", + "uuid", +] + [[package]] name = "foldhash" version = "0.1.5" @@ -1628,6 +1848,12 @@ dependencies = [ "percent-encoding", ] +[[package]] +name = "funty" +version = "2.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e6d5a32815ae3f33302d95fdcb2ce17862f8c65363dcfd29360480ba1001fc9c" + [[package]] name = "futures" version = "0.3.32" @@ -1733,8 +1959,10 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ff2abc00be7fca6ebc474524697ae276ad847ad0a6b3faa4bcb027e9a4614ad0" dependencies = [ "cfg-if", + "js-sys", "libc", "wasi", + "wasm-bindgen", ] [[package]] @@ -1756,8 +1984,11 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "300e883d756b2e4ec94e02791f39b04b522276138852cfc41d9fb7e904106099" dependencies = [ "cfg-if", + "js-sys", "libc", "r-efi 6.0.0", + "rand_core 0.10.1", + "wasm-bindgen", ] [[package]] @@ -1766,6 +1997,18 @@ version = "0.3.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "0cc23270f6e1808e30a928bdc84dea0b9b4136a8bc82338574f23baf47bbd280" +[[package]] +name = "gloo-timers" +version = "0.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bbb143cf96099802033e0d4f4963b19fd2e0b728bcf076cd9cf7f6634f092994" +dependencies = [ + "futures-channel", + "futures-core", + "js-sys", + "wasm-bindgen", +] + [[package]] name = "half" version = "2.7.1" @@ -1848,12 +2091,100 @@ dependencies = [ "itoa", ] +[[package]] +name = "http-body" +version = "1.0.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1efedce1fb8e6913f23e0c92de8e62cd5b772a67e7b3946df930a62566c93184" +dependencies = [ + "bytes", + "http", +] + +[[package]] +name = "http-body-util" +version = "0.1.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b021d93e26becf5dc7e1b75b1bed1fd93124b374ceb73f43d4d4eafec896a64a" +dependencies = [ + "bytes", + "futures-core", + "http", + "http-body", + "pin-project-lite", +] + +[[package]] +name = "httparse" +version = "1.10.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6dbf3de79e51f3d586ab4cb9d5c3e2c14aa28ed23d180cf89b4df0454a69cc87" + [[package]] name = "humantime" version = "2.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "135b12329e5e3ce057a9f972339ea52bc954fe1e9358ef27f95e89716fbc5424" +[[package]] +name = "hyper" +version = "1.10.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "55281c53a1894c864990125767da440a4e630446785086f52523b20033b74498" +dependencies = [ + "atomic-waker", + "bytes", + "futures-channel", + "futures-core", + "http", + "http-body", + "httparse", + "itoa", + "pin-project-lite", + "smallvec", + "tokio", + "want", +] + +[[package]] +name = "hyper-rustls" +version = "0.27.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "33ca68d021ef39cf6463ab54c1d0f5daf03377b70561305bb89a8f83aab66e0f" +dependencies = [ + "http", + "hyper", + "hyper-util", + "rustls", + "tokio", + "tokio-rustls", + "tower-service", + "webpki-roots", +] + +[[package]] +name = "hyper-util" +version = "0.1.20" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "96547c2556ec9d12fb1578c4eaf448b04993e7fb79cbaad930a656880a6bdfa0" +dependencies = [ + "base64", + "bytes", + "futures-channel", + "futures-util", + "http", + "http-body", + "hyper", + "ipnet", + "libc", + "percent-encoding", + "pin-project-lite", + "socket2", + "tokio", + "tower-service", + "tracing", +] + [[package]] name = "iana-time-zone" version = "0.1.65" @@ -1997,6 +2328,12 @@ version = "3.0.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "8bb03732005da905c88227371639bf1ad885cc712789c011c31c5fb3ab3ccf02" +[[package]] +name = "ipnet" +version = "2.12.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d98f6fed1fde3f8c21bc40a1abb88dd75e67924f9cffc3ef95607bad8017f8e2" + [[package]] name = "is-terminal" version = "0.4.17" @@ -2008,6 +2345,12 @@ dependencies = [ "windows-sys 0.61.2", ] +[[package]] +name = "is_terminal_polyfill" +version = "1.70.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a6cb138bb79a146c1bd460005623e142ef0181e3d0219cb493e02f7d08a35695" + [[package]] name = "itertools" version = "0.10.5" @@ -2032,6 +2375,50 @@ version = "1.0.18" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "8f42a60cbdf9a97f5d2305f08a87dc4e09308d1276d28c869c684d7777685682" +[[package]] +name = "jiff" +version = "0.2.31" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ccfe6121cbe750cf81efa362d85c0bde7ea298ec43092d3a193baca59cdbd634" +dependencies = [ + "defmt", + "jiff-static", + "jiff-tzdb-platform", + "js-sys", + "log", + "portable-atomic", + "portable-atomic-util", + "serde_core", + "wasm-bindgen", + "windows-link", +] + +[[package]] +name = "jiff-static" +version = "0.2.31" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e165e897f662d428f3cd3828a919dbe067c2d42bb1031eede74ef9d27ecdedd2" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "jiff-tzdb" +version = "0.1.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6142247df1a93c2b3587402a19710be3e6e942f1581a1702e76408f2c21d6590" + +[[package]] +name = "jiff-tzdb-platform" +version = "0.1.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "875a5a69ac2bab1a891711cf5eccbec1ce0341ea805560dcd90b7a2e925132e8" +dependencies = [ + "jiff-tzdb", +] + [[package]] name = "jni" version = "0.21.1" @@ -2213,6 +2600,12 @@ dependencies = [ "vcpkg", ] +[[package]] +name = "linked-hash-map" +version = "0.5.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0717cef1bc8b636c6e1c1bbdefc09e6322da8a9321966e8928ef80d20f7f770f" + [[package]] name = "linux-raw-sys" version = "0.12.1" @@ -2239,6 +2632,15 @@ name = "log" version = "0.4.33" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "0ceec5bc11778974d1bcb055b18002eba7f4b3518b6a0081b3af5f21666da9ad" +dependencies = [ + "value-bag", +] + +[[package]] +name = "lru-slab" +version = "0.1.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "112b39cec0b298b6c1999fee3e31427f74f676e4cb9879ed1a121b43661a4154" [[package]] name = "lz4_flex" @@ -2285,15 +2687,42 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "88904434abc2901f197fe8cc55f0445e7ded921dba5911dad2e2b39b48e663c4" [[package]] -name = "miniz_oxide" -version = "0.8.9" +name = "metrics" +version = "0.24.6" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1fa76a2c86f704bdb222d66965fb3d63269ce38518b83cb0575fca855ebb6316" +checksum = "89550ee9f79e88fef3119de263694973a8adb26c21d75322164fb8c493039fe2" dependencies = [ - "adler2", + "portable-atomic", + "rapidhash", +] + +[[package]] +name = "miniz_oxide" +version = "0.8.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1fa76a2c86f704bdb222d66965fb3d63269ce38518b83cb0575fca855ebb6316" +dependencies = [ + "adler2", "simd-adler32", ] +[[package]] +name = "mio" +version = "1.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "02bd0af71c67b473010cbbc60715ee815645a4dc942899111f494b4b737d6fda" +dependencies = [ + "libc", + "wasi", + "windows-sys 0.61.2", +] + +[[package]] +name = "multimap" +version = "0.10.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1d87ecb2933e8aeadb3e3a02b828fed80a7528047e68b4f424523a0981a3a084" + [[package]] name = "num" version = "0.4.3" @@ -2431,12 +2860,45 @@ version = "1.21.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9f7c3e4beb33f85d45ae3e3a1792185706c8e16d043238c593331cc7cd313b50" +[[package]] +name = "once_cell_polyfill" +version = "1.70.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "384b8ab6d37215f3c5301a95a4accb5d64aa607f1fcb26a11b5303878451b4fe" + [[package]] name = "oorandom" version = "11.1.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "d6790f58c7ff633d8771f42965289203411a5e5c68388703c06e14f24770b41e" +[[package]] +name = "opendal" +version = "0.55.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d075ab8a203a6ab4bc1bce0a4b9fe486a72bf8b939037f4b78d95386384bc80a" +dependencies = [ + "anyhow", + "backon", + "base64", + "bytes", + "futures", + "getrandom 0.2.17", + "http", + "http-body", + "jiff", + "log", + "md-5", + "percent-encoding", + "quick-xml", + "reqwest", + "serde", + "serde_json", + "tokio", + "url", + "uuid", +] + [[package]] name = "orc-rust" version = "0.8.0" @@ -2473,6 +2935,17 @@ dependencies = [ "num-traits", ] +[[package]] +name = "ordered-float" +version = "5.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b7d950ca161dc355eaf28f82b11345ed76c6e1f6eb1f4f4479e0323b9e2fbd0e" +dependencies = [ + "num-traits", + "rand 0.8.6", + "serde", +] + [[package]] name = "parking_lot" version = "0.12.5" @@ -2532,6 +3005,31 @@ dependencies = [ "zstd", ] +[[package]] +name = "parse-display" +version = "0.10.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "287d8d3ebdce117b8539f59411e4ed9ec226e0a4153c7f55495c6070d68e6f72" +dependencies = [ + "parse-display-derive", + "regex", + "regex-syntax", +] + +[[package]] +name = "parse-display-derive" +version = "0.10.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7fc048687be30d79502dea2f623d052f3a074012c6eac41726b7ab17213616b1" +dependencies = [ + "proc-macro2", + "quote", + "regex", + "regex-syntax", + "structmeta", + "syn", +] + [[package]] name = "paste" version = "1.0.15" @@ -2614,6 +3112,21 @@ dependencies = [ "plotters-backend", ] +[[package]] +name = "portable-atomic" +version = "1.13.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c33a9471896f1c69cecef8d20cbe2f7accd12527ce60845ff44c153bb2a21b49" + +[[package]] +name = "portable-atomic-util" +version = "0.2.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c2a106d1259c23fac8e543272398ae0e3c0b8d33c88ed73d0cc71b0f1d902618" +dependencies = [ + "portable-atomic", +] + [[package]] name = "potential_utf" version = "0.1.5" @@ -2632,6 +3145,16 @@ dependencies = [ "zerocopy", ] +[[package]] +name = "prettyplease" +version = "0.2.37" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "479ca8adacdd7ce8f1fb39ce9ecccbfe93a3f1344b3d0d97f20bc0196208f62b" +dependencies = [ + "proc-macro2", + "syn", +] + [[package]] name = "proc-macro-crate" version = "3.5.0" @@ -2670,6 +3193,25 @@ dependencies = [ "prost-derive 0.14.4", ] +[[package]] +name = "prost-build" +version = "0.14.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "03da047801ff44bb6a4d407d4860c05fd70bb81714e6b2f3812603d5b145b042" +dependencies = [ + "heck", + "itertools 0.10.5", + "log", + "multimap", + "petgraph", + "prettyplease", + "prost 0.14.4", + "prost-types", + "regex", + "syn", + "tempfile", +] + [[package]] name = "prost-derive" version = "0.13.5" @@ -2677,7 +3219,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "8a56d757972c98b346a9b766e3f02746cde6dd1cd1d1d563472929fdd74bec4d" dependencies = [ "anyhow", - "itertools 0.14.0", + "itertools 0.10.5", "proc-macro2", "quote", "syn", @@ -2690,7 +3232,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b570b25f7617e43d59005d0990ccb79e950a423952cea19671b7a876da390adf" dependencies = [ "anyhow", - "itertools 0.14.0", + "itertools 0.10.5", "proc-macro2", "quote", "syn", @@ -2739,6 +3281,72 @@ dependencies = [ "prost-reflect", ] +[[package]] +name = "quick-xml" +version = "0.38.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b66c2058c55a409d601666cffe35f04333cf1013010882cec174a7467cd4e21c" +dependencies = [ + "memchr", + "serde", +] + +[[package]] +name = "quinn" +version = "0.11.11" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0c1a41e437b6bbd489372cd4971de128e85c855f56c57f283d20ff016cf7c0a8" +dependencies = [ + "bytes", + "cfg_aliases", + "pin-project-lite", + "quinn-proto", + "quinn-udp", + "rustc-hash", + "rustls", + "socket2", + "thiserror 2.0.18", + "tokio", + "tracing", + "web-time", +] + +[[package]] +name = "quinn-proto" +version = "0.11.16" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2f4bfc015262b9df63c8845072ce59068853ff5872180c2ce2f13038b970e560" +dependencies = [ + "bytes", + "getrandom 0.4.3", + "lru-slab", + "rand 0.10.2", + "rand_pcg", + "ring", + "rustc-hash", + "rustls", + "rustls-pki-types", + "slab", + "thiserror 2.0.18", + "tinyvec", + "tracing", + "web-time", +] + +[[package]] +name = "quinn-udp" +version = "0.5.15" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "35a133f956daabe89a61a685c2649f13d82d5aa4bd5d12d1277e1072a21c0694" +dependencies = [ + "cfg_aliases", + "libc", + "once_cell", + "socket2", + "tracing", + "windows-sys 0.61.2", +] + [[package]] name = "quote" version = "1.0.46" @@ -2760,6 +3368,22 @@ version = "6.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f8dcc9c7d52a811697d2151c701e0d08956f92b0e24136cf4cf27b57a6a0d9bf" +[[package]] +name = "radium" +version = "0.7.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "dc33ff2d4973d518d823d61aa239014831e521c75da58e3df4840d3f47749d09" + +[[package]] +name = "rand" +version = "0.8.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5ca0ecfa931c29007047d1bc58e623ab12e5590e8c7cc53200d5202b69266d8a" +dependencies = [ + "rand_core 0.6.4", + "serde", +] + [[package]] name = "rand" version = "0.9.4" @@ -2767,7 +3391,18 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "44c5af06bb1b7d3216d91932aed5265164bf384dc89cd6ba05cf59a35f5f76ea" dependencies = [ "rand_chacha", - "rand_core", + "rand_core 0.9.5", +] + +[[package]] +name = "rand" +version = "0.10.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c7f5fa3a058cd35567ef9bfa5e75732bee0f9e4c55fa90477bef2dfcdbc4be80" +dependencies = [ + "chacha20", + "getrandom 0.4.3", + "rand_core 0.10.1", ] [[package]] @@ -2777,7 +3412,16 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "d3022b5f1df60f26e1ffddd6c66e8aa15de382ae63b3a0c1bfc0e4d3e3f325cb" dependencies = [ "ppv-lite86", - "rand_core", + "rand_core 0.9.5", +] + +[[package]] +name = "rand_core" +version = "0.6.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ec0be4795e2f6a28069bec0b5ff3e2ac9bafc99e6a9a7dc3547996c5c816922c" +dependencies = [ + "serde", ] [[package]] @@ -2789,6 +3433,30 @@ dependencies = [ "getrandom 0.3.4", ] +[[package]] +name = "rand_core" +version = "0.10.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "63b8176103e19a2643978565ca18b50549f6101881c443590420e4dc998a3c69" + +[[package]] +name = "rand_pcg" +version = "0.10.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "caa0f4137e1c0a72f4c651489402276c8e8e1cf081f3b0ba156d2cbeef09e86a" +dependencies = [ + "rand_core 0.10.1", +] + +[[package]] +name = "rapidhash" +version = "4.5.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5da7e78a036ce858e8d55b7e7dc8ba3a88b78350fd2155d3591bbd966b58589e" +dependencies = [ + "rustversion", +] + [[package]] name = "rayon" version = "1.12.0" @@ -2864,7 +3532,7 @@ version = "0.5.18" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ed2bf2547551a7053d6fdfafda3f938979645c44812fbfcda098faae3f1a362d" dependencies = [ - "bitflags", + "bitflags 2.13.0", ] [[package]] @@ -2916,6 +3584,67 @@ version = "0.8.11" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "d6f6ff9a378485b298a5286656da665ba74413d36db0979633275d2e708145d4" +[[package]] +name = "reqwest" +version = "0.12.28" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "eddd3ca559203180a307f12d114c268abf583f59b03cb906fd0b3ff8646c1147" +dependencies = [ + "base64", + "bytes", + "futures-core", + "futures-util", + "http", + "http-body", + "http-body-util", + "hyper", + "hyper-rustls", + "hyper-util", + "js-sys", + "log", + "percent-encoding", + "pin-project-lite", + "quinn", + "rustls", + "rustls-pki-types", + "serde", + "serde_json", + "serde_urlencoded", + "sync_wrapper", + "tokio", + "tokio-rustls", + "tokio-util", + "tower", + "tower-http", + "tower-service", + "url", + "wasm-bindgen", + "wasm-bindgen-futures", + "wasm-streams", + "web-sys", + "webpki-roots", +] + +[[package]] +name = "ring" +version = "0.17.14" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a4689e6c2294d81e88dc6261c768b63bc4fcdb852be6d1352498b114f61383b7" +dependencies = [ + "cc", + "cfg-if", + "getrandom 0.2.17", + "libc", + "untrusted", + "windows-sys 0.52.0", +] + +[[package]] +name = "rustc-hash" +version = "2.1.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6b1e7f9a428571be2dc5bc0505c13fb6bf936822b894ec87abf8a08a4e51742d" + [[package]] name = "rustc_version" version = "0.4.1" @@ -2931,13 +3660,48 @@ version = "1.1.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b6fe4565b9518b83ef4f91bb47ce29620ca828bd32cb7e408f0062e9930ba190" dependencies = [ - "bitflags", + "bitflags 2.13.0", "errno", "libc", "linux-raw-sys", "windows-sys 0.61.2", ] +[[package]] +name = "rustls" +version = "0.23.41" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6b92b125634d9b795e7beca796cc790df15a7fb38323bf3196fda83292d06b1f" +dependencies = [ + "once_cell", + "ring", + "rustls-pki-types", + "rustls-webpki", + "subtle", + "zeroize", +] + +[[package]] +name = "rustls-pki-types" +version = "1.15.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "764899a24af3980067ee14bc143654f297b22eaebfe3c7b6b211920a5a59b046" +dependencies = [ + "web-time", + "zeroize", +] + +[[package]] +name = "rustls-webpki" +version = "0.103.13" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "61c429a8649f110dddef65e2a5ad240f747e85f7758a6bccc7e5777bd33f756e" +dependencies = [ + "ring", + "rustls-pki-types", + "untrusted", +] + [[package]] name = "rustversion" version = "1.0.22" @@ -3007,6 +3771,15 @@ dependencies = [ "syn", ] +[[package]] +name = "serde_fmt" +version = "1.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6e497af288b3b95d067a23a4f749f2861121ffcb2f6d8379310dcda040c345ed" +dependencies = [ + "serde_core", +] + [[package]] name = "serde_json" version = "1.0.150" @@ -3020,6 +3793,18 @@ dependencies = [ "zmij", ] +[[package]] +name = "serde_urlencoded" +version = "0.7.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d3491c14715ca2294c4d6a88f15e84739788c1d030eed8c110436aafdaa2f3fd" +dependencies = [ + "form_urlencoded", + "itoa", + "ryu", + "serde", +] + [[package]] name = "sha2" version = "0.10.9" @@ -3037,6 +3822,16 @@ version = "2.0.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f8fadd59c855ef2080decdef8ff161eb6661b86933c9d82e5ba29dc602a55aba" +[[package]] +name = "signal-hook-registry" +version = "1.4.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c4db69cba1110affc0e9f7bcd48bbf87b3f4fc7c61fc9155afd4c469eb3d6c1b" +dependencies = [ + "errno", + "libc", +] + [[package]] name = "simd-adler32" version = "0.3.9" @@ -3108,6 +3903,16 @@ version = "1.1.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1b6b67fb9a61334225b5b790716f609cd58395f895b3fe8b328786812a40bc3b" +[[package]] +name = "socket2" +version = "0.6.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "52d1cfed4120b4d927bf7c0f86d2087a4a7d6027c906d9f9d525a80573b9be51" +dependencies = [ + "libc", + "windows-sys 0.61.2", +] + [[package]] name = "sqlparser" version = "0.61.0" @@ -3162,6 +3967,7 @@ dependencies = [ "csv-core", "datafusion", "datafusion-orc", + "fluss-rs", "futures", "jni", "libmimalloc-sys", @@ -3179,6 +3985,54 @@ dependencies = [ "tokio", ] +[[package]] +name = "strsim" +version = "0.11.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7da8b5736845d9f2fcb837ea5d9e2628564b3b043a70948a3f0b778838c5fb4f" + +[[package]] +name = "structmeta" +version = "0.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2e1575d8d40908d70f6fd05537266b90ae71b15dbbe7a8b7dffa2b759306d329" +dependencies = [ + "proc-macro2", + "quote", + "structmeta-derive", + "syn", +] + +[[package]] +name = "structmeta-derive" +version = "0.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "152a0b65a590ff6c3da95cabe2353ee04e6167c896b28e3b14478c2636c922fc" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "strum" +version = "0.26.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8fec0f0aef304996cf250b31b5a10dee7980c85da9d759361292b8bca5a18f06" + +[[package]] +name = "strum_macros" +version = "0.26.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4c6bee85a5a24955dc440386795aa378cd9cf82acd5f764469152d2270e581be" +dependencies = [ + "heck", + "proc-macro2", + "quote", + "rustversion", + "syn", +] + [[package]] name = "strum_macros" version = "0.28.0" @@ -3197,6 +4051,84 @@ version = "2.6.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "13c2bddecc57b384dee18652358fb23172facb8a2c51ccc10d74c157bdea3292" +[[package]] +name = "sval" +version = "2.20.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5fb9efbae90f97301f4d25f3be63dfd99d6b7af9d088228a52ec960d649b2e7d" + +[[package]] +name = "sval_buffer" +version = "2.20.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ff5c0280ea0af40b3a1fd0b532680b5067482ffb81412ee66ec04b1d9952b49a" +dependencies = [ + "sval", + "sval_ref", +] + +[[package]] +name = "sval_dynamic" +version = "2.20.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "59b9067f2e68f58e110cf8019a268057e3791cf74b0fdb1b3c7c6e49104f44e6" +dependencies = [ + "sval", +] + +[[package]] +name = "sval_fmt" +version = "2.20.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "96ebdbf0e4b175884aa587fcf551c16dabe245ea04235abdd8cbbd160b27ed5a" +dependencies = [ + "itoa", + "ryu", + "sval", +] + +[[package]] +name = "sval_json" +version = "2.20.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1e448d9fa216a6c16670b28d624fbbcf5c04e41eb187bd7c52e01222ffa72a12" +dependencies = [ + "itoa", + "ryu", + "sval", +] + +[[package]] +name = "sval_nested" +version = "2.20.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "021de5b5c26efd544c694cef9b8a9abe8633481bf3be1ea145a18a740818b291" +dependencies = [ + "sval", + "sval_buffer", + "sval_ref", +] + +[[package]] +name = "sval_ref" +version = "2.20.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "54ef5ffec8bc52ded04ee424ab8d959e25e64fd40a48a16d21f8991ce824c1bb" +dependencies = [ + "sval", +] + +[[package]] +name = "sval_serde" +version = "2.20.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b983833a8a2390f89ebcf9b9acd06883017014b4ffd72ee28e0c9de6852039f5" +dependencies = [ + "serde_core", + "sval", + "sval_nested", +] + [[package]] name = "syn" version = "2.0.118" @@ -3208,6 +4140,15 @@ dependencies = [ "unicode-ident", ] +[[package]] +name = "sync_wrapper" +version = "1.0.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0bf256ce5efdfa370213c1dabab5935a12e49f2c58d15e9eac2870d3b4f27263" +dependencies = [ + "futures-core", +] + [[package]] name = "synstructure" version = "0.13.2" @@ -3219,6 +4160,12 @@ dependencies = [ "syn", ] +[[package]] +name = "tap" +version = "1.0.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "55937e1799185b12863d447f42597ed69d9928686b8d88a1df17376a097d8369" + [[package]] name = "tempfile" version = "3.27.0" @@ -3280,7 +4227,7 @@ checksum = "7e54bc85fc7faa8bc175c4bab5b92ba8d9a3ce893d0e9f42cc455c8ab16a9e09" dependencies = [ "byteorder", "integer-encoding", - "ordered-float", + "ordered-float 2.10.1", ] [[package]] @@ -3312,6 +4259,21 @@ dependencies = [ "serde_json", ] +[[package]] +name = "tinyvec" +version = "1.11.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3e61e67053d25a4e82c844e8424039d9745781b3fc4f32b8d55ed50f5f667ef3" +dependencies = [ + "tinyvec_macros", +] + +[[package]] +name = "tinyvec_macros" +version = "0.1.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1f3ccbac311fea05f86f61904b462b55fb3df8837a366dfc601a0161d0532f20" + [[package]] name = "tokio" version = "1.52.3" @@ -3319,8 +4281,14 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "8fc7f01b389ac15039e4dc9531aa973a135d7a4135281b12d7c1bc79fd57fffe" dependencies = [ "bytes", + "libc", + "mio", + "parking_lot", "pin-project-lite", + "signal-hook-registry", + "socket2", "tokio-macros", + "windows-sys 0.61.2", ] [[package]] @@ -3334,6 +4302,16 @@ dependencies = [ "syn", ] +[[package]] +name = "tokio-rustls" +version = "0.26.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1729aa945f29d91ba541258c8df89027d5792d85a8841fb65e8bf0f4ede4ef61" +dependencies = [ + "rustls", + "tokio", +] + [[package]] name = "tokio-stream" version = "0.1.18" @@ -3389,6 +4367,51 @@ dependencies = [ "winnow", ] +[[package]] +name = "tower" +version = "0.5.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ebe5ef63511595f1344e2d5cfa636d973292adc0eec1f0ad45fae9f0851ab1d4" +dependencies = [ + "futures-core", + "futures-util", + "pin-project-lite", + "sync_wrapper", + "tokio", + "tower-layer", + "tower-service", +] + +[[package]] +name = "tower-http" +version = "0.6.11" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4cfcf7e2740e6fc6d4d688b4ef00650406bb94adf4731e43c096c3a19fe40840" +dependencies = [ + "bitflags 2.13.0", + "bytes", + "futures-util", + "http", + "http-body", + "pin-project-lite", + "tower", + "tower-layer", + "tower-service", + "url", +] + +[[package]] +name = "tower-layer" +version = "0.3.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "121c2a6cda46980bb0fcd1647ffaf6cd3fc79a013de288782836f6df9c48780e" + +[[package]] +name = "tower-service" +version = "0.3.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8df9b6e13f2d32c91b9bd719c00d1958837bc7dec474d94952798cc8e69eeec3" + [[package]] name = "tracing" version = "0.1.44" @@ -3420,12 +4443,24 @@ dependencies = [ "once_cell", ] +[[package]] +name = "try-lock" +version = "0.2.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e421abadd41a4225275504ea4d6566923418b7f05506fbc9c0fe86ba7396114b" + [[package]] name = "twox-hash" version = "2.1.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9ea3136b675547379c4bd395ca6b938e5ad3c3d20fad76e7fe85f9e0d011419c" +[[package]] +name = "typeid" +version = "1.0.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bc7d623258602320d5c55d1bc22793b57daff0ec7efc270ea7d55ce1d5f5471c" + [[package]] name = "typenum" version = "1.20.1" @@ -3450,6 +4485,12 @@ version = "0.2.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b4ac048d71ede7ee76d585517add45da530660ef4390e49b098733c6e897f254" +[[package]] +name = "untrusted" +version = "0.9.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8ecb6da28b8a351d773b68d5825ac39017e680750f980f3a1a85cd8dd28a47c1" + [[package]] name = "url" version = "2.5.8" @@ -3468,6 +4509,12 @@ version = "1.0.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b6c140620e7ffbb22c2dee59cafe6084a59b5ffc27a8859a5f0d494b5d52b6be" +[[package]] +name = "utf8parse" +version = "0.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "06abde3611657adf66d383f00b093d7faecc7fa57071cce2578660c9f1010821" + [[package]] name = "uuid" version = "1.23.4" @@ -3476,9 +4523,46 @@ checksum = "bf80a72845275afea99e7f2b434723d3bc7e38470fcd1c7ed39a599c73319a53" dependencies = [ "getrandom 0.4.3", "js-sys", + "serde_core", "wasm-bindgen", ] +[[package]] +name = "value-bag" +version = "1.12.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7ba6f5989077681266825251a52748b8c1d8a4ad098cc37e440103d0ea717fc0" +dependencies = [ + "value-bag-serde1", + "value-bag-sval2", +] + +[[package]] +name = "value-bag-serde1" +version = "1.12.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "16530907bfe2999a1773ca5900a65101e092c70f642f25cc23ca0c43573262c5" +dependencies = [ + "erased-serde", + "serde_core", + "serde_fmt", +] + +[[package]] +name = "value-bag-sval2" +version = "1.12.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d00ae130edd690eaa877e4f40605d534790d1cf1d651e7685bd6a144521b251f" +dependencies = [ + "sval", + "sval_buffer", + "sval_dynamic", + "sval_fmt", + "sval_json", + "sval_ref", + "sval_serde", +] + [[package]] name = "value-trait" version = "0.12.1" @@ -3513,6 +4597,15 @@ dependencies = [ "winapi-util", ] +[[package]] +name = "want" +version = "0.3.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bfa7760aed19e106de2c7c0b581b509f2f25d3dacaf737cb82ac61bc6d760b0e" +dependencies = [ + "try-lock", +] + [[package]] name = "wasi" version = "0.11.1+wasi-snapshot-preview1" @@ -3583,6 +4676,19 @@ dependencies = [ "unicode-ident", ] +[[package]] +name = "wasm-streams" +version = "0.4.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "15053d8d85c7eccdbefef60f06769760a563c7f0a9d6902a13d35c7800b0ad65" +dependencies = [ + "futures-util", + "js-sys", + "wasm-bindgen", + "wasm-bindgen-futures", + "web-sys", +] + [[package]] name = "web-sys" version = "0.3.103" @@ -3603,6 +4709,15 @@ dependencies = [ "wasm-bindgen", ] +[[package]] +name = "webpki-roots" +version = "1.0.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bf85cb06032201fa7c6f829d7db5a7e5aa45bcc0655327713065f6f0576731bf" +dependencies = [ + "rustls-pki-types", +] + [[package]] name = "winapi-util" version = "0.1.11" @@ -3677,7 +4792,16 @@ version = "0.45.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "75283be5efb2831d37ea142365f009c02ec203cd29a3ebecbc093d52315b66d0" dependencies = [ - "windows-targets", + "windows-targets 0.42.2", +] + +[[package]] +name = "windows-sys" +version = "0.52.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "282be5f36a8ce781fad8c8ae18fa3f9beff57ec1b52cb3de0789201425d9a33d" +dependencies = [ + "windows-targets 0.52.6", ] [[package]] @@ -3695,13 +4819,29 @@ version = "0.42.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "8e5180c00cd44c9b1c88adb3693291f1cd93605ded80c250a75d472756b4d071" dependencies = [ - "windows_aarch64_gnullvm", - "windows_aarch64_msvc", - "windows_i686_gnu", - "windows_i686_msvc", - "windows_x86_64_gnu", - "windows_x86_64_gnullvm", - "windows_x86_64_msvc", + "windows_aarch64_gnullvm 0.42.2", + "windows_aarch64_msvc 0.42.2", + "windows_i686_gnu 0.42.2", + "windows_i686_msvc 0.42.2", + "windows_x86_64_gnu 0.42.2", + "windows_x86_64_gnullvm 0.42.2", + "windows_x86_64_msvc 0.42.2", +] + +[[package]] +name = "windows-targets" +version = "0.52.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9b724f72796e036ab90c1021d4780d4d3d648aca59e491e6b98e725b84e99973" +dependencies = [ + "windows_aarch64_gnullvm 0.52.6", + "windows_aarch64_msvc 0.52.6", + "windows_i686_gnu 0.52.6", + "windows_i686_gnullvm", + "windows_i686_msvc 0.52.6", + "windows_x86_64_gnu 0.52.6", + "windows_x86_64_gnullvm 0.52.6", + "windows_x86_64_msvc 0.52.6", ] [[package]] @@ -3710,42 +4850,90 @@ version = "0.42.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "597a5118570b68bc08d8d59125332c54f1ba9d9adeedeef5b99b02ba2b0698f8" +[[package]] +name = "windows_aarch64_gnullvm" +version = "0.52.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "32a4622180e7a0ec044bb555404c800bc9fd9ec262ec147edd5989ccd0c02cd3" + [[package]] name = "windows_aarch64_msvc" version = "0.42.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "e08e8864a60f06ef0d0ff4ba04124db8b0fb3be5776a5cd47641e942e58c4d43" +[[package]] +name = "windows_aarch64_msvc" +version = "0.52.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "09ec2a7bb152e2252b53fa7803150007879548bc709c039df7627cabbd05d469" + [[package]] name = "windows_i686_gnu" version = "0.42.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "c61d927d8da41da96a81f029489353e68739737d3beca43145c8afec9a31a84f" +[[package]] +name = "windows_i686_gnu" +version = "0.52.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8e9b5ad5ab802e97eb8e295ac6720e509ee4c243f69d781394014ebfe8bbfa0b" + +[[package]] +name = "windows_i686_gnullvm" +version = "0.52.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0eee52d38c090b3caa76c563b86c3a4bd71ef1a819287c19d586d7334ae8ed66" + [[package]] name = "windows_i686_msvc" version = "0.42.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "44d840b6ec649f480a41c8d80f9c65108b92d89345dd94027bfe06ac444d1060" +[[package]] +name = "windows_i686_msvc" +version = "0.52.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "240948bc05c5e7c6dabba28bf89d89ffce3e303022809e73deaefe4f6ec56c66" + [[package]] name = "windows_x86_64_gnu" version = "0.42.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "8de912b8b8feb55c064867cf047dda097f92d51efad5b491dfb98f6bbb70cb36" +[[package]] +name = "windows_x86_64_gnu" +version = "0.52.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "147a5c80aabfbf0c7d901cb5895d1de30ef2907eb21fbbab29ca94c5b08b1a78" + [[package]] name = "windows_x86_64_gnullvm" version = "0.42.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "26d41b46a36d453748aedef1486d5c7a85db22e56aff34643984ea85514e94a3" +[[package]] +name = "windows_x86_64_gnullvm" +version = "0.52.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "24d5b23dc417412679681396f2b49f3de8c1473deb516bd34410872eff51ed0d" + [[package]] name = "windows_x86_64_msvc" version = "0.42.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9aec5da331524158c6d1a4ac0ab1541149c0b9505fde06423b02f5ef0106b9f0" +[[package]] +name = "windows_x86_64_msvc" +version = "0.52.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "589f6da84c646204747d1270a2a5661ea66ed1cced2631d546fdfb155959f9ec" + [[package]] name = "winnow" version = "1.0.3" @@ -3767,6 +4955,15 @@ version = "0.6.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1ffae5123b2d3fc086436f8834ae3ab053a283cfac8fe0a0b8eaae044768a4c4" +[[package]] +name = "wyz" +version = "0.5.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "05f360fc0b24296329c78fda852a1e9ae82de9cf7b27dae4b7f62f118f77b9ed" +dependencies = [ + "tap", +] + [[package]] name = "yoke" version = "0.8.3" @@ -3831,6 +5028,12 @@ dependencies = [ "synstructure", ] +[[package]] +name = "zeroize" +version = "1.9.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e13c156562582aa81c60cb29407084cdb54c4164760106ab78e6c5b0858cf64e" + [[package]] name = "zerotrie" version = "0.2.4" diff --git a/native/Cargo.toml b/native/Cargo.toml index 1730b202..d6962fb4 100644 --- a/native/Cargo.toml +++ b/native/Cargo.toml @@ -56,6 +56,12 @@ datafusion = "53.1.0" # without a system dependency. (SSL/SASL-GSSAPI for the Kerberos cluster would add the `ssl` + # `gssapi-vendored` features — a heavier build, deferred until needed.) rdkafka = { version = "0.36", default-features = false, features = ["libz-static"], optional = true } +# Native Fluss log-table reader, behind the `fluss` feature. Pinned to a temporary fork of +# apache/fluss-rust whose only change is bumping its arrow dependency to align with this workspace's +# arrow version (resolving to 58.3.0 in Cargo.lock), so the batches the reader hands back are this +# crate's arrow types rather than a second arrow in the build. Replace with upstream +# apache/fluss-rust once the arrow bump is upstreamed there. +fluss_rs = { package = "fluss-rs", git = "https://github.com/michaelmitchell-bit/fluss-rust.git", rev = "7182719652d5f6227f15611f888430d74f2eb26b", default-features = true, optional = true } tokio = { version = "1", features = ["rt-multi-thread"] } futures = "0.3" # Fast hashing for the per-row grouping map (the same hasher arrow/datafusion use internally). @@ -88,6 +94,7 @@ libmimalloc-sys = { version = "0.1", optional = true } default = ["kafka"] kafka = ["dep:rdkafka"] kafka-bench = ["kafka"] +fluss = ["dep:fluss_rs"] # Rebind the C allocator to mimalloc for everything linked into THIS library (librdkafka included), # via link-time symbol aliases emitted by build.rs. Library-scoped: the hosting JVM process keeps # its own allocator, so this carries none of the risk of a process-wide malloc override (a zone-swap diff --git a/native/src/bridge.rs b/native/src/bridge.rs index c9d82d5b..6649051b 100644 --- a/native/src/bridge.rs +++ b/native/src/bridge.rs @@ -398,7 +398,7 @@ pub(crate) fn read_columns(env: &JNIEnv, columns: &JIntArray) -> Vec { } /// Reads a JVM String[] into a Vec. -#[cfg(feature = "kafka")] +#[cfg(any(feature = "kafka", feature = "fluss"))] pub(crate) fn read_string_array(env: &mut JNIEnv, array: &JObjectArray) -> Vec { let length = env.get_array_length(array).expect("failed to read string[] length"); let mut out = Vec::with_capacity(length as usize); diff --git a/native/src/fluss.rs b/native/src/fluss.rs new file mode 100644 index 00000000..eb440569 --- /dev/null +++ b/native/src/fluss.rs @@ -0,0 +1,687 @@ +use crate::*; + +#[cfg(feature = "fluss")] +const NO_PARTITION: i64 = i64::MIN; +#[cfg(feature = "fluss")] +const NO_STOPPING_OFFSET: i64 = i64::MIN; + +/// Whether this native library was built with the native Fluss reader feature. +#[no_mangle] +pub extern "system" fn Java_io_github_jordepic_streamfusion_Native_flussFeatureBuilt<'local>( + _env: JNIEnv<'local>, + _class: JClass<'local>, +) -> jni::sys::jboolean { + cfg!(feature = "fluss") as jni::sys::jboolean +} + +#[cfg(feature = "fluss")] +struct FlussSplitReader { + _connection: fluss_rs::client::FlussConnection, + scanner: fluss_rs::client::RecordBatchLogScanner, + split_ids: HashMap, + stopping_offsets: HashMap, + pending: std::collections::VecDeque<(String, i64, RecordBatch)>, +} + +#[cfg(feature = "fluss")] +impl FlussSplitReader { + fn open( + config: &[(String, String)], + database_name: &str, + table_name: &str, + projected_fields: &[usize], + ) -> Result { + let config = fluss_config(config)?; + let table_path = + fluss_rs::metadata::TablePath::new(database_name.to_string(), table_name.to_string()); + let (connection, scanner) = runtime().block_on(async { + let connection = fluss_rs::client::FlussConnection::new(config) + .await + .map_err(|e| format!("failed to create Fluss connection: {e}"))?; + let scanner = { + let table = connection + .get_table(&table_path) + .await + .map_err(|e| format!("failed to get Fluss table: {e}"))?; + let scan = table.new_scan(); + let scan = if projected_fields.is_empty() { + scan + } else { + scan.project(projected_fields) + .map_err(|e| format!("failed to project Fluss scan: {e}"))? + }; + scan.create_record_batch_log_scanner() + .map_err(|e| format!("failed to create Fluss RecordBatch log scanner: {e}"))? + }; + Ok::<_, String>((connection, scanner)) + })?; + Ok(Self { + _connection: connection, + scanner, + split_ids: HashMap::default(), + stopping_offsets: HashMap::default(), + pending: std::collections::VecDeque::new(), + }) + } + + fn assign_splits( + &mut self, + split_ids: &[String], + table_ids: &[i64], + partition_ids: &[i64], + buckets: &[i64], + start_offsets: &[i64], + stopping_offsets: &[i64], + ) -> Result<(), String> { + if split_ids.len() != table_ids.len() + || split_ids.len() != partition_ids.len() + || split_ids.len() != buckets.len() + || split_ids.len() != start_offsets.len() + || split_ids.len() != stopping_offsets.len() + { + return Err("mismatched Fluss split assignment array lengths".to_string()); + } + // fluss-rs subscribe_buckets/subscribe_partition_buckets take std HashMaps. + let mut bucket_offsets: std::collections::HashMap = + std::collections::HashMap::new(); + let mut partition_bucket_offsets: std::collections::HashMap<(i64, i32), i64> = + std::collections::HashMap::new(); + for i in 0..split_ids.len() { + let table_bucket = table_bucket(table_ids[i], partition_ids[i], buckets[i]); + self.split_ids + .insert(table_bucket.clone(), split_ids[i].clone()); + if stopping_offsets[i] != NO_STOPPING_OFFSET { + self.stopping_offsets + .insert(table_bucket.clone(), stopping_offsets[i]); + } + if partition_ids[i] == NO_PARTITION { + bucket_offsets.insert(buckets[i] as i32, start_offsets[i]); + } else { + partition_bucket_offsets + .insert((partition_ids[i], buckets[i] as i32), start_offsets[i]); + } + } + if !bucket_offsets.is_empty() { + runtime() + .block_on(self.scanner.subscribe_buckets(&bucket_offsets)) + .map_err(|e| format!("failed to subscribe Fluss buckets: {e}"))?; + } + if !partition_bucket_offsets.is_empty() { + runtime() + .block_on( + self.scanner + .subscribe_partition_buckets(&partition_bucket_offsets), + ) + .map_err(|e| format!("failed to subscribe Fluss partition buckets: {e}"))?; + } + Ok(()) + } + + fn unassign_splits( + &mut self, + table_ids: &[i64], + partition_ids: &[i64], + buckets: &[i64], + ) -> Result<(), String> { + if table_ids.len() != partition_ids.len() || table_ids.len() != buckets.len() { + return Err("mismatched Fluss split unassignment array lengths".to_string()); + } + let mut removed_split_ids = HashSet::default(); + for i in 0..table_ids.len() { + let table_bucket = table_bucket(table_ids[i], partition_ids[i], buckets[i]); + self.unsubscribe_bucket(&table_bucket); + if let Some(split_id) = self.split_ids.remove(&table_bucket) { + removed_split_ids.insert(split_id); + } + self.stopping_offsets.remove(&table_bucket); + } + self.pending + .retain(|(split_id, _, _)| !removed_split_ids.contains(split_id)); + Ok(()) + } + + fn poll(&mut self, timeout: std::time::Duration) -> Result { + let batches = match runtime().block_on(self.scanner.poll(timeout)) { + Ok(batches) => batches, + Err(fluss_rs::error::Error::FlussAPIError { api_error }) + if fluss_rs::error::FlussError::for_code(api_error.code) + == fluss_rs::error::FlussError::PartitionNotExists => + { + self.remove_missing_partition(api_error.message.as_str())?; + return Ok(self.pending.len()); + } + Err(e) => return Err(format!("failed to poll Fluss batches: {e}")), + }; + for batch in batches { + self.enqueue(batch)?; + } + Ok(self.pending.len()) + } + + fn remove_missing_partition(&mut self, message: &str) -> Result<(), String> { + // If the server's error wording changed and no partition id can be parsed, + // keep polling: the enumerator's PartitionsRemovedEvent path cleans up. + let Some(partition_id) = parse_missing_partition_id(message) else { + return Ok(()); + }; + let removed_buckets: Vec<_> = self + .split_ids + .keys() + .filter(|table_bucket| table_bucket.partition_id() == Some(partition_id)) + .cloned() + .collect(); + if removed_buckets.is_empty() { + return Err(format!( + "failed to poll Fluss batches: partition {partition_id} disappeared but no assigned split matched it: {message}" + )); + } + + let removed_split_ids: HashSet<_> = removed_buckets + .iter() + .filter_map(|table_bucket| self.split_ids.get(table_bucket).cloned()) + .collect(); + for table_bucket in removed_buckets { + self.unsubscribe_bucket(&table_bucket); + self.split_ids.remove(&table_bucket); + self.stopping_offsets.remove(&table_bucket); + } + self.pending + .retain(|(split_id, _, _)| !removed_split_ids.contains(split_id)); + Ok(()) + } + + fn enqueue(&mut self, scan_batch: fluss_rs::record::ScanBatch) -> Result<(), String> { + let table_bucket = scan_batch.bucket().clone(); + let Some(split_id) = self.split_ids.get(&table_bucket).cloned() else { + return Ok(()); + }; + + let base_offset = scan_batch.base_offset(); + let mut batch = scan_batch.into_batch(); + let mut next_offset = base_offset + batch.num_rows() as i64; + + if let Some(stopping_offset) = self.stopping_offsets.get(&table_bucket).copied() { + if base_offset >= stopping_offset { + self.finish_stopped_bucket(&table_bucket); + return Ok(()); + } + if next_offset > stopping_offset { + let keep_rows = (stopping_offset - base_offset).max(0) as usize; + if keep_rows == 0 { + self.finish_stopped_bucket(&table_bucket); + return Ok(()); + } + batch = batch.slice(0, keep_rows); + next_offset = stopping_offset; + } + if next_offset >= stopping_offset { + self.finish_stopped_bucket(&table_bucket); + } + } + + if batch.num_rows() > 0 { + batch = normalize_timestamp_units(batch)?; + self.pending.push_back((split_id, next_offset, batch)); + } + Ok(()) + } + + /// Removes a bucket that reached its stopping offset so later in-flight + /// batches for it drop at the split_ids gate without another unsubscribe RPC. + fn finish_stopped_bucket(&mut self, table_bucket: &fluss_rs::metadata::TableBucket) { + self.split_ids.remove(table_bucket); + self.stopping_offsets.remove(table_bucket); + self.unsubscribe_bucket(table_bucket); + } + + fn unsubscribe_bucket(&self, table_bucket: &fluss_rs::metadata::TableBucket) { + if let Some(partition_id) = table_bucket.partition_id() { + let _ = runtime().block_on( + self.scanner + .unsubscribe_partition(partition_id, table_bucket.bucket_id()), + ); + } else { + let _ = runtime().block_on(self.scanner.unsubscribe(table_bucket.bucket_id())); + } + } +} + +#[cfg(feature = "fluss")] +fn normalize_timestamp_units(batch: RecordBatch) -> Result { + let mut changed = false; + let mut fields = Vec::with_capacity(batch.num_columns()); + let mut columns = Vec::with_capacity(batch.num_columns()); + for (field, column) in batch.schema().fields().iter().zip(batch.columns()) { + let (normalized_field, normalized_column, field_changed) = + normalize_field_array(field, column.clone())?; + changed |= field_changed; + fields.push(normalized_field); + columns.push(normalized_column); + } + if !changed { + return Ok(batch); + } + RecordBatch::try_new(Arc::new(Schema::new(fields)), columns) + .map_err(|e| format!("failed to normalize Fluss batch timestamp units: {e}")) +} + +#[cfg(feature = "fluss")] +fn normalize_field_array( + field: &FieldRef, + array: ArrayRef, +) -> Result<(FieldRef, ArrayRef, bool), String> { + let (data_type, array, changed) = normalize_array(field.data_type(), array)?; + if !changed { + return Ok((field.clone(), array, false)); + } + Ok(( + Arc::new(field.as_ref().clone().with_data_type(data_type)), + array, + true, + )) +} + +#[cfg(feature = "fluss")] +fn normalize_array( + data_type: &DataType, + array: ArrayRef, +) -> Result<(DataType, ArrayRef, bool), String> { + match data_type { + DataType::Timestamp(unit, None) if *unit != arrow::datatypes::TimeUnit::Nanosecond => { + let target = DataType::Timestamp(arrow::datatypes::TimeUnit::Nanosecond, None); + let casted = arrow::compute::cast(array.as_ref(), &target) + .map_err(|e| format!("failed to cast Fluss timestamp to nanoseconds: {e}"))?; + Ok((target, casted, true)) + } + DataType::Struct(fields) => { + let struct_array = array + .as_any() + .downcast_ref::() + .ok_or_else(|| { + format!( + "expected StructArray for Fluss field, got {:?}", + array.data_type() + ) + })?; + let mut changed = false; + let mut normalized_fields = Vec::with_capacity(fields.len()); + let mut normalized_columns = Vec::with_capacity(fields.len()); + for (index, field) in fields.iter().enumerate() { + let (normalized_field, normalized_column, field_changed) = + normalize_field_array(field, struct_array.column(index).clone())?; + changed |= field_changed; + normalized_fields.push(normalized_field); + normalized_columns.push(normalized_column); + } + if !changed { + return Ok((data_type.clone(), array, false)); + } + let normalized_fields: Fields = normalized_fields.into(); + let normalized = StructArray::try_new( + normalized_fields.clone(), + normalized_columns, + struct_array.nulls().cloned(), + ) + .map_err(|e| format!("failed to normalize nested Fluss struct timestamps: {e}"))?; + Ok(( + DataType::Struct(normalized_fields), + Arc::new(normalized), + true, + )) + } + _ => Ok((data_type.clone(), array, false)), + } +} + +#[cfg(feature = "fluss")] +fn table_bucket(table_id: i64, partition_id: i64, bucket: i64) -> fluss_rs::metadata::TableBucket { + if partition_id == NO_PARTITION { + fluss_rs::metadata::TableBucket::new(table_id, bucket as i32) + } else { + fluss_rs::metadata::TableBucket::new_with_partition( + table_id, + Some(partition_id), + bucket as i32, + ) + } +} + +#[cfg(feature = "fluss")] +fn parse_missing_partition_id(message: &str) -> Option { + let marker = "partition id '"; + let start = message.find(marker)? + marker.len(); + let end = message[start..].find('\'')?; + message[start..start + end].parse().ok() +} + +#[cfg(feature = "fluss")] +fn fluss_config(values: &[(String, String)]) -> Result { + let mut config = fluss_rs::config::Config::default(); + for (key, value) in values { + match key.as_str() { + "bootstrap_servers" => config.bootstrap_servers = value.clone(), + "scanner_remote_log_prefetch_num" => { + config.scanner_remote_log_prefetch_num = parse(key, value)? + } + "remote_file_download_thread_num" => { + config.remote_file_download_thread_num = parse(key, value)? + } + "scanner_log_max_poll_records" => { + config.scanner_log_max_poll_records = parse(key, value)? + } + "scanner_log_fetch_max_bytes" => { + config.scanner_log_fetch_max_bytes = parse(key, value)? + } + "scanner_log_fetch_min_bytes" => { + config.scanner_log_fetch_min_bytes = parse(key, value)? + } + "scanner_log_fetch_wait_max_time_ms" => { + config.scanner_log_fetch_wait_max_time_ms = parse(key, value)? + } + "scanner_log_fetch_max_bytes_for_bucket" => { + config.scanner_log_fetch_max_bytes_for_bucket = parse(key, value)? + } + "connect_timeout_ms" => config.connect_timeout_ms = parse(key, value)?, + "security_protocol" => config.security_protocol = value.clone(), + "security_sasl_mechanism" => config.security_sasl_mechanism = value.clone(), + "security_sasl_username" => config.security_sasl_username = value.clone(), + "security_sasl_password" => config.security_sasl_password = value.clone(), + other => return Err(format!("unsupported fluss-rs config key {other}")), + } + } + Ok(config) +} + +#[cfg(feature = "fluss")] +fn parse(key: &str, value: &str) -> Result +where + T: std::str::FromStr, + T::Err: std::fmt::Display, +{ + value + .parse() + .map_err(|e| format!("failed to parse Fluss config {key}={value}: {e}")) +} + +#[cfg(feature = "fluss")] +fn fluss_jni(env: &mut JNIEnv, default: T, f: F) -> T +where + F: FnOnce(&mut JNIEnv) -> Result, +{ + match std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| f(env))) { + Ok(Ok(value)) => value, + Ok(Err(message)) => { + throw_fluss_exception(env, &message); + default + } + Err(payload) => { + throw_fluss_exception( + env, + &format!("native Fluss reader panic: {}", panic_message(payload)), + ); + default + } + } +} + +#[cfg(feature = "fluss")] +fn throw_fluss_exception(env: &mut JNIEnv, message: &str) { + let _ = env.throw_new("java/io/IOException", message); +} + +#[cfg(feature = "fluss")] +fn panic_message(payload: Box) -> String { + if let Some(message) = payload.downcast_ref::<&str>() { + (*message).to_string() + } else if let Some(message) = payload.downcast_ref::() { + message.clone() + } else { + "unknown panic".to_string() + } +} + +#[cfg(feature = "fluss")] +#[no_mangle] +pub extern "system" fn Java_io_github_jordepic_streamfusion_Native_openFlussReader<'local>( + mut env: JNIEnv<'local>, + _class: JClass<'local>, + config_keys: JObjectArray<'local>, + config_values: JObjectArray<'local>, + database_name: JString<'local>, + table_name: JString<'local>, + projected_fields: JIntArray<'local>, +) -> jlong { + fluss_jni(&mut env, 0, |env| { + let keys = read_string_array(env, &config_keys); + let values = read_string_array(env, &config_values); + let config: Vec<(String, String)> = keys.into_iter().zip(values).collect(); + let database_name: String = env + .get_string(&database_name) + .map(Into::into) + .map_err(|e| format!("failed to read Fluss database name: {e}"))?; + let table_name: String = env + .get_string(&table_name) + .map(Into::into) + .map_err(|e| format!("failed to read Fluss table name: {e}"))?; + let projected_fields = read_columns(env, &projected_fields); + Ok(into_handle(FlussSplitReader::open( + &config, + &database_name, + &table_name, + &projected_fields, + )?)) + }) +} + +#[cfg(feature = "fluss")] +#[no_mangle] +pub extern "system" fn Java_io_github_jordepic_streamfusion_Native_assignFlussSplits<'local>( + mut env: JNIEnv<'local>, + _class: JClass<'local>, + handle: jlong, + split_ids: JObjectArray<'local>, + table_ids: JLongArray<'local>, + partition_ids: JLongArray<'local>, + buckets: JLongArray<'local>, + start_offsets: JLongArray<'local>, + stopping_offsets: JLongArray<'local>, +) { + fluss_jni(&mut env, (), |env| { + let reader = unsafe { &mut *(handle as *mut FlussSplitReader) }; + let split_ids = read_string_array(env, &split_ids); + let table_ids = read_longs(env, &table_ids); + let partition_ids = read_longs(env, &partition_ids); + let buckets = read_longs(env, &buckets); + let start_offsets = read_longs(env, &start_offsets); + let stopping_offsets = read_longs(env, &stopping_offsets); + reader.assign_splits( + &split_ids, + &table_ids, + &partition_ids, + &buckets, + &start_offsets, + &stopping_offsets, + ) + }); +} + +#[cfg(feature = "fluss")] +#[no_mangle] +pub extern "system" fn Java_io_github_jordepic_streamfusion_Native_unassignFlussSplits<'local>( + mut env: JNIEnv<'local>, + _class: JClass<'local>, + handle: jlong, + table_ids: JLongArray<'local>, + partition_ids: JLongArray<'local>, + buckets: JLongArray<'local>, +) { + fluss_jni(&mut env, (), |env| { + let reader = unsafe { &mut *(handle as *mut FlussSplitReader) }; + let table_ids = read_longs(env, &table_ids); + let partition_ids = read_longs(env, &partition_ids); + let buckets = read_longs(env, &buckets); + reader.unassign_splits(&table_ids, &partition_ids, &buckets) + }); +} + +#[cfg(feature = "fluss")] +#[no_mangle] +pub extern "system" fn Java_io_github_jordepic_streamfusion_Native_pollFlussBatch<'local>( + mut env: JNIEnv<'local>, + _class: JClass<'local>, + handle: jlong, + timeout_ms: jlong, +) -> jint { + fluss_jni(&mut env, 0, |_env| { + let reader = unsafe { &mut *(handle as *mut FlussSplitReader) }; + Ok(reader.poll(std::time::Duration::from_millis(timeout_ms as u64))? as jint) + }) +} + +#[cfg(feature = "fluss")] +#[no_mangle] +pub extern "system" fn Java_io_github_jordepic_streamfusion_Native_drainFlussSplit<'local>( + mut env: JNIEnv<'local>, + _class: JClass<'local>, + handle: jlong, + split_meta: JLongArray<'local>, + out_split_id: JObjectArray<'local>, + out_array_address: jlong, + out_schema_address: jlong, +) -> jint { + fluss_jni(&mut env, 0, |env| { + let reader = unsafe { &mut *(handle as *mut FlussSplitReader) }; + let (split_id, next_offset, batch) = reader + .pending + .pop_front() + .ok_or_else(|| "drainFlussSplit called with no pending batch".to_string())?; + let rows = batch.num_rows() as jint; + env.set_long_array_region(&split_meta, 0, &[next_offset]) + .map_err(|e| format!("failed to write Fluss split meta: {e}"))?; + let split_id = env + .new_string(&split_id) + .map_err(|e| format!("failed to allocate Fluss split id string: {e}"))?; + env.set_object_array_element(&out_split_id, 0, &split_id) + .map_err(|e| format!("failed to write Fluss split id: {e}"))?; + export_record_batch(batch, out_array_address, out_schema_address); + Ok(rows) + }) +} + +#[cfg(feature = "fluss")] +#[no_mangle] +pub extern "system" fn Java_io_github_jordepic_streamfusion_Native_closeFlussReader<'local>( + mut env: JNIEnv<'local>, + _class: JClass<'local>, + handle: jlong, +) { + fluss_jni(&mut env, (), |_env| { + unsafe { + drop(from_handle::(handle)); + } + Ok(()) + }); +} + +#[cfg(all(test, feature = "fluss"))] +mod tests { + use super::*; + use arrow::datatypes::TimeUnit; + + #[test] + fn translated_keys_populate_fluss_rs_config_fields() { + let config = fluss_config(&[ + ("bootstrap_servers".into(), "localhost:9123".into()), + ("scanner_log_max_poll_records".into(), "2048".into()), + ("scanner_remote_log_prefetch_num".into(), "8".into()), + ("remote_file_download_thread_num".into(), "4".into()), + ("scanner_log_fetch_max_bytes".into(), "16777216".into()), + ( + "scanner_log_fetch_max_bytes_for_bucket".into(), + "1048576".into(), + ), + ("scanner_log_fetch_min_bytes".into(), "1".into()), + ("scanner_log_fetch_wait_max_time_ms".into(), "500".into()), + ("connect_timeout_ms".into(), "30000".into()), + ("security_protocol".into(), "sasl".into()), + ("security_sasl_mechanism".into(), "PLAIN".into()), + ("security_sasl_username".into(), "alice".into()), + ("security_sasl_password".into(), "secret".into()), + ]) + .expect("translated config"); + + assert_eq!(config.bootstrap_servers, "localhost:9123"); + assert_eq!(config.scanner_log_max_poll_records, 2048); + assert_eq!(config.scanner_remote_log_prefetch_num, 8); + assert_eq!(config.remote_file_download_thread_num, 4); + assert_eq!(config.scanner_log_fetch_max_bytes, 16_777_216); + assert_eq!(config.scanner_log_fetch_max_bytes_for_bucket, 1_048_576); + assert_eq!(config.scanner_log_fetch_min_bytes, 1); + assert_eq!(config.scanner_log_fetch_wait_max_time_ms, 500); + assert_eq!(config.connect_timeout_ms, 30_000); + assert_eq!(config.security_protocol, "sasl"); + assert_eq!(config.security_sasl_mechanism, "PLAIN"); + assert_eq!(config.security_sasl_username, "alice"); + assert_eq!(config.security_sasl_password, "secret"); + } + + #[test] + fn normalizes_timestamp_units_recursively() { + let timestamp_ms = DataType::Timestamp(TimeUnit::Millisecond, None); + let timestamp_ns = DataType::Timestamp(TimeUnit::Nanosecond, None); + let nested_fields: Fields = vec![ + Field::new("id", DataType::Int64, false), + Field::new("ts", timestamp_ms.clone(), true), + ] + .into(); + let nested: ArrayRef = Arc::new( + StructArray::try_new( + nested_fields.clone(), + vec![ + Arc::new(Int64Array::from(vec![10, 20])) as ArrayRef, + Arc::new(TimestampMillisecondArray::from(vec![Some(3), Some(4)])) as ArrayRef, + ], + None, + ) + .expect("nested struct"), + ); + let batch = RecordBatch::try_new( + Arc::new(Schema::new(vec![ + Field::new("ts", timestamp_ms, true), + Field::new("nested", DataType::Struct(nested_fields), true), + ])), + vec![ + Arc::new(TimestampMillisecondArray::from(vec![Some(1), Some(2)])) as ArrayRef, + nested, + ], + ) + .expect("batch"); + + let normalized = normalize_timestamp_units(batch).expect("normalize"); + + assert_eq!(normalized.schema().field(0).data_type(), ×tamp_ns); + let top = normalized + .column(0) + .as_any() + .downcast_ref::() + .expect("top timestamp ns"); + assert_eq!(top.value(0), 1_000_000); + match normalized.schema().field(1).data_type() { + DataType::Struct(fields) => { + assert_eq!(fields[1].data_type(), ×tamp_ns); + } + other => panic!("expected nested struct, got {other:?}"), + } + let nested = normalized + .column(1) + .as_any() + .downcast_ref::() + .expect("nested struct"); + let nested_ts = nested + .column(1) + .as_any() + .downcast_ref::() + .expect("nested timestamp ns"); + assert_eq!(nested_ts.value(1), 4_000_000); + } +} diff --git a/native/src/lib.rs b/native/src/lib.rs index a97bc413..07786ed2 100644 --- a/native/src/lib.rs +++ b/native/src/lib.rs @@ -63,6 +63,7 @@ mod expr; mod files; mod flatten; mod flink_text; +mod fluss; mod formats; mod group_agg; mod interval_join; @@ -88,7 +89,7 @@ mod window_join; #[allow(unused_imports)] pub(crate) use { aggregates::*, bridge::*, calc::*, changelog::*, dedup::*, exchange::*, expr::*, files::*, - flatten::*, formats::*, group_agg::*, interval_join::*, ipc::*, join_common::*, json::*, + flatten::*, fluss::*, formats::*, group_agg::*, interval_join::*, ipc::*, join_common::*, json::*, kafka::*, keys::*, memory::*, normalizer::*, over_agg::*, session_agg::*, sorter::*, temporal_join::*, topn::*, updating_join::*, window_agg::*, window_join::*, }; diff --git a/pom.xml b/pom.xml index 90af28c9..8d5995a8 100644 --- a/pom.xml +++ b/pom.xml @@ -29,6 +29,8 @@ 18.3.0 2.2.1 2.12 + 0.9.1-incubating + 3.8.3 4.32.1 + + org.apache.fluss + fluss-flink-2.2 + ${fluss.version} + provided + + + org.apache.fluss + fluss-server + ${fluss.version} + test + + + org.apache.fluss + fluss-server + ${fluss.version} + test-jar + test + + + org.apache.fluss + fluss-common + ${fluss.version} + test-jar + test + + + org.apache.fluss + fluss-rpc + ${fluss.version} + test-jar + test + + + org.apache.fluss + fluss-test-utils + ${fluss.version} + test + + + org.apache.curator + curator-test + 5.4.0 + test + + + + org.apache.zookeeper + zookeeper + + + + + org.apache.zookeeper + zookeeper + ${zookeeper.version} + test + + + org.apache.zookeeper + zookeeper-jute + ${zookeeper.version} + test + diff --git a/src/main/java/io/github/jordepic/streamfusion/Native.java b/src/main/java/io/github/jordepic/streamfusion/Native.java index f19d908d..b1d09fbe 100644 --- a/src/main/java/io/github/jordepic/streamfusion/Native.java +++ b/src/main/java/io/github/jordepic/streamfusion/Native.java @@ -926,6 +926,66 @@ public static native long benchmarkConsumeOnly( */ public static native boolean kafkaFeatureBuilt(); + /** Whether the loaded native library was built with the {@code fluss} cargo feature. */ + public static native boolean flussFeatureBuilt(); + + /** + * Opens a native Fluss log-table reader for one source subtask. Fluss' JVM enumerator assigns splits; + * this reader subscribes those concrete table buckets through fluss-rs and exports Arrow batches. + * + * @param configKeys translated fluss-rs config field names + * @param configValues values index-aligned with {@code configKeys} + * @param databaseName Fluss database name + * @param tableName Fluss table name + * @param projectedFields projected column indices, or empty for all columns + */ + public static native long openFlussReader( + String[] configKeys, + String[] configValues, + String databaseName, + String tableName, + int[] projectedFields); + + /** + * Adds assigned log splits to the native Fluss reader. Index-aligned arrays; {@code Long.MIN_VALUE} + * marks a non-partitioned split or no stopping offset. + */ + public static native void assignFlussSplits( + long handle, + String[] splitIds, + long[] tableIds, + long[] partitionIds, + long[] buckets, + long[] startOffsets, + long[] stoppingOffsets); + + /** + * Removes finished Fluss splits from the native scanner's subscriptions so bounded tails don't keep + * polling completed buckets. + */ + public static native void unassignFlussSplits( + long handle, long[] tableIds, long[] partitionIds, long[] buckets); + + /** + * Polls one Fluss scanner cycle and returns the number of per-split Arrow batches ready to drain. + */ + public static native int pollFlussBatch(long handle, long timeoutMillis); + + /** + * Drains one pending Fluss batch, writes {@code [nextOffset]} into {@code splitMeta}, writes the + * split id into {@code outSplitId[0]}, exports Arrow into the consumer C structs, and returns the row + * count. Call it {@link #pollFlussBatch}'s return-value times. + */ + public static native int drainFlussSplit( + long handle, + long[] splitMeta, + String[] outSplitId, + long outArrayAddress, + long outSchemaAddress); + + /** Releases a native Fluss reader and closes its fluss-rs connection. */ + public static native void closeFlussReader(long handle); + /** * Opens a native Kafka split reader for one subtask and returns an opaque handle, released with * {@link #closeKafkaConsumer}. One rdkafka consumer multiplexes the subtask's partitions; splits are diff --git a/src/main/java/io/github/jordepic/streamfusion/fluss/FlussConfigTranslator.java b/src/main/java/io/github/jordepic/streamfusion/fluss/FlussConfigTranslator.java new file mode 100644 index 00000000..c41d4bef --- /dev/null +++ b/src/main/java/io/github/jordepic/streamfusion/fluss/FlussConfigTranslator.java @@ -0,0 +1,180 @@ +package io.github.jordepic.streamfusion.fluss; + +import java.util.LinkedHashMap; +import java.util.Map; +import java.util.Optional; +import org.apache.flink.configuration.MemorySize; +import org.apache.flink.util.TimeUtils; + +/** + * Translates the Fluss Java/Flink table options that belong to the Fluss client into the field names + * used by {@code fluss-rs::Config}. Flink source-coordination options are deliberately kept out of + * the native config: those decide startup offsets, partition discovery, split assignment, and + * checkpoints on the JVM side. Writer/lookup client options are likewise dropped without falling + * back — they configure the write and lookup paths, which a read-only source never executes. Any + * other {@code client.*} option without a known fluss-rs mapping produces a fallback, so a setting + * the native path cannot honor declines instead of silently diverging. + */ +public final class FlussConfigTranslator { + + private FlussConfigTranslator() {} + + public static final class Result { + private final Map config; + private final Map coordinationOptions; + private final String fallbackReason; + + private Result( + Map config, Map coordinationOptions, String fallbackReason) { + this.config = config; + this.coordinationOptions = coordinationOptions; + this.fallbackReason = fallbackReason; + } + + static Result translated(Map config, Map coordinationOptions) { + return new Result(config, coordinationOptions, null); + } + + static Result fallback(String reason) { + return new Result(null, Map.of(), reason); + } + + public boolean isTranslated() { + return config != null; + } + + public Map config() { + return config; + } + + public Map coordinationOptions() { + return coordinationOptions; + } + + public Optional fallbackReason() { + return Optional.ofNullable(fallbackReason); + } + } + + private static final Map DIRECT = + Map.ofEntries( + Map.entry("bootstrap.servers", "bootstrap_servers"), + Map.entry("client.scanner.log.max-poll-records", "scanner_log_max_poll_records"), + Map.entry("client.scanner.remote-log.prefetch-num", "scanner_remote_log_prefetch_num"), + Map.entry("client.remote-file.download-thread-num", "remote_file_download_thread_num"), + Map.entry("client.security.protocol", "security_protocol"), + Map.entry("client.security.sasl.mechanism", "security_sasl_mechanism"), + Map.entry("client.security.sasl.username", "security_sasl_username"), + Map.entry("client.security.sasl.password", "security_sasl_password")); + + private static final Map MEMORY = + Map.ofEntries( + Map.entry("client.scanner.log.fetch.max-bytes", "scanner_log_fetch_max_bytes"), + Map.entry( + "client.scanner.log.fetch.max-bytes-for-bucket", + "scanner_log_fetch_max_bytes_for_bucket"), + Map.entry("client.scanner.log.fetch.min-bytes", "scanner_log_fetch_min_bytes")); + + private static final Map DURATION = + Map.ofEntries( + Map.entry("client.connect-timeout", "connect_timeout_ms"), + Map.entry("client.scanner.log.fetch.wait-max-time", "scanner_log_fetch_wait_max_time_ms")); + + private static final String[] COORDINATION = { + "scan.startup.mode", + "scan.startup.timestamp", + "scan.partition.discovery.interval", + "scan.kv.snapshot.lease.id", + "scan.kv.snapshot.lease.duration" + }; + + private static final String[] NO_RUST_CONFIG = { + "client.id", + "client.request-timeout", + "client.scanner.log.check-crc", + "client.scanner.io.tmpdir", + "client.metrics.enabled", + "client.security.sasl.jaas.config" + }; + + // Write/lookup-path client options: legitimate table/client properties, but they configure paths a + // read-only source never executes, so they are ignored rather than translated or fallback-triggering. + private static final String[] READ_IRRELEVANT_PREFIXES = {"client.writer.", "client.lookup."}; + + /** Translates known client options, or returns a fallback reason for a setting not safely mirrored. */ + public static Result translate(Map options) { + Map out = new LinkedHashMap<>(); + Map coordination = new LinkedHashMap<>(); + + for (String key : COORDINATION) { + if (options.containsKey(key)) { + coordination.put(key, options.get(key)); + } + } + + for (String key : NO_RUST_CONFIG) { + if (options.containsKey(key)) { + return Result.fallback("no fluss-rs Config field for " + key); + } + } + + for (String key : options.keySet()) { + if (key.startsWith("client.") + && !DIRECT.containsKey(key) + && !MEMORY.containsKey(key) + && !DURATION.containsKey(key) + && !readIrrelevant(key)) { + return Result.fallback("unrecognized Fluss client option " + key); + } + } + + for (Map.Entry entry : DIRECT.entrySet()) { + if (options.containsKey(entry.getKey())) { + out.put(entry.getValue(), options.get(entry.getKey())); + } + } + for (Map.Entry entry : MEMORY.entrySet()) { + if (options.containsKey(entry.getKey())) { + out.put(entry.getValue(), Long.toString(parseMemoryBytes(options.get(entry.getKey())))); + } + } + for (Map.Entry entry : DURATION.entrySet()) { + if (options.containsKey(entry.getKey())) { + out.put(entry.getValue(), Long.toString(parseDurationMillis(options.get(entry.getKey())))); + } + } + + String protocol = out.get("security_protocol"); + if (protocol != null + && !protocol.equalsIgnoreCase("plaintext") + && !protocol.equalsIgnoreCase("sasl")) { + return Result.fallback( + "fluss-rs supports only the PLAINTEXT and SASL security protocols, not " + protocol); + } + String mechanism = out.getOrDefault("security_sasl_mechanism", "PLAIN"); + if (protocol != null + && protocol.equalsIgnoreCase("sasl") + && !mechanism.equalsIgnoreCase("PLAIN")) { + return Result.fallback("fluss-rs currently supports only PLAIN SASL"); + } + + return Result.translated(out, coordination); + } + + private static boolean readIrrelevant(String key) { + for (String prefix : READ_IRRELEVANT_PREFIXES) { + if (key.startsWith(prefix)) { + return true; + } + } + return false; + } + + private static long parseMemoryBytes(String value) { + return MemorySize.parseBytes(value); + } + + private static long parseDurationMillis(String value) { + return TimeUtils.parseDuration(value).toMillis(); + } +} diff --git a/src/main/java/io/github/jordepic/streamfusion/fluss/FlussSplitTranslator.java b/src/main/java/io/github/jordepic/streamfusion/fluss/FlussSplitTranslator.java new file mode 100644 index 00000000..55a51962 --- /dev/null +++ b/src/main/java/io/github/jordepic/streamfusion/fluss/FlussSplitTranslator.java @@ -0,0 +1,32 @@ +package io.github.jordepic.streamfusion.fluss; + +import org.apache.fluss.flink.source.split.LogSplit; +import org.apache.fluss.flink.source.split.SourceSplitBase; +import org.apache.fluss.metadata.TableBucket; + +/** Converts Fluss's public source splits into the minimal native log-reader assignment. */ +public final class FlussSplitTranslator { + + private FlussSplitTranslator() {} + + public static boolean isNativeLogSplit(SourceSplitBase split) { + return split.isLogSplit(); + } + + public static NativeFlussLogSplit translateLogSplit(SourceSplitBase split) { + if (!split.isLogSplit()) { + throw new IllegalArgumentException( + "native Fluss proof path only accepts log splits, got " + split.getClass().getSimpleName()); + } + LogSplit log = split.asLogSplit(); + TableBucket bucket = log.getTableBucket(); + return new NativeFlussLogSplit( + log.splitId(), + bucket.getTableId(), + bucket.getPartitionId(), + log.getPartitionName(), + bucket.getBucket(), + log.getStartingOffset(), + log.getStoppingOffset().orElse(null)); + } +} diff --git a/src/main/java/io/github/jordepic/streamfusion/fluss/NativeFlussFetcherManager.java b/src/main/java/io/github/jordepic/streamfusion/fluss/NativeFlussFetcherManager.java new file mode 100644 index 00000000..d1ac4887 --- /dev/null +++ b/src/main/java/io/github/jordepic/streamfusion/fluss/NativeFlussFetcherManager.java @@ -0,0 +1,60 @@ +package io.github.jordepic.streamfusion.fluss; + +import java.util.List; +import java.util.Set; +import java.util.function.Consumer; +import java.util.function.Supplier; +import org.apache.flink.configuration.Configuration; +import org.apache.flink.connector.base.source.reader.fetcher.SingleThreadFetcherManager; +import org.apache.flink.connector.base.source.reader.fetcher.SplitFetcher; +import org.apache.flink.connector.base.source.reader.fetcher.SplitFetcherTask; +import org.apache.flink.connector.base.source.reader.splitreader.SplitReader; +import org.apache.fluss.flink.source.split.SourceSplitBase; +import org.apache.fluss.metadata.TableBucket; + +/** Single-thread Fluss fetcher manager with the partition-removal ACK hook Fluss expects. */ +final class NativeFlussFetcherManager + extends SingleThreadFetcherManager { + + NativeFlussFetcherManager( + Supplier> splitReaderSupplier, + Configuration config) { + super(splitReaderSupplier, config); + } + + void removeSplitsAndAck( + List splits, + Set removedBuckets, + Consumer> unsubscribeCallback) { + SplitFetcher fetcher = getRunningFetcher(); + if (fetcher == null) { + unsubscribeCallback.accept(removedBuckets); + return; + } + + Set bucketsToAck = Set.copyOf(removedBuckets); + fetcher.removeSplits(splits); + fetcher.enqueueTask(new PartitionRemovalAckTask(bucketsToAck, unsubscribeCallback)); + } + + private final class PartitionRemovalAckTask implements SplitFetcherTask { + private final Set removedBuckets; + private final Consumer> unsubscribeCallback; + + private PartitionRemovalAckTask( + Set removedBuckets, + Consumer> unsubscribeCallback) { + this.removedBuckets = removedBuckets; + this.unsubscribeCallback = unsubscribeCallback; + } + + @Override + public boolean run() { + unsubscribeCallback.accept(removedBuckets); + return true; + } + + @Override + public void wakeUp() {} + } +} diff --git a/src/main/java/io/github/jordepic/streamfusion/fluss/NativeFlussLogSplit.java b/src/main/java/io/github/jordepic/streamfusion/fluss/NativeFlussLogSplit.java new file mode 100644 index 00000000..0d4c1967 --- /dev/null +++ b/src/main/java/io/github/jordepic/streamfusion/fluss/NativeFlussLogSplit.java @@ -0,0 +1,60 @@ +package io.github.jordepic.streamfusion.fluss; + +import java.util.OptionalLong; + +/** Concrete log-table split assignment passed from Flink's Fluss enumerator to the native reader. */ +public final class NativeFlussLogSplit { + + private final String splitId; + private final long tableId; + private final Long partitionId; + private final String partitionName; + private final int bucket; + private final long startingOffset; + private final Long stoppingOffset; + + NativeFlussLogSplit( + String splitId, + long tableId, + Long partitionId, + String partitionName, + int bucket, + long startingOffset, + Long stoppingOffset) { + this.splitId = splitId; + this.tableId = tableId; + this.partitionId = partitionId; + this.partitionName = partitionName; + this.bucket = bucket; + this.startingOffset = startingOffset; + this.stoppingOffset = stoppingOffset; + } + + public String splitId() { + return splitId; + } + + public long tableId() { + return tableId; + } + + public OptionalLong partitionId() { + return partitionId == null ? OptionalLong.empty() : OptionalLong.of(partitionId); + } + + public String partitionName() { + return partitionName; + } + + public int bucket() { + return bucket; + } + + public long startingOffset() { + return startingOffset; + } + + public OptionalLong stoppingOffset() { + return stoppingOffset == null ? OptionalLong.empty() : OptionalLong.of(stoppingOffset); + } +} diff --git a/src/main/java/io/github/jordepic/streamfusion/fluss/NativeFlussRecord.java b/src/main/java/io/github/jordepic/streamfusion/fluss/NativeFlussRecord.java new file mode 100644 index 00000000..718bdc2d --- /dev/null +++ b/src/main/java/io/github/jordepic/streamfusion/fluss/NativeFlussRecord.java @@ -0,0 +1,23 @@ +package io.github.jordepic.streamfusion.fluss; + +import io.github.jordepic.streamfusion.operator.ArrowBatch; + +/** One Arrow batch from a Fluss log split plus the split offset after the batch. */ +final class NativeFlussRecord { + + private final ArrowBatch batch; + private final long nextOffset; + + NativeFlussRecord(ArrowBatch batch, long nextOffset) { + this.batch = batch; + this.nextOffset = nextOffset; + } + + ArrowBatch batch() { + return batch; + } + + long nextOffset() { + return nextOffset; + } +} diff --git a/src/main/java/io/github/jordepic/streamfusion/fluss/NativeFlussRecordEmitter.java b/src/main/java/io/github/jordepic/streamfusion/fluss/NativeFlussRecordEmitter.java new file mode 100644 index 00000000..647106a7 --- /dev/null +++ b/src/main/java/io/github/jordepic/streamfusion/fluss/NativeFlussRecordEmitter.java @@ -0,0 +1,18 @@ +package io.github.jordepic.streamfusion.fluss; + +import io.github.jordepic.streamfusion.operator.ArrowBatch; +import org.apache.flink.api.connector.source.SourceOutput; +import org.apache.flink.connector.base.source.reader.RecordEmitter; +import org.apache.fluss.flink.source.split.SourceSplitState; + +/** Emits native Fluss Arrow batches and advances the Fluss log split state. */ +final class NativeFlussRecordEmitter + implements RecordEmitter { + + @Override + public void emitRecord( + NativeFlussRecord record, SourceOutput output, SourceSplitState splitState) { + output.collect(record.batch()); + splitState.asLogSplitState().setNextOffset(record.nextOffset()); + } +} diff --git a/src/main/java/io/github/jordepic/streamfusion/fluss/NativeFlussSource.java b/src/main/java/io/github/jordepic/streamfusion/fluss/NativeFlussSource.java new file mode 100644 index 00000000..76e32547 --- /dev/null +++ b/src/main/java/io/github/jordepic/streamfusion/fluss/NativeFlussSource.java @@ -0,0 +1,158 @@ +package io.github.jordepic.streamfusion.fluss; + +import io.github.jordepic.streamfusion.operator.ArrowBatch; +import java.io.IOException; +import java.util.List; +import java.util.Map; +import java.util.Set; +import java.util.function.Supplier; +import org.apache.flink.api.connector.source.Boundedness; +import org.apache.flink.api.connector.source.Source; +import org.apache.flink.api.connector.source.SourceReader; +import org.apache.flink.api.connector.source.SourceReaderContext; +import org.apache.flink.api.connector.source.SplitEnumerator; +import org.apache.flink.api.connector.source.SplitEnumeratorContext; +import org.apache.flink.connector.base.source.reader.splitreader.SplitReader; +import org.apache.flink.core.io.SimpleVersionedSerializer; +import org.apache.fluss.client.initializer.OffsetsInitializer; +import org.apache.fluss.flink.source.enumerator.FlinkSourceEnumerator; +import org.apache.fluss.flink.source.reader.LeaseContext; +import org.apache.fluss.flink.source.split.SourceSplitBase; +import org.apache.fluss.flink.source.split.SourceSplitSerializer; +import org.apache.fluss.flink.source.state.FlussSourceEnumeratorStateSerializer; +import org.apache.fluss.flink.source.state.SourceEnumeratorState; +import org.apache.fluss.lake.source.LakeSource; +import org.apache.fluss.lake.source.LakeSplit; +import org.apache.fluss.metadata.TableBucket; +import org.apache.fluss.metadata.TablePath; +import org.apache.fluss.predicate.Predicate; + +/** + * Native Fluss source for log tables. The JVM keeps Fluss' enumerator, split serializers, startup + * offsets, and checkpoint state; the task-side reader consumes assigned log splits with fluss-rs and + * emits Arrow batches directly. + */ +public final class NativeFlussSource + implements Source { + + private static final long serialVersionUID = 1L; + private static final long POLL_TIMEOUT_MILLIS = 100L; + + private final org.apache.fluss.config.Configuration flussConfig; + private final TablePath tablePath; + private final boolean hasPrimaryKey; + private final boolean partitioned; + private final OffsetsInitializer offsetsInitializer; + private final long scanPartitionDiscoveryIntervalMs; + private final boolean streaming; + private final Predicate partitionFilters; + private final LakeSource lakeSource; + private final LeaseContext leaseContext; + private final String[] nativeConfigKeys; + private final String[] nativeConfigValues; + private final int[] projectedFields; + + public NativeFlussSource( + org.apache.fluss.config.Configuration flussConfig, + TablePath tablePath, + boolean hasPrimaryKey, + boolean partitioned, + OffsetsInitializer offsetsInitializer, + long scanPartitionDiscoveryIntervalMs, + boolean streaming, + Predicate partitionFilters, + LakeSource lakeSource, + LeaseContext leaseContext, + String[] nativeConfigKeys, + String[] nativeConfigValues, + int[] projectedFields) { + this.flussConfig = flussConfig; + this.tablePath = tablePath; + this.hasPrimaryKey = hasPrimaryKey; + this.partitioned = partitioned; + this.offsetsInitializer = offsetsInitializer; + this.scanPartitionDiscoveryIntervalMs = scanPartitionDiscoveryIntervalMs; + this.streaming = streaming; + this.partitionFilters = partitionFilters; + this.lakeSource = lakeSource; + this.leaseContext = leaseContext; + this.nativeConfigKeys = nativeConfigKeys; + this.nativeConfigValues = nativeConfigValues; + this.projectedFields = projectedFields; + } + + @Override + public Boundedness getBoundedness() { + return streaming ? Boundedness.CONTINUOUS_UNBOUNDED : Boundedness.BOUNDED; + } + + @Override + public SourceReader createReader(SourceReaderContext context) { + Supplier> splitReaderSupplier = + () -> + new NativeFlussSplitReader( + nativeConfigKeys, + nativeConfigValues, + tablePath.getDatabaseName(), + tablePath.getTableName(), + projectedFields, + POLL_TIMEOUT_MILLIS); + return new NativeFlussSourceReader( + splitReaderSupplier, new NativeFlussRecordEmitter(), context.getConfiguration(), context); + } + + @Override + public SplitEnumerator createEnumerator( + SplitEnumeratorContext enumContext) { + return new FlinkSourceEnumerator( + tablePath, + flussConfig, + hasPrimaryKey, + partitioned, + enumContext, + offsetsInitializer, + scanPartitionDiscoveryIntervalMs, + streaming, + partitionFilters, + lakeSource, + leaseContext, + false); + } + + @Override + public SplitEnumerator restoreEnumerator( + SplitEnumeratorContext enumContext, SourceEnumeratorState checkpoint) + throws IOException { + Set assignedBuckets = checkpoint.getAssignedBuckets(); + Map assignedPartitions = checkpoint.getAssignedPartitions(); + List remainingSplits = checkpoint.getRemainingHybridLakeFlussSplits(); + LeaseContext restoredLeaseContext = + new LeaseContext(checkpoint.getLeaseId(), leaseContext.getKvSnapshotLeaseDurationMs()); + return new FlinkSourceEnumerator( + tablePath, + flussConfig, + hasPrimaryKey, + partitioned, + enumContext, + assignedBuckets, + assignedPartitions, + remainingSplits, + offsetsInitializer, + scanPartitionDiscoveryIntervalMs, + streaming, + partitionFilters, + lakeSource, + restoredLeaseContext, + true); + } + + @Override + public SimpleVersionedSerializer getSplitSerializer() { + return new SourceSplitSerializer(lakeSource); + } + + @Override + public SimpleVersionedSerializer getEnumeratorCheckpointSerializer() { + return new FlussSourceEnumeratorStateSerializer(lakeSource); + } +} diff --git a/src/main/java/io/github/jordepic/streamfusion/fluss/NativeFlussSourceReader.java b/src/main/java/io/github/jordepic/streamfusion/fluss/NativeFlussSourceReader.java new file mode 100644 index 00000000..94f15bc9 --- /dev/null +++ b/src/main/java/io/github/jordepic/streamfusion/fluss/NativeFlussSourceReader.java @@ -0,0 +1,118 @@ +package io.github.jordepic.streamfusion.fluss; + +import io.github.jordepic.streamfusion.operator.ArrowBatch; +import java.util.ArrayList; +import java.util.HashMap; +import java.util.HashSet; +import java.util.List; +import java.util.Map; +import java.util.Set; +import java.util.function.Supplier; +import org.apache.flink.api.connector.source.SourceEvent; +import org.apache.flink.api.connector.source.SourceReaderContext; +import org.apache.flink.configuration.Configuration; +import org.apache.flink.connector.base.source.reader.RecordEmitter; +import org.apache.flink.connector.base.source.reader.SingleThreadMultiplexSourceReaderBase; +import org.apache.flink.connector.base.source.reader.splitreader.SplitReader; +import org.apache.fluss.flink.source.event.PartitionBucketsUnsubscribedEvent; +import org.apache.fluss.flink.source.event.PartitionsRemovedEvent; +import org.apache.fluss.flink.source.split.LogSplitState; +import org.apache.fluss.flink.source.split.SourceSplitBase; +import org.apache.fluss.flink.source.split.SourceSplitState; +import org.apache.fluss.metadata.TableBucket; + +/** FLIP-27 source reader that swaps Fluss' row split reader for a native Arrow log reader. */ +final class NativeFlussSourceReader + extends SingleThreadMultiplexSourceReaderBase< + NativeFlussRecord, ArrowBatch, SourceSplitBase, SourceSplitState> { + + private final NativeFlussFetcherManager fetcherManager; + private final Map assignedSplits = new HashMap<>(); + + NativeFlussSourceReader( + Supplier> splitReaderSupplier, + RecordEmitter recordEmitter, + Configuration config, + SourceReaderContext context) { + this(new NativeFlussFetcherManager(splitReaderSupplier, config), recordEmitter, config, context); + } + + private NativeFlussSourceReader( + NativeFlussFetcherManager fetcherManager, + RecordEmitter recordEmitter, + Configuration config, + SourceReaderContext context) { + super(fetcherManager, recordEmitter, config, context); + this.fetcherManager = fetcherManager; + } + + @Override + public void addSplits(List splits) { + super.addSplits(splits); + for (SourceSplitBase split : splits) { + assignedSplits.put(split.splitId(), split); + } + } + + @Override + public void handleSourceEvents(SourceEvent sourceEvent) { + if (sourceEvent instanceof PartitionsRemovedEvent) { + handlePartitionsRemoved(((PartitionsRemovedEvent) sourceEvent).getRemovedPartitions()); + return; + } + super.handleSourceEvents(sourceEvent); + } + + @Override + protected void onSplitFinished(Map finishedSplitIds) { + for (String splitId : finishedSplitIds.keySet()) { + assignedSplits.remove(splitId); + } + } + + @Override + protected SourceSplitState initializedState(SourceSplitBase split) { + if (!split.isLogSplit()) { + throw new IllegalArgumentException( + "native Fluss source supports log splits only, got " + split.getClass().getSimpleName()); + } + return new LogSplitState(split.asLogSplit()); + } + + @Override + protected SourceSplitBase toSplitType(String splitId, SourceSplitState splitState) { + return splitState.toSourceSplit(); + } + + private void handlePartitionsRemoved(Map removedPartitions) { + if (removedPartitions.isEmpty()) { + context.sendSourceEventToCoordinator(new PartitionBucketsUnsubscribedEvent(Set.of())); + return; + } + + List splitsToRemove = new ArrayList<>(); + Set bucketsToAck = new HashSet<>(); + for (SourceSplitBase split : assignedSplits.values()) { + TableBucket tableBucket = split.getTableBucket(); + Long partitionId = tableBucket.getPartitionId(); + if (partitionId != null && removedPartitions.containsKey(partitionId)) { + splitsToRemove.add(split); + bucketsToAck.add(tableBucket); + } + } + + if (splitsToRemove.isEmpty()) { + context.sendSourceEventToCoordinator(new PartitionBucketsUnsubscribedEvent(Set.of())); + return; + } + + for (SourceSplitBase split : splitsToRemove) { + assignedSplits.remove(split.splitId()); + } + fetcherManager.removeSplitsAndAck( + splitsToRemove, + bucketsToAck, + buckets -> + context.sendSourceEventToCoordinator(new PartitionBucketsUnsubscribedEvent(buckets))); + } +} diff --git a/src/main/java/io/github/jordepic/streamfusion/fluss/NativeFlussSplitReader.java b/src/main/java/io/github/jordepic/streamfusion/fluss/NativeFlussSplitReader.java new file mode 100644 index 00000000..b0710d28 --- /dev/null +++ b/src/main/java/io/github/jordepic/streamfusion/fluss/NativeFlussSplitReader.java @@ -0,0 +1,212 @@ +package io.github.jordepic.streamfusion.fluss; + +import io.github.jordepic.streamfusion.Native; +import io.github.jordepic.streamfusion.operator.ArrowBatch; +import io.github.jordepic.streamfusion.operator.NativeAllocator; +import java.util.ArrayList; +import java.util.HashMap; +import java.util.HashSet; +import java.util.List; +import java.util.Map; +import java.util.OptionalLong; +import java.util.Set; +import org.apache.arrow.c.ArrowArray; +import org.apache.arrow.c.ArrowSchema; +import org.apache.arrow.c.Data; +import org.apache.arrow.memory.BufferAllocator; +import org.apache.arrow.vector.VectorSchemaRoot; +import org.apache.flink.connector.base.source.reader.RecordsBySplits; +import org.apache.flink.connector.base.source.reader.RecordsWithSplitIds; +import org.apache.flink.connector.base.source.reader.splitreader.SplitReader; +import org.apache.flink.connector.base.source.reader.splitreader.SplitsChange; +import org.apache.flink.connector.base.source.reader.splitreader.SplitsRemoval; +import org.apache.fluss.flink.source.split.SourceSplitBase; + +/** + * Native Fluss split reader for one Flink subtask. Fluss' enumerator assigns concrete log splits; + * this reader subscribes those buckets in fluss-rs and drains Arrow {@code RecordBatch}es directly. + */ +final class NativeFlussSplitReader implements SplitReader { + + private static final long NO_PARTITION = Long.MIN_VALUE; + private static final long NO_STOPPING_OFFSET = Long.MIN_VALUE; + + private final long handle; + private final long pollTimeoutMillis; + private final BufferAllocator allocator = NativeAllocator.SHARED; + private final Map stoppingOffsets = new HashMap<>(); + private final Map positions = new HashMap<>(); + private final Map splitsById = new HashMap<>(); + private final Set finished = new HashSet<>(); + private final Set pendingFinishedSplits = new HashSet<>(); + + NativeFlussSplitReader( + String[] configKeys, + String[] configValues, + String databaseName, + String tableName, + int[] projectedFields, + long pollTimeoutMillis) { + this.pollTimeoutMillis = pollTimeoutMillis; + this.handle = + Native.openFlussReader(configKeys, configValues, databaseName, tableName, projectedFields); + } + + @Override + public RecordsWithSplitIds fetch() { + if (!pendingFinishedSplits.isEmpty()) { + RecordsBySplits.Builder finishedBuilder = new RecordsBySplits.Builder<>(); + finishedBuilder.addFinishedSplits(pendingFinishedSplits); + pendingFinishedSplits.clear(); + return finishedBuilder.build(); + } + int pending = Native.pollFlussBatch(handle, pollTimeoutMillis); + RecordsBySplits.Builder builder = new RecordsBySplits.Builder<>(); + for (int i = 0; i < pending; i++) { + try (ArrowArray outArray = ArrowArray.allocateNew(allocator); + ArrowSchema outSchema = ArrowSchema.allocateNew(allocator)) { + long[] meta = new long[1]; + String[] splitId = new String[1]; + Native.drainFlussSplit( + handle, meta, splitId, outArray.memoryAddress(), outSchema.memoryAddress()); + VectorSchemaRoot root = + Data.importVectorSchemaRoot(allocator, outArray, outSchema, NativeAllocator.DICTIONARIES); + positions.put(splitId[0], meta[0]); + builder.add(splitId[0], new NativeFlussRecord(new ArrowBatch(root), meta[0])); + } + } + + List justFinished = new ArrayList<>(); + for (Map.Entry stop : stoppingOffsets.entrySet()) { + String splitId = stop.getKey(); + if (!finished.contains(splitId) + && positions.getOrDefault(splitId, Long.MIN_VALUE) >= stop.getValue()) { + builder.addFinishedSplit(splitId); + finished.add(splitId); + justFinished.add(splitsById.get(splitId)); + } + } + if (!justFinished.isEmpty()) { + Native.unassignFlussSplits( + handle, + tableIds(justFinished), + partitionIds(justFinished), + buckets(justFinished)); + } + return builder.build(); + } + + @Override + public void handleSplitsChanges(SplitsChange splitsChanges) { + if (splitsChanges instanceof SplitsRemoval) { + removeSplits(splitsChanges.splits()); + return; + } + List splits = splitsChanges.splits(); + List nativeSplits = new ArrayList<>(splits.size()); + for (SourceSplitBase split : splits) { + NativeFlussLogSplit nativeSplit = FlussSplitTranslator.translateLogSplit(split); + OptionalLong stoppingOffset = nativeSplit.stoppingOffset(); + if (stoppingOffset.isPresent() && nativeSplit.startingOffset() >= stoppingOffset.getAsLong()) { + pendingFinishedSplits.add(nativeSplit.splitId()); + continue; + } + nativeSplits.add(nativeSplit); + splitsById.put(nativeSplit.splitId(), nativeSplit); + positions.put(nativeSplit.splitId(), nativeSplit.startingOffset()); + stoppingOffset.ifPresent(stop -> stoppingOffsets.put(nativeSplit.splitId(), stop)); + } + if (!nativeSplits.isEmpty()) { + Native.assignFlussSplits( + handle, + splitIds(nativeSplits), + tableIds(nativeSplits), + partitionIds(nativeSplits), + buckets(nativeSplits), + startingOffsets(nativeSplits), + stoppingOffsets(nativeSplits)); + } + } + + private void removeSplits(List splits) { + List nativeSplits = new ArrayList<>(splits.size()); + for (SourceSplitBase split : splits) { + NativeFlussLogSplit nativeSplit = splitsById.remove(split.splitId()); + boolean assigned = nativeSplit != null; + if (nativeSplit == null) { + nativeSplit = FlussSplitTranslator.translateLogSplit(split); + } + nativeSplits.add(nativeSplit); + stoppingOffsets.remove(nativeSplit.splitId()); + positions.remove(nativeSplit.splitId()); + if (assigned && !finished.remove(nativeSplit.splitId())) { + pendingFinishedSplits.add(nativeSplit.splitId()); + } + } + if (!nativeSplits.isEmpty()) { + Native.unassignFlussSplits( + handle, + tableIds(nativeSplits), + partitionIds(nativeSplits), + buckets(nativeSplits)); + } + } + + @Override + public void wakeUp() { + // Native poll uses a short bounded timeout; no interrupt is needed. + } + + @Override + public void close() { + Native.closeFlussReader(handle); + } + + private static String[] splitIds(List splits) { + String[] values = new String[splits.size()]; + for (int i = 0; i < splits.size(); i++) { + values[i] = splits.get(i).splitId(); + } + return values; + } + + private static long[] tableIds(List splits) { + long[] values = new long[splits.size()]; + for (int i = 0; i < splits.size(); i++) { + values[i] = splits.get(i).tableId(); + } + return values; + } + + private static long[] partitionIds(List splits) { + long[] values = new long[splits.size()]; + for (int i = 0; i < splits.size(); i++) { + values[i] = splits.get(i).partitionId().orElse(NO_PARTITION); + } + return values; + } + + private static long[] buckets(List splits) { + long[] values = new long[splits.size()]; + for (int i = 0; i < splits.size(); i++) { + values[i] = splits.get(i).bucket(); + } + return values; + } + + private static long[] startingOffsets(List splits) { + long[] values = new long[splits.size()]; + for (int i = 0; i < splits.size(); i++) { + values[i] = splits.get(i).startingOffset(); + } + return values; + } + + private static long[] stoppingOffsets(List splits) { + long[] values = new long[splits.size()]; + for (int i = 0; i < splits.size(); i++) { + values[i] = splits.get(i).stoppingOffset().orElse(NO_STOPPING_OFFSET); + } + return values; + } +} diff --git a/src/main/java/io/github/jordepic/streamfusion/planner/FlussTables.java b/src/main/java/io/github/jordepic/streamfusion/planner/FlussTables.java new file mode 100644 index 00000000..7424d00a --- /dev/null +++ b/src/main/java/io/github/jordepic/streamfusion/planner/FlussTables.java @@ -0,0 +1,339 @@ +package io.github.jordepic.streamfusion.planner; + +import io.github.jordepic.streamfusion.Native; +import io.github.jordepic.streamfusion.fluss.FlussConfigTranslator; +import io.github.jordepic.streamfusion.fluss.NativeFlussSource; +import java.lang.reflect.Field; +import java.util.Map; +import org.apache.flink.table.connector.source.DynamicTableSource; +import org.apache.flink.table.planner.plan.nodes.physical.stream.StreamPhysicalTableSourceScan; +import org.apache.flink.table.planner.plan.schema.TableSourceTable; +import org.apache.flink.table.types.logical.LogicalType; +import org.apache.flink.table.types.logical.RowType; +import org.apache.fluss.client.initializer.OffsetsInitializer; +import org.apache.fluss.flink.FlinkConnectorOptions.ScanStartupMode; +import org.apache.fluss.flink.source.FlinkTableSource; +import org.apache.fluss.flink.source.reader.LeaseContext; +import org.apache.fluss.flink.utils.FlinkConnectorOptionsUtils.StartupOptions; +import org.apache.fluss.lake.source.LakeSource; +import org.apache.fluss.lake.source.LakeSplit; +import org.apache.fluss.metadata.TablePath; +import org.apache.fluss.predicate.Predicate; + +/** Planner support for replacing Fluss log-table scans with the native fluss-rs reader. */ +final class FlussTables { + + private FlussTables() {} + + /** Null when the native Fluss source can faithfully run this scan, else the fallback reason. */ + static String fallbackReason(StreamPhysicalTableSourceScan scan) { + if (!Native.flussFeatureBuilt()) { + return "the native library was built without the fluss feature"; + } + return plan(scan).fallbackReason; + } + + static NativeFlussSource build(StreamPhysicalTableSourceScan scan) { + Planned planned = plan(scan); + if (planned.plan == null) { + throw new IllegalArgumentException( + "Fluss scan is not supported by the native Fluss source: " + planned.fallbackReason); + } + SourcePlan plan = planned.plan; + return new NativeFlussSource( + plan.flussConfig, + plan.tablePath, + plan.hasPrimaryKey, + plan.partitioned, + plan.offsetsInitializer, + plan.scanPartitionDiscoveryIntervalMs, + plan.streaming, + plan.partitionFilters, + plan.lakeSource, + plan.leaseContext, + plan.nativeConfigKeys, + plan.nativeConfigValues, + plan.projectedFields); + } + + private static Planned plan(StreamPhysicalTableSourceScan scan) { + if (scan.requireWatermark()) { + return Planned.fallback( + "the table's WATERMARK is pushed into the scan, which the native source does not" + + " regenerate"); + } + if (!FilesystemTables.allPhysicalColumns(scan)) { + return Planned.fallback("metadata/computed columns are not produced natively"); + } + DynamicTableSource source = tableSource(scan); + if (!(source instanceof FlinkTableSource)) { + return Planned.fallback("the scan's table source is not the Fluss FlinkTableSource"); + } + Map options = FilesystemTables.options(scan); + if (options == null) { + return Planned.fallback("the table's connector options could not be resolved"); + } + // fluss-rs's RecordBatchLogScanner reads only the ARROW log format (its scan validation errors + // on any other); the option key/default mirror Fluss' ConfigOptions.TABLE_LOG_FORMAT. + String logFormat = options.getOrDefault("table.log.format", "ARROW"); + if (!"ARROW".equalsIgnoreCase(logFormat)) { + return Planned.fallback( + "table.log.format=" + logFormat + " — fluss-rs scans only the ARROW log format"); + } + FlinkTableSource flussSource = (FlinkTableSource) source; + try { + TablePath tablePath = field(flussSource, "tablePath", TablePath.class); + org.apache.fluss.config.Configuration flussConfig = + field(flussSource, "flussConfig", org.apache.fluss.config.Configuration.class); + int[] primaryKeyIndexes = field(flussSource, "primaryKeyIndexes", int[].class); + int[] partitionKeyIndexes = field(flussSource, "partitionKeyIndexes", int[].class); + boolean streaming = field(flussSource, "streaming", Boolean.class); + StartupOptions startupOptions = field(flussSource, "startupOptions", StartupOptions.class); + long scanPartitionDiscoveryIntervalMs = + field(flussSource, "scanPartitionDiscoveryIntervalMs", Long.class); + boolean isDataLakeEnabled = field(flussSource, "isDataLakeEnabled", Boolean.class); + LeaseContext leaseContext = field(flussSource, "leaseContext", LeaseContext.class); + int[] projectedFields = field(flussSource, "projectedFields", int[].class); + Object singleRowFilter = fieldObject(flussSource, "singleRowFilter"); + Object modificationScanType = fieldObject(flussSource, "modificationScanType"); + boolean selectRowCount = field(flussSource, "selectRowCount", Boolean.class); + long limit = field(flussSource, "limit", Long.class); + Predicate partitionFilters = field(flussSource, "partitionFilters", Predicate.class); + @SuppressWarnings("unchecked") + LakeSource lakeSource = + (LakeSource) fieldObject(flussSource, "lakeSource"); + + if (primaryKeyIndexes.length > 0) { + // fluss-rs RecordBatchLogScanner is log-table append-only only. + return Planned.fallback("primary-key tables read a changelog the native log scanner" + + " does not carry"); + } + if (isDataLakeEnabled || lakeSource != null) { + return Planned.fallback("datalake-enabled tables read through the lake source"); + } + if (singleRowFilter != null) { + return Planned.fallback("a pushed-down single-row filter is not supported natively"); + } + if (modificationScanType != null) { + return Planned.fallback("a modification scan type is not supported natively"); + } + if (selectRowCount) { + return Planned.fallback("a row-count scan is not supported natively"); + } + if (limit > 0) { + return Planned.fallback("a pushed-down LIMIT is not supported natively"); + } + if (partitionFilters != null) { + return Planned.fallback("pushed-down partition filters are not supported natively"); + } + if (projectedFields != null && projectedFields.length == 0) { + return Planned.fallback("an empty projection is not supported natively"); + } + String columnReason = unsupportedColumnReason(scan, projectedFields); + if (columnReason != null) { + return Planned.fallback(columnReason); + } + + FlussConfigTranslator.Result translated = FlussConfigTranslator.translate(flussConfig.toMap()); + if (!translated.isTranslated()) { + return Planned.fallback( + translated.fallbackReason().orElse("client config is not translatable to fluss-rs")); + } + String[] keys = translated.config().keySet().toArray(new String[0]); + String[] values = new String[keys.length]; + for (int i = 0; i < keys.length; i++) { + values[i] = translated.config().get(keys[i]); + } + + return Planned.of( + new SourcePlan( + flussConfig, + tablePath, + false, + partitionKeyIndexes.length > 0, + offsetsInitializer(startupOptions), + scanPartitionDiscoveryIntervalMs, + streaming, + null, + null, + leaseContext == null ? LeaseContext.DEFAULT : leaseContext, + keys, + values, + projectedFields == null ? new int[0] : projectedFields)); + } catch (ReflectiveOperationException | RuntimeException e) { + return Planned.fallback("Fluss table source introspection failed: " + e); + } + } + + /** + * The first projected column whose type is outside the verified fluss-rs ↔ vendored-Arrow surface + * (as a fallback reason), or null when every read column is safe. + * + *

The whitelist is the intersection of fluss-rs' {@code to_arrow_type} export and what the + * vendored {@code ArrowConversion} readers accept: the fixed-width scalars, CHAR/VARCHAR (Utf8 + * both sides), DECIMAL (Decimal128 both sides), DATE (Date32), TIME (the same per-precision + * Time32/Time64 unit on both sides), plain TIMESTAMP (timezone-free Arrow timestamp), VARBINARY + * (Binary), and nested ROWs whose leaves are themselves supported. TIMESTAMP_LTZ is excluded: + * fluss-rs exports it as a UTC-zoned timestamp vector, while {@code ArrowConversion} currently + * rejects zoned timestamp vectors. BINARY is excluded because {@code ArrowConversion.toArrowSchema} + * has no BINARY mapping, and ARRAY/MAP are left gated until they have Fluss parity coverage. + */ + private static String unsupportedColumnReason(StreamPhysicalTableSourceScan scan, int[] projectedFields) { + RowType rowType = FilesystemTables.physicalRowType(scan); + if (rowType == null) { + return "the table's physical row type could not be resolved"; + } + int[] fields = projectedFields == null ? null : projectedFields; + if (fields == null) { + fields = new int[rowType.getFieldCount()]; + for (int i = 0; i < fields.length; i++) { + fields[i] = i; + } + } + for (int field : fields) { + if (field < 0 || field >= rowType.getFieldCount()) { + return "projected Fluss field index " + field + " is outside the physical row type"; + } + LogicalType type = rowType.getTypeAt(field); + String reason = unsupportedTypeReason(type); + if (reason != null) { + return reason; + } + } + return null; + } + + private static String unsupportedTypeReason(LogicalType type) { + switch (type.getTypeRoot()) { + case BOOLEAN: + case TINYINT: + case SMALLINT: + case INTEGER: + case BIGINT: + case FLOAT: + case DOUBLE: + case CHAR: + case VARCHAR: + case DECIMAL: + case DATE: + case TIME_WITHOUT_TIME_ZONE: + case TIMESTAMP_WITHOUT_TIME_ZONE: + case VARBINARY: + return null; + case ROW: + for (LogicalType child : type.getChildren()) { + String reason = unsupportedTypeReason(child); + if (reason != null) { + return reason; + } + } + return null; + default: + return "column type " + + type.asSummaryString() + + " is not verified across the fluss-rs Arrow boundary"; + } + } + + private static DynamicTableSource tableSource(StreamPhysicalTableSourceScan scan) { + try { + TableSourceTable table = scan.getTable().unwrap(TableSourceTable.class); + return table == null ? null : table.tableSource(); + } catch (RuntimeException e) { + return null; + } + } + + private static OffsetsInitializer offsetsInitializer(StartupOptions options) { + if (options == null || options.startupMode == null) { + return OffsetsInitializer.full(); + } + ScanStartupMode mode = options.startupMode; + if (mode == ScanStartupMode.EARLIEST) { + return OffsetsInitializer.earliest(); + } + if (mode == ScanStartupMode.LATEST) { + return OffsetsInitializer.latest(); + } + if (mode == ScanStartupMode.TIMESTAMP) { + return OffsetsInitializer.timestamp(options.startupTimestampMs); + } + return OffsetsInitializer.full(); + } + + private static Object fieldObject(Object target, String name) throws ReflectiveOperationException { + Field field = target.getClass().getDeclaredField(name); + field.setAccessible(true); + return field.get(target); + } + + @SuppressWarnings("unchecked") + private static T field(Object target, String name, Class type) + throws ReflectiveOperationException { + Object value = fieldObject(target, name); + return (T) value; + } + + private static final class Planned { + private final SourcePlan plan; + private final String fallbackReason; + + private Planned(SourcePlan plan, String fallbackReason) { + this.plan = plan; + this.fallbackReason = fallbackReason; + } + + static Planned of(SourcePlan plan) { + return new Planned(plan, null); + } + + static Planned fallback(String reason) { + return new Planned(null, reason); + } + } + + private static final class SourcePlan { + private final org.apache.fluss.config.Configuration flussConfig; + private final TablePath tablePath; + private final boolean hasPrimaryKey; + private final boolean partitioned; + private final OffsetsInitializer offsetsInitializer; + private final long scanPartitionDiscoveryIntervalMs; + private final boolean streaming; + private final Predicate partitionFilters; + private final LakeSource lakeSource; + private final LeaseContext leaseContext; + private final String[] nativeConfigKeys; + private final String[] nativeConfigValues; + private final int[] projectedFields; + + private SourcePlan( + org.apache.fluss.config.Configuration flussConfig, + TablePath tablePath, + boolean hasPrimaryKey, + boolean partitioned, + OffsetsInitializer offsetsInitializer, + long scanPartitionDiscoveryIntervalMs, + boolean streaming, + Predicate partitionFilters, + LakeSource lakeSource, + LeaseContext leaseContext, + String[] nativeConfigKeys, + String[] nativeConfigValues, + int[] projectedFields) { + this.flussConfig = flussConfig; + this.tablePath = tablePath; + this.hasPrimaryKey = hasPrimaryKey; + this.partitioned = partitioned; + this.offsetsInitializer = offsetsInitializer; + this.scanPartitionDiscoveryIntervalMs = scanPartitionDiscoveryIntervalMs; + this.streaming = streaming; + this.partitionFilters = partitionFilters; + this.lakeSource = lakeSource; + this.leaseContext = leaseContext; + this.nativeConfigKeys = nativeConfigKeys; + this.nativeConfigValues = nativeConfigValues; + this.projectedFields = projectedFields; + } + } +} diff --git a/src/main/java/io/github/jordepic/streamfusion/planner/NativeFlussSourceExecNode.java b/src/main/java/io/github/jordepic/streamfusion/planner/NativeFlussSourceExecNode.java new file mode 100644 index 00000000..f08fe33d --- /dev/null +++ b/src/main/java/io/github/jordepic/streamfusion/planner/NativeFlussSourceExecNode.java @@ -0,0 +1,54 @@ +package io.github.jordepic.streamfusion.planner; + +import io.github.jordepic.streamfusion.fluss.NativeFlussSource; +import io.github.jordepic.streamfusion.operator.ArrowBatch; +import io.github.jordepic.streamfusion.operator.ArrowBatchTypeInformation; +import java.util.Collections; +import org.apache.flink.api.common.eventtime.WatermarkStrategy; +import org.apache.flink.api.dag.Transformation; +import org.apache.flink.configuration.ReadableConfig; +import org.apache.flink.streaming.api.datastream.DataStreamSource; +import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment; +import org.apache.flink.table.planner.delegation.PlannerBase; +import org.apache.flink.table.planner.plan.nodes.exec.ExecNodeBase; +import org.apache.flink.table.planner.plan.nodes.exec.ExecNodeConfig; +import org.apache.flink.table.planner.plan.nodes.exec.ExecNodeContext; +import org.apache.flink.table.planner.plan.nodes.exec.stream.StreamExecNode; +import org.apache.flink.table.types.logical.RowType; + +/** Exec node that contributes the native fluss-rs log source transformation. */ +public class NativeFlussSourceExecNode extends ExecNodeBase + implements StreamExecNode { + + private static final String TRANSFORMATION = "native-fluss-source"; + + private final NativeFlussSource source; + + public NativeFlussSourceExecNode( + ReadableConfig tableConfig, + RowType outputType, + String description, + NativeFlussSource source) { + super( + ExecNodeContext.newNodeId(), + new ExecNodeContext("stream-exec-native-fluss-source_1"), + tableConfig, + Collections.emptyList(), + outputType, + description); + this.source = source; + } + + @Override + protected Transformation translateToPlanInternal( + PlannerBase planner, ExecNodeConfig config) { + StreamExecutionEnvironment env = planner.getExecEnv(); + DataStreamSource stream = + env.fromSource( + source, + WatermarkStrategy.noWatermarks(), + TRANSFORMATION, + ArrowBatchTypeInformation.INSTANCE); + return stream.getTransformation(); + } +} diff --git a/src/main/java/io/github/jordepic/streamfusion/planner/PhysicalPlanScan.java b/src/main/java/io/github/jordepic/streamfusion/planner/PhysicalPlanScan.java index f8ecefd6..0a49b50f 100644 --- a/src/main/java/io/github/jordepic/streamfusion/planner/PhysicalPlanScan.java +++ b/src/main/java/io/github/jordepic/streamfusion/planner/PhysicalPlanScan.java @@ -2,10 +2,12 @@ import java.util.ArrayList; import java.util.List; +import java.util.Map; import org.apache.calcite.rel.RelNode; import org.apache.calcite.rel.core.Calc; import org.apache.calcite.rel.core.Sort; import org.apache.calcite.rel.type.RelDataType; +import org.apache.flink.table.connector.source.DynamicTableSource; import org.apache.flink.table.planner.plan.nodes.physical.stream.StreamPhysicalCalc; import org.apache.flink.table.planner.plan.nodes.physical.stream.StreamPhysicalChangelogNormalize; import org.apache.flink.table.planner.plan.nodes.physical.stream.StreamPhysicalCorrelate; @@ -43,6 +45,7 @@ import org.apache.flink.table.planner.plan.optimize.program.FlinkOptimizeProgram; import org.apache.flink.table.planner.plan.utils.ChangelogPlanUtils; import org.apache.flink.table.planner.plan.optimize.program.StreamOptimizeContext; +import org.apache.flink.table.planner.plan.schema.TableSourceTable; /** * Optimizer program appended after the host engine's physical optimization. It rewrites the @@ -673,6 +676,22 @@ private RelNode rewrite(RelNode node) { scan.getCluster(), scan.getTraitSet(), scan.getRowType(), OrcSourceMatcher.path(scan)); } + if (current instanceof StreamPhysicalTableSourceScan) { + StreamPhysicalTableSourceScan scan = (StreamPhysicalTableSourceScan) current; + Map options = FilesystemTables.options(scan); + boolean flussConnectorOption = options != null && "fluss".equals(options.get("connector")); + if ((flussConnectorOption || isFlussTableSource(scan)) + && NativeConfig.operatorEnabled("flussSource")) { + String flussFallback = FlussTables.fallbackReason(scan); + if (flussFallback == null) { + substitutions++; + return new StreamPhysicalNativeFlussSource( + scan.getCluster(), scan.getTraitSet(), scan.getRowType(), scan); + } + recordFallback("fluss source: " + flussFallback); + } + } + // The fully-native rdkafka source (Rust owns the consume) is the default Kafka path for the // formats it decodes (JSON/bare-Avro/protobuf): the consume fast path made it decisively faster // than the decode path (divergences/19), and it is the only path that regenerates a pushed-down @@ -1350,6 +1369,20 @@ private RelNode kafkaDecode(RelNode current) { scan.getCluster(), scan.getTraitSet(), scan.getRowType(), FilesystemTables.options(scan)); } + private static boolean isFlussTableSource(StreamPhysicalTableSourceScan scan) { + try { + TableSourceTable table = scan.getTable().unwrap(TableSourceTable.class); + if (table == null) { + return false; + } + DynamicTableSource source = table.tableSource(); + return source != null + && "org.apache.fluss.flink.source.FlinkTableSource".equals(source.getClass().getName()); + } catch (LinkageError | RuntimeException e) { + return false; + } + } + /** * Replaces a keyed host exchange with a native columnar one (splitting the batch by the key), so the * shuffle is always part of the columnar island. When the exchange's producer is rowwise diff --git a/src/main/java/io/github/jordepic/streamfusion/planner/StreamPhysicalNativeFlussSource.java b/src/main/java/io/github/jordepic/streamfusion/planner/StreamPhysicalNativeFlussSource.java new file mode 100644 index 00000000..e2935a87 --- /dev/null +++ b/src/main/java/io/github/jordepic/streamfusion/planner/StreamPhysicalNativeFlussSource.java @@ -0,0 +1,61 @@ +package io.github.jordepic.streamfusion.planner; + +import java.util.List; +import org.apache.calcite.plan.RelOptCluster; +import org.apache.calcite.plan.RelTraitSet; +import org.apache.calcite.rel.AbstractRelNode; +import org.apache.calcite.rel.RelNode; +import org.apache.calcite.rel.RelWriter; +import org.apache.calcite.rel.type.RelDataType; +import org.apache.flink.table.planner.calcite.FlinkTypeFactory$; +import org.apache.flink.table.planner.plan.nodes.exec.ExecNode; +import org.apache.flink.table.planner.plan.nodes.physical.stream.StreamPhysicalRel; +import org.apache.flink.table.planner.plan.nodes.physical.stream.StreamPhysicalTableSourceScan; +import org.apache.flink.table.planner.utils.ShortcutUtils; + +/** Leaf physical node for a native Fluss log-table source that emits Arrow batches directly. */ +public class StreamPhysicalNativeFlussSource extends AbstractRelNode + implements StreamPhysicalRel, ColumnarOutput { + + private final StreamPhysicalTableSourceScan scan; + private final RelDataType outputRowType; + + public StreamPhysicalNativeFlussSource( + RelOptCluster cluster, + RelTraitSet traitSet, + RelDataType outputRowType, + StreamPhysicalTableSourceScan scan) { + super(cluster, traitSet); + this.outputRowType = outputRowType; + this.scan = scan; + } + + @Override + public boolean requireWatermark() { + return false; + } + + @Override + protected RelDataType deriveRowType() { + return outputRowType; + } + + @Override + public RelNode copy(RelTraitSet traitSet, List inputs) { + return new StreamPhysicalNativeFlussSource(getCluster(), traitSet, outputRowType, scan); + } + + @Override + public RelWriter explainTerms(RelWriter writer) { + return super.explainTerms(writer).item("connector", "fluss"); + } + + @Override + public ExecNode translateToExecNode() { + return new NativeFlussSourceExecNode( + ShortcutUtils.unwrapTableConfig(this), + FlinkTypeFactory$.MODULE$.toLogicalRowType(outputRowType), + getRelDetailedDescription(), + FlussTables.build(scan)); + } +} diff --git a/src/test/java/io/github/jordepic/streamfusion/NexmarkMatrixBenchmark.java b/src/test/java/io/github/jordepic/streamfusion/NexmarkMatrixBenchmark.java index 38b7ce20..f7ae5c33 100644 --- a/src/test/java/io/github/jordepic/streamfusion/NexmarkMatrixBenchmark.java +++ b/src/test/java/io/github/jordepic/streamfusion/NexmarkMatrixBenchmark.java @@ -10,9 +10,19 @@ import java.util.List; import java.util.Map; import java.util.Set; +import java.util.concurrent.CountDownLatch; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.TimeoutException; +import java.util.concurrent.atomic.AtomicLong; +import org.apache.flink.core.execution.JobClient; import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment; +import org.apache.flink.streaming.api.functions.sink.legacy.RichSinkFunction; +import org.apache.flink.table.api.Table; import org.apache.flink.table.api.TableEnvironment; import org.apache.flink.table.api.bridge.java.StreamTableEnvironment; +import org.apache.flink.types.Row; +import org.apache.flink.util.CloseableIterator; +import org.apache.fluss.server.testutils.FlussClusterExtension; import org.junit.jupiter.api.Test; import org.junit.jupiter.api.condition.EnabledIfEnvironmentVariable; import org.testcontainers.containers.KafkaContainer; @@ -32,6 +42,15 @@ * Its rowtime is a plain {@code TIMESTAMP(3)} (unlike the Kafka {@code TIMESTAMP_LTZ}), so the {@code * DATE_FORMAT}/{@code HOUR} queries that are generator-only on Kafka run here too. * + *

The optional Fluss rung ({@code SF_MATRIX_FLUSS=true}) preloads the same wide event row into a + * local Fluss test cluster, then reads it back through both Fluss's stock Flink connector and the + * native fluss-rs log-table source. Both engines run the identical SQL (no limit) in the same + * default streaming environment the other rungs use; because the Fluss log table is unbounded, each + * run counts changelog rows in the sink and cancels the job once the Nth row arrives, reporting + * time-to-Nth-row. N is the query's full output cardinality over the preloaded events, measured once + * per query with stock Flink on the bounded generator (the same rows the preload wrote) through the + * same plain-TIMESTAMP views, so both engines are cancelled at the same row. + * *

The query set is every query StreamFusion accelerates: q0–q5, q7–q23 (q1's and q14's decimal are * exact and native by default; q21's REGEXP_EXTRACT/LOWER and q14's HOUR route through the host * implementation via the columnar JVM upcall; q13 is a synchronous lookup join against a bounded @@ -49,7 +68,8 @@ * 500,000), {@code SF_MATRIX_QUERIES} a comma-separated query subset (e.g. {@code q0,q7,q15}), {@code * SF_LADDER_FORMATS} the Kafka formats (default {@code json,avro,protobuf}), {@code SF_MATRIX_GENERATOR} * ({@code false} to skip the generator column), {@code SF_MATRIX_PARQUET} ({@code false} to skip the - * Parquet column), {@code SF_MATRIX_KAFKA} ({@code false} to skip Kafka). + * Parquet column), {@code SF_MATRIX_KAFKA} ({@code false} to skip Kafka), {@code SF_MATRIX_FLUSS} + * ({@code true} to include the first stock Flink-on-Fluss baseline). */ @EnabledIfEnvironmentVariable(named = "SF_BENCHMARK", matches = "true") class NexmarkMatrixBenchmark { @@ -431,6 +451,16 @@ public long eval(String s, String c) { + " bid ROW," + " `dateTime` TIMESTAMP(3)"; + private static final String FLUSS_CATALOG = "fluss_catalog"; + private static final String FLUSS_TABLE = FLUSS_CATALOG + ".fluss.nexmark_events"; + // How long a Fluss run may take to reach its Nth sink row before the benchmark fails loudly (the + // unbounded source never finishes on its own, so a wrong target must not hang the matrix forever). + private static final long FLUSS_NTH_ROW_TIMEOUT_SECONDS = 600L; + // Latch/counter for the Fluss rung's count-N-then-cancel sink. Static volatile because Flink + // serializes the sink function into the task, so instance fields cannot signal the driver — the + // same pattern as NativeFlussSourceSqlHarnessTest's CollectingSink. + private static volatile CountDownLatch flussTargetReached; + private static volatile AtomicLong flussRowsSeen; @Test void matrix() throws Exception { @@ -438,9 +468,15 @@ void matrix() throws Exception { boolean runGenerator = !"false".equals(System.getenv("SF_MATRIX_GENERATOR")); boolean runParquet = !"false".equals(System.getenv("SF_MATRIX_PARQUET")); boolean runKafka = !"false".equals(System.getenv("SF_MATRIX_KAFKA")); + boolean runFluss = "true".equals(System.getenv("SF_MATRIX_FLUSS")); String formatsEnv = System.getenv("SF_LADDER_FORMATS"); String[] formats = formatsEnv != null ? formatsEnv.split(",") : new String[] {"json", "avro", "protobuf"}; + if (runFluss && !Native.flussFeatureBuilt()) { + throw new IllegalArgumentException( + "SF_MATRIX_FLUSS=true now reports native Fluss vs stock Flink-on-Fluss, so the native " + + "library must be built with the fluss cargo feature."); + } // result[label] -> ordered cells (rendered at the end as one table). Map> report = new LinkedHashMap<>(); @@ -476,6 +512,42 @@ void matrix() throws Exception { } } + if (runFluss) { + FlussClusterExtension cluster = FlussClusterExtension.builder().setNumOfTabletServers(1).build(); + cluster.start(); + try { + String bootstrapServers = cluster.getBootstrapServers(); + writeFlussSource(bootstrapServers); + for (Query q : queries) { + if (!flussBaselineSupported(q)) { + report + .get(q.label) + .add( + skipCell( + "fluss", + "plain-timestamp baseline does not cover time-attribute/proctime/lookup")); + continue; + } + long targetRows = flussTargetRows(q); + if (targetRows == 0) { + report + .get(q.label) + .add(skipCell("fluss", "query emits no rows over the preloaded events")); + continue; + } + double flink = flussBest(bootstrapServers, q, false, null, targetRows); + double nativeRun = flussBest(bootstrapServers, q, true, null, targetRows); + report.get(q.label).add(cell("fluss", flink, nativeRun)); + if (q.nativeVariantProps != null) { + double variant = flussBest(bootstrapServers, q, true, q.nativeVariantProps, targetRows); + report.get(q.label).add(variantCell("fluss", q.nativeVariantLabel, flink, variant)); + } + } + } finally { + cluster.close(); + } + } + if (runKafka) { for (String format : formats) { try (KafkaContainer kafka = @@ -633,6 +705,10 @@ private static String cell(String source, double flink, double nativeRun) { source, flink, ROWS / flink, nativeRun, ROWS / nativeRun, flink / nativeRun); } + private static String skipCell(String source, String reason) { + return String.format("%-10s skipped (%s)", source, reason); + } + private static String variantCell(String source, String label, double flink, double variant) { return String.format( "%-10s [%s] Native %6.3fs (%,.0f ev/s) %.2fx", @@ -662,6 +738,183 @@ private static double runGeneratorOnce(Query q, boolean nativeRun) throws Except return execute(tEnv, scan, q, nativeRun, "TIMESTAMP(3)"); } + // ----- Fluss source ----- + + /** Writes the wide event row to a local Fluss log table once; every Fluss query reads it back. */ + private static void writeFlussSource(String bootstrapServers) throws Exception { + TableEnvironment tEnv = NexmarkBenchmark.environment(ROWS); + createFlussCatalog(tEnv, bootstrapServers); + tEnv.executeSql("DROP TABLE IF EXISTS " + FLUSS_TABLE); + tEnv.executeSql( + "CREATE TABLE " + + FLUSS_TABLE + + " (" + + PARQUET_SCHEMA + + ") WITH ('bucket.num' = '1')"); + tEnv.executeSql( + "INSERT INTO " + + FLUSS_TABLE + + " SELECT event_type, person, auction, bid, `dateTime` FROM events") + .await(); + } + + private static double flussBest( + String bootstrapServers, Query q, boolean nativeRun, Map extra, long targetRows) + throws Exception { + double best = Double.MAX_VALUE; + for (int run = 0; run < WARMUP + RUNS; run++) { + double seconds = + withProps( + q, nativeRun, extra, () -> runFlussOnce(bootstrapServers, nativeRun, q, targetRows)); + if (run >= WARMUP) { + best = Math.min(best, seconds); + } + } + return best; + } + + /** + * One Fluss run: both engines get the identical SQL over the unbounded log table in the same + * default streaming environment the other rungs use. The log table never reaches end-of-input, so + * instead of awaiting the insert the query streams into a {@link CountingSink} and the job is + * cancelled once the {@code targetRows}th changelog row arrives — the elapsed time-to-Nth-row is + * the measurement (submission and startup included, like the other rungs' await). + */ + private static double runFlussOnce( + String bootstrapServers, boolean nativeRun, Query q, long targetRows) throws Exception { + StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); + env.setParallelism(1); + env.getConfig().enableObjectReuse(); + StreamTableEnvironment tEnv = StreamTableEnvironment.create(env); + createFlussCatalog(tEnv, bootstrapServers); + tEnv.executeSql("CREATE TEMPORARY VIEW src AS SELECT * FROM " + FLUSS_TABLE); + createEventViews(tEnv); + tEnv.createTemporarySystemFunction("count_char", CountChar.class); + runSetup(tEnv, q); + PhysicalPlanScan scan = nativeRun ? NativePlanner.install(tEnv) : null; + // The query's SELECT without the blackhole INSERT wrapper, routed to the counting sink instead + // (toChangelogStream, so the updating queries' retractions count like any other sink row). + Table table = tEnv.sqlQuery(q.insertSql.substring("INSERT INTO sink ".length())); + flussRowsSeen = new AtomicLong(); + flussTargetReached = new CountDownLatch(1); + tEnv.toChangelogStream(table).addSink(new CountingSink(targetRows)).name("count-fluss-bench"); + long start = System.nanoTime(); + JobClient job = env.executeAsync("fluss-nexmark-" + q.label); + try { + if (nativeRun && scan.substitutions() == 0) { + throw new IllegalStateException( + q.label + ": native island did not engage; comparison is moot. " + scan.fallbackReasons()); + } + if (!flussTargetReached.await(FLUSS_NTH_ROW_TIMEOUT_SECONDS, TimeUnit.SECONDS)) { + throw new TimeoutException( + q.label + + ": Fluss run saw " + + flussRowsSeen.get() + + " of " + + targetRows + + " sink rows within " + + FLUSS_NTH_ROW_TIMEOUT_SECONDS + + "s"); + } + return (System.nanoTime() - start) / 1e9; + } finally { + try { + job.cancel().get(); + } catch (Exception ignored) { + // The job may already be terminating (e.g. it failed before the Nth row); the exception + // propagating out of the try block is the interesting one, so cancellation noise stays here. + } + flussRowsSeen = null; + flussTargetReached = null; + } + } + + /** + * The number of changelog rows {@code q} emits over the preloaded events — measured once per query + * with stock Flink on the bounded generator (the same rows the Fluss preload wrote) through the + * same plain-TIMESTAMP views the Fluss runs use, so both engines are cancelled at the same Nth + * sink row. + */ + private static long flussTargetRows(Query q) throws Exception { + TableEnvironment tEnv = NexmarkBenchmark.environment(ROWS); + // Rebuild the person/auction/bid views over a materialized (non-time-attribute) dateTime so the + // calibration plan has the same shape as the Fluss runs', whose log table has no watermark. + tEnv.dropTemporaryView("person"); + tEnv.dropTemporaryView("auction"); + tEnv.dropTemporaryView("bid"); + tEnv.executeSql( + "CREATE TEMPORARY VIEW src AS SELECT event_type, person, auction, bid," + + " CAST(`dateTime` AS TIMESTAMP(3)) AS `dateTime` FROM events"); + createEventViews(tEnv); + tEnv.createTemporarySystemFunction("count_char", CountChar.class); + runSetup(tEnv, q); + long rows = 0; + try (CloseableIterator it = + tEnv.executeSql(q.insertSql.substring("INSERT INTO sink ".length())).collect()) { + while (it.hasNext()) { + it.next(); + rows++; + } + } + return rows; + } + + /** The person/auction/bid logical streams over a wide-event {@code src} with a plain TIMESTAMP. */ + private static void createEventViews(TableEnvironment tEnv) { + tEnv.executeSql( + "CREATE TEMPORARY VIEW person AS SELECT person.id AS id, person.name AS name," + + " person.emailAddress AS emailAddress, person.creditCard AS creditCard, person.city AS" + + " city, person.state AS state, `dateTime`, person.extra AS extra FROM src WHERE" + + " event_type = 0"); + tEnv.executeSql( + "CREATE TEMPORARY VIEW auction AS SELECT auction.id AS id, auction.itemName AS itemName," + + " auction.description AS description, auction.initialBid AS initialBid, auction.reserve" + + " AS reserve, `dateTime`, auction.expires AS expires, auction.seller AS seller," + + " auction.category AS category, auction.extra AS extra FROM src WHERE event_type = 1"); + tEnv.executeSql( + "CREATE TEMPORARY VIEW bid AS SELECT bid.auction AS auction, bid.bidder AS bidder, bid.price" + + " AS price, bid.channel AS channel, bid.url AS url, `dateTime`, bid.extra AS extra FROM" + + " src WHERE event_type = 2"); + } + + private static void createFlussCatalog(TableEnvironment tEnv, String bootstrapServers) { + tEnv.executeSql( + "CREATE CATALOG " + + FLUSS_CATALOG + + " WITH ('type' = 'fluss', 'bootstrap.servers' = '" + + bootstrapServers + + "')"); + } + + private static boolean flussBaselineSupported(Query q) { + return !Set.of("q5", "q7", "q8", "q11", "q12", "q13").contains(q.label); + } + + /** + * Counts changelog rows and releases the latch at the Nth — the count-N-then-cancel sink the Fluss + * rung times against (the latch pattern of NativeFlussSourceSqlHarnessTest's CollectingSink, + * replicated locally so the benchmark stays self-contained). + */ + private static final class CountingSink extends RichSinkFunction { + private final long targetRows; + + private CountingSink(long targetRows) { + this.targetRows = targetRows; + } + + @Override + public void invoke(Row value, Context context) { + AtomicLong seen = flussRowsSeen; + CountDownLatch latch = flussTargetReached; + if (seen == null || latch == null) { + return; + } + if (seen.incrementAndGet() >= targetRows) { + latch.countDown(); + } + } + } + // ----- parquet file source ----- /** Writes the wide event row to a fresh local Parquet directory once; every query reads it back. */ diff --git a/src/test/java/io/github/jordepic/streamfusion/fluss/FlussConfigTranslatorTest.java b/src/test/java/io/github/jordepic/streamfusion/fluss/FlussConfigTranslatorTest.java new file mode 100644 index 00000000..8903f018 --- /dev/null +++ b/src/test/java/io/github/jordepic/streamfusion/fluss/FlussConfigTranslatorTest.java @@ -0,0 +1,141 @@ +package io.github.jordepic.streamfusion.fluss; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertTrue; + +import java.util.Map; +import org.junit.jupiter.api.Test; + +class FlussConfigTranslatorTest { + + private static Map translated(Map options) { + FlussConfigTranslator.Result result = FlussConfigTranslator.translate(options); + assertTrue( + result.isTranslated(), () -> "expected translation, got " + result.fallbackReason()); + return result.config(); + } + + private static String fallback(Map options) { + FlussConfigTranslator.Result result = FlussConfigTranslator.translate(options); + assertFalse(result.isTranslated(), "expected fallback, got translation"); + return result.fallbackReason().orElseThrow(); + } + + @Test + void mapsScannerClientKnobsToRustConfigFields() { + Map config = + translated( + Map.of( + "bootstrap.servers", "localhost:9123", + "client.scanner.log.max-poll-records", "2048", + "client.scanner.log.fetch.max-bytes", "16 mb", + "client.scanner.log.fetch.max-bytes-for-bucket", "1 mb", + "client.scanner.log.fetch.min-bytes", "1 bytes", + "client.scanner.log.fetch.wait-max-time", "PT0.5S", + "client.scanner.remote-log.prefetch-num", "8", + "client.remote-file.download-thread-num", "4")); + + assertEquals("localhost:9123", config.get("bootstrap_servers")); + assertEquals("2048", config.get("scanner_log_max_poll_records")); + assertEquals("16777216", config.get("scanner_log_fetch_max_bytes")); + assertEquals("1048576", config.get("scanner_log_fetch_max_bytes_for_bucket")); + assertEquals("1", config.get("scanner_log_fetch_min_bytes")); + assertEquals("500", config.get("scanner_log_fetch_wait_max_time_ms")); + assertEquals("8", config.get("scanner_remote_log_prefetch_num")); + assertEquals("4", config.get("remote_file_download_thread_num")); + } + + @Test + void usesFlinkParsersForMemoryAndDurationUnits() { + Map config = + translated( + Map.of( + "client.scanner.log.fetch.max-bytes", "1 tb", + "client.connect-timeout", "1 d")); + + assertEquals("1099511627776", config.get("scanner_log_fetch_max_bytes")); + assertEquals("86400000", config.get("connect_timeout_ms")); + } + + @Test + void ignoresWriterAndLookupOptionsWithoutTranslatingThem() { + FlussConfigTranslator.Result result = + FlussConfigTranslator.translate( + Map.of( + "bootstrap.servers", "localhost:9123", + "client.writer.acks", "all", + "client.writer.batch-size", "2 mb", + "client.writer.bucket.no-key-assigner", "round_robin", + "client.lookup.max-batch-size", "256", + "client.lookup.batch-timeout", "50 ms")); + + assertTrue(result.isTranslated(), () -> "expected translation, got " + result.fallbackReason()); + assertEquals(Map.of("bootstrap_servers", "localhost:9123"), result.config()); + } + + @Test + void fallsBackOnUnrecognizedClientOptions() { + String reason = + fallback( + Map.of( + "bootstrap.servers", "localhost:9123", + "client.scanner.log.fetch.max-bytes", "16 mb", + "client.scanner.new-knob", "on")); + + assertTrue(reason.contains("client.scanner.new-knob"), reason); + } + + @Test + void keepsFlinkCoordinationOptionsOutOfNativeConfig() { + FlussConfigTranslator.Result result = + FlussConfigTranslator.translate( + Map.of( + "bootstrap.servers", "localhost:9123", + "scan.startup.mode", "earliest", + "scan.partition.discovery.interval", "1 min")); + + assertTrue(result.isTranslated()); + assertFalse(result.config().containsKey("scan.startup.mode")); + assertFalse(result.config().containsKey("scan.partition.discovery.interval")); + assertEquals("earliest", result.coordinationOptions().get("scan.startup.mode")); + assertEquals("1 min", result.coordinationOptions().get("scan.partition.discovery.interval")); + } + + @Test + void mapsBasicPlainSaslButRejectsUnsupportedMechanism() { + Map config = + translated( + Map.of( + "client.security.protocol", "sasl", + "client.security.sasl.mechanism", "PLAIN", + "client.security.sasl.username", "alice", + "client.security.sasl.password", "secret")); + + assertEquals("sasl", config.get("security_protocol")); + assertEquals("PLAIN", config.get("security_sasl_mechanism")); + assertEquals("alice", config.get("security_sasl_username")); + assertEquals("secret", config.get("security_sasl_password")); + assertTrue( + fallback( + Map.of( + "client.security.protocol", "sasl", + "client.security.sasl.mechanism", "SCRAM-SHA-256")) + .contains("PLAIN")); + } + + @Test + void rejectsSecurityProtocolsFlussRsDoesNotImplement() { + assertTrue(fallback(Map.of("client.security.protocol", "ssl")).contains("ssl")); + assertEquals( + "PLAINTEXT", + translated(Map.of("client.security.protocol", "PLAINTEXT")).get("security_protocol")); + } + + @Test + void fallsBackOnClientOptionsNotRepresentedInFlussRsConfig() { + assertTrue(fallback(Map.of("client.request-timeout", "30 s")).contains("request-timeout")); + assertTrue(fallback(Map.of("client.scanner.log.check-crc", "true")).contains("check-crc")); + assertTrue(fallback(Map.of("client.security.sasl.jaas.config", "x")).contains("jaas")); + } +} diff --git a/src/test/java/io/github/jordepic/streamfusion/fluss/FlussSplitTranslatorTest.java b/src/test/java/io/github/jordepic/streamfusion/fluss/FlussSplitTranslatorTest.java new file mode 100644 index 00000000..6d13fead --- /dev/null +++ b/src/test/java/io/github/jordepic/streamfusion/fluss/FlussSplitTranslatorTest.java @@ -0,0 +1,51 @@ +package io.github.jordepic.streamfusion.fluss; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertThrows; +import static org.junit.jupiter.api.Assertions.assertTrue; + +import org.apache.fluss.flink.source.split.HybridSnapshotLogSplit; +import org.apache.fluss.flink.source.split.LogSplit; +import org.apache.fluss.metadata.TableBucket; +import org.junit.jupiter.api.Test; + +class FlussSplitTranslatorTest { + + @Test + void extractsNativeAssignmentFromNonPartitionedLogSplit() { + LogSplit split = new LogSplit(new TableBucket(7L, 2), null, 11L, 42L); + + NativeFlussLogSplit nativeSplit = FlussSplitTranslator.translateLogSplit(split); + + assertEquals(split.splitId(), nativeSplit.splitId()); + assertEquals(7L, nativeSplit.tableId()); + assertFalse(nativeSplit.partitionId().isPresent()); + assertEquals(2, nativeSplit.bucket()); + assertEquals(11L, nativeSplit.startingOffset()); + assertEquals(42L, nativeSplit.stoppingOffset().orElseThrow()); + } + + @Test + void extractsNativeAssignmentFromPartitionedLogSplit() { + LogSplit split = new LogSplit(new TableBucket(7L, 99L, 3), "dt=2026-07-04", 5L); + + NativeFlussLogSplit nativeSplit = FlussSplitTranslator.translateLogSplit(split); + + assertEquals(7L, nativeSplit.tableId()); + assertEquals(99L, nativeSplit.partitionId().orElseThrow()); + assertEquals("dt=2026-07-04", nativeSplit.partitionName()); + assertEquals(3, nativeSplit.bucket()); + assertEquals(5L, nativeSplit.startingOffset()); + assertFalse(nativeSplit.stoppingOffset().isPresent()); + } + + @Test + void rejectsHybridSnapshotLogSplitForTheLogTableProofPath() { + HybridSnapshotLogSplit split = + new HybridSnapshotLogSplit(new TableBucket(7L, 2), null, 123L, 4L); + + assertFalse(FlussSplitTranslator.isNativeLogSplit(split)); + assertThrows(IllegalArgumentException.class, () -> FlussSplitTranslator.translateLogSplit(split)); + } +} diff --git a/src/test/java/io/github/jordepic/streamfusion/fluss/NativeFlussRecordEmitterTest.java b/src/test/java/io/github/jordepic/streamfusion/fluss/NativeFlussRecordEmitterTest.java new file mode 100644 index 00000000..2a29351c --- /dev/null +++ b/src/test/java/io/github/jordepic/streamfusion/fluss/NativeFlussRecordEmitterTest.java @@ -0,0 +1,63 @@ +package io.github.jordepic.streamfusion.fluss; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertSame; + +import io.github.jordepic.streamfusion.operator.ArrowBatch; +import java.util.List; +import org.apache.arrow.memory.BufferAllocator; +import org.apache.arrow.memory.RootAllocator; +import org.apache.arrow.vector.VectorSchemaRoot; +import org.apache.arrow.vector.types.pojo.Schema; +import org.apache.flink.api.common.eventtime.Watermark; +import org.apache.flink.api.connector.source.SourceOutput; +import org.apache.fluss.flink.source.split.LogSplit; +import org.apache.fluss.flink.source.split.LogSplitState; +import org.apache.fluss.metadata.TableBucket; +import org.junit.jupiter.api.Test; + +class NativeFlussRecordEmitterTest { + + @Test + void emitsArrowBatchAndAdvancesLogSplitOffset() throws Exception { + NativeFlussRecordEmitter emitter = new NativeFlussRecordEmitter(); + LogSplitState splitState = + new LogSplitState(new LogSplit(new TableBucket(7L, 2), null, 11L, 99L)); + CapturingOutput output = new CapturingOutput(); + + try (BufferAllocator allocator = new RootAllocator()) { + VectorSchemaRoot root = VectorSchemaRoot.create(new Schema(List.of()), allocator); + ArrowBatch batch = new ArrowBatch(root); + + emitter.emitRecord(new NativeFlussRecord(batch, 42L), output, splitState); + + assertSame(batch, output.record); + assertEquals(42L, splitState.toSourceSplit().getStartingOffset()); + output.record.root().close(); + } + } + + private static final class CapturingOutput implements SourceOutput { + + private ArrowBatch record; + + @Override + public void collect(ArrowBatch record) { + this.record = record; + } + + @Override + public void collect(ArrowBatch record, long timestamp) { + this.record = record; + } + + @Override + public void emitWatermark(Watermark watermark) {} + + @Override + public void markIdle() {} + + @Override + public void markActive() {} + } +} diff --git a/src/test/java/io/github/jordepic/streamfusion/fluss/NativeFlussSourceSqlHarnessTest.java b/src/test/java/io/github/jordepic/streamfusion/fluss/NativeFlussSourceSqlHarnessTest.java new file mode 100644 index 00000000..f8954aa3 --- /dev/null +++ b/src/test/java/io/github/jordepic/streamfusion/fluss/NativeFlussSourceSqlHarnessTest.java @@ -0,0 +1,642 @@ +package io.github.jordepic.streamfusion.fluss; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertTrue; + +import io.github.jordepic.streamfusion.Native; +import io.github.jordepic.streamfusion.planner.NativePlanner; +import io.github.jordepic.streamfusion.planner.PhysicalPlanScan; +import java.time.LocalDateTime; +import java.util.ArrayList; +import java.util.Collections; +import java.util.Comparator; +import java.util.HashMap; +import java.util.HashSet; +import java.util.List; +import java.util.Map; +import java.util.Set; +import java.util.concurrent.CountDownLatch; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.TimeoutException; +import org.apache.arrow.vector.VectorSchemaRoot; +import org.apache.flink.connector.base.source.reader.RecordsWithSplitIds; +import org.apache.flink.connector.base.source.reader.splitreader.SplitsAddition; +import org.apache.flink.connector.base.source.reader.splitreader.SplitsRemoval; +import org.apache.flink.core.execution.JobClient; +import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment; +import org.apache.flink.streaming.api.functions.sink.legacy.RichSinkFunction; +import org.apache.flink.table.api.Table; +import org.apache.flink.table.api.bridge.java.StreamTableEnvironment; +import org.apache.flink.types.Row; +import org.apache.fluss.client.Connection; +import org.apache.fluss.client.ConnectionFactory; +import org.apache.fluss.client.admin.Admin; +import org.apache.fluss.flink.source.split.LogSplit; +import org.apache.fluss.flink.source.split.SourceSplitBase; +import org.apache.fluss.metadata.PartitionInfo; +import org.apache.fluss.metadata.TableBucket; +import org.apache.fluss.metadata.TablePath; +import org.apache.fluss.server.testutils.FlussClusterExtension; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.condition.EnabledIf; + +/** + * End-to-end SQL test for the native Fluss log-table source. It writes rows through Flink's stock + * Fluss connector, reads them back through the native planner path, and asserts the optimizer + * substituted the native Fluss source so fallback cannot masquerade as coverage. + * + *

Run with a native library built using {@code -Dnative.cargo.args="build --features fluss"}. + */ +@EnabledIf("nativeFlussFeatureBuilt") +class NativeFlussSourceSqlHarnessTest { + + private static final String CATALOG = "fluss_it_catalog"; + private static final String DATABASE = "fluss"; + private static volatile CountDownLatch firstRowCollected; + private static volatile CountDownLatch rowsCollected; + private static volatile List> collectedRows; + + @Test + void nativeFlussSourceReadsLogTableThroughSql() throws Exception { + FlussClusterExtension cluster = FlussClusterExtension.builder().setNumOfTabletServers(1).build(); + String bootstrapServers = null; + String tablePath = null; + try { + cluster.start(); + bootstrapServers = cluster.getBootstrapServers(); + tablePath = CATALOG + "." + DATABASE + ".native_fluss_source_it_" + System.nanoTime(); + writeRows(bootstrapServers, tablePath); + + List> rows = + readRows(bootstrapServers, "SELECT id, name, score FROM " + tablePath, 3); + + assertEquals( + List.of(List.of(1L, "alice", 10), List.of(2L, "bob", 20), List.of(3L, "carol", 30)), + rows); + } finally { + if (bootstrapServers != null && tablePath != null) { + dropTable(bootstrapServers, tablePath); + } + cluster.close(); + } + } + + @Test + void nativeFlussSourceReadsProjectedColumnsThroughSql() throws Exception { + FlussClusterExtension cluster = FlussClusterExtension.builder().setNumOfTabletServers(1).build(); + String bootstrapServers = null; + String tablePath = null; + try { + cluster.start(); + bootstrapServers = cluster.getBootstrapServers(); + tablePath = CATALOG + "." + DATABASE + ".native_fluss_projection_it_" + System.nanoTime(); + writeRows(bootstrapServers, tablePath); + + List> rows = readRows(bootstrapServers, "SELECT name FROM " + tablePath, 3); + + assertEquals(List.of(List.of("alice"), List.of("bob"), List.of("carol")), rows); + } finally { + if (bootstrapServers != null && tablePath != null) { + dropTable(bootstrapServers, tablePath); + } + cluster.close(); + } + } + + @Test + void nativeFlussSourceGatesOnlyProjectedColumnsThroughSql() throws Exception { + FlussClusterExtension cluster = FlussClusterExtension.builder().setNumOfTabletServers(1).build(); + String bootstrapServers = null; + String tablePath = null; + try { + cluster.start(); + bootstrapServers = cluster.getBootstrapServers(); + tablePath = + CATALOG + "." + DATABASE + ".native_fluss_projected_gate_it_" + System.nanoTime(); + writeRowsWithUnusedUnsupportedColumn(bootstrapServers, tablePath); + + List> rows = readRows(bootstrapServers, "SELECT id FROM " + tablePath, 2); + + assertEquals(List.of(List.of(1L), List.of(2L)), rows); + } finally { + if (bootstrapServers != null && tablePath != null) { + dropTable(bootstrapServers, tablePath); + } + cluster.close(); + } + } + + @Test + void nativeFlussSourceReadsNestedRowFieldsThroughSql() throws Exception { + FlussClusterExtension cluster = FlussClusterExtension.builder().setNumOfTabletServers(1).build(); + String bootstrapServers = null; + String tablePath = null; + try { + cluster.start(); + bootstrapServers = cluster.getBootstrapServers(); + tablePath = CATALOG + "." + DATABASE + ".native_fluss_nested_it_" + System.nanoTime(); + writeNestedRows(bootstrapServers, tablePath); + + List> rows = + readRows( + bootstrapServers, + "SELECT bid.auction, bid.bidder, bid.price, bid.`dateTime`, bid.extra FROM " + + tablePath + + " WHERE event_type = 2", + 2); + + assertEquals( + List.of( + List.of( + 100L, + 7L, + 42L, + LocalDateTime.parse("2024-01-01T00:00:00.123"), + "bextra-1"), + List.of( + 101L, + 8L, + 43L, + LocalDateTime.parse("2024-01-01T00:00:00.456"), + "bextra-2")), + rows); + } finally { + if (bootstrapServers != null && tablePath != null) { + dropTable(bootstrapServers, tablePath); + } + cluster.close(); + } + } + + @Test + void nativeFlussSourceReadsStaticPartitionedLogTableThroughSql() throws Exception { + FlussClusterExtension cluster = FlussClusterExtension.builder().setNumOfTabletServers(1).build(); + String bootstrapServers = null; + String tablePath = null; + try { + cluster.start(); + bootstrapServers = cluster.getBootstrapServers(); + tablePath = CATALOG + "." + DATABASE + ".native_fluss_partitioned_it_" + System.nanoTime(); + writePartitionedRows(bootstrapServers, tablePath); + + List> rows = + readRows(bootstrapServers, "SELECT id, region, score FROM " + tablePath, 3); + + assertEquals( + List.of(List.of(1L, "US", 10), List.of(2L, "EU", 20), List.of(3L, "US", 30)), + rows); + } finally { + if (bootstrapServers != null && tablePath != null) { + dropTable(bootstrapServers, tablePath); + } + cluster.close(); + } + } + + @Test + void nativeFlussSourceDiscoversDynamicPartitionedLogTableThroughSql() throws Exception { + FlussClusterExtension cluster = FlussClusterExtension.builder().setNumOfTabletServers(1).build(); + String bootstrapServers = null; + String tablePath = null; + try { + cluster.start(); + bootstrapServers = cluster.getBootstrapServers(); + tablePath = + CATALOG + "." + DATABASE + ".native_fluss_dynamic_partitioned_it_" + System.nanoTime(); + createPartitionedTable(bootstrapServers, tablePath, "100 ms"); + addPartitionedRow(bootstrapServers, tablePath, "US", 1, 10); + + String finalBootstrapServers = bootstrapServers; + String finalTablePath = tablePath; + List> rows = + readRows( + bootstrapServers, + "SELECT id, region, score FROM " + tablePath, + 2, + () -> addPartitionedRow(finalBootstrapServers, finalTablePath, "EU", 2, 20)); + + assertEquals(List.of(List.of(1L, "US", 10), List.of(2L, "EU", 20)), rows); + } finally { + if (bootstrapServers != null && tablePath != null) { + dropTable(bootstrapServers, tablePath); + } + cluster.close(); + } + } + + @Test + void nativeFlussSourceAcknowledgesDroppedDynamicPartitionThroughSql() throws Exception { + FlussClusterExtension cluster = FlussClusterExtension.builder().setNumOfTabletServers(1).build(); + String bootstrapServers = null; + String tablePath = null; + try { + cluster.start(); + bootstrapServers = cluster.getBootstrapServers(); + tablePath = + CATALOG + "." + DATABASE + ".native_fluss_dropped_partition_it_" + System.nanoTime(); + createPartitionedTable(bootstrapServers, tablePath, "100 ms"); + addPartitionedRow(bootstrapServers, tablePath, "US", 1, 10); + + String finalBootstrapServers = bootstrapServers; + String finalTablePath = tablePath; + List> rows = + readRowsAfterFirstRow( + bootstrapServers, + "SELECT id, region, score FROM " + tablePath, + 2, + () -> { + dropPartition(finalBootstrapServers, finalTablePath, "US"); + addPartitionedRow(finalBootstrapServers, finalTablePath, "EU", 2, 20); + }); + + assertEquals(List.of(List.of(1L, "US", 10), List.of(2L, "EU", 20)), rows); + } finally { + if (bootstrapServers != null && tablePath != null) { + dropTable(bootstrapServers, tablePath); + } + cluster.close(); + } + } + + @Test + void nativeFlussSourceReadsMultiBucketLogTableThroughSql() throws Exception { + FlussClusterExtension cluster = FlussClusterExtension.builder().setNumOfTabletServers(1).build(); + String bootstrapServers = null; + String tablePath = null; + try { + cluster.start(); + bootstrapServers = cluster.getBootstrapServers(); + tablePath = CATALOG + "." + DATABASE + ".native_fluss_multi_bucket_it_" + System.nanoTime(); + writeMultiBucketRows(bootstrapServers, tablePath); + + List> rows = + readRows(bootstrapServers, "SELECT id, name, score FROM " + tablePath, 9); + + assertEquals( + List.of( + List.of(1L, "alice", 10), + List.of(2L, "bob", 20), + List.of(3L, "carol", 30), + List.of(4L, "dave", 40), + List.of(5L, "erin", 50), + List.of(6L, "frank", 60), + List.of(7L, "grace", 70), + List.of(8L, "heidi", 80), + List.of(9L, "ivan", 90)), + rows); + } finally { + if (bootstrapServers != null && tablePath != null) { + dropTable(bootstrapServers, tablePath); + } + cluster.close(); + } + } + + @Test + void nativeFlussSplitReaderReportsRemovedSplitsAsFinished() throws Exception { + FlussClusterExtension cluster = FlussClusterExtension.builder().setNumOfTabletServers(1).build(); + String bootstrapServers = null; + String tablePath = null; + try { + cluster.start(); + bootstrapServers = cluster.getBootstrapServers(); + String tableName = "native_fluss_split_removal_it_" + System.nanoTime(); + tablePath = CATALOG + "." + DATABASE + "." + tableName; + createPartitionedTable(bootstrapServers, tablePath, "100 ms"); + addPartitionedRow(bootstrapServers, tablePath, "US", 1, 10); + addPartitionedRow(bootstrapServers, tablePath, "EU", 2, 20); + + Map splitsByPartition = + logSplitsByPartition(cluster.getClientConfig(), tableName); + SourceSplitBase usSplit = splitsByPartition.get("US"); + SourceSplitBase euSplit = splitsByPartition.get("EU"); + + NativeFlussSplitReader reader = + new NativeFlussSplitReader( + new String[] {"bootstrap_servers"}, + new String[] {bootstrapServers}, + DATABASE, + tableName, + new int[0], + 100L); + try { + reader.handleSplitsChanges(new SplitsAddition<>(List.of(usSplit, euSplit))); + Set finishedSplitIds = new HashSet<>(); + long deadline = System.nanoTime() + TimeUnit.SECONDS.toNanos(30); + int rows = 0; + while (rows < 2) { + if (System.nanoTime() > deadline) { + throw new TimeoutException("timed out waiting for rows from both partition splits"); + } + rows += drainBatches(reader.fetch(), finishedSplitIds); + } + + dropPartition(bootstrapServers, tablePath, "US"); + reader.handleSplitsChanges(new SplitsRemoval<>(List.of(usSplit))); + while (!finishedSplitIds.contains(usSplit.splitId()) && System.nanoTime() < deadline) { + drainBatches(reader.fetch(), finishedSplitIds); + } + + assertTrue( + finishedSplitIds.contains(usSplit.splitId()), + "removed split was never reported finished; finished=" + finishedSplitIds); + assertFalse(finishedSplitIds.contains(euSplit.splitId())); + } finally { + reader.close(); + } + } finally { + if (bootstrapServers != null && tablePath != null) { + dropTable(bootstrapServers, tablePath); + } + cluster.close(); + } + } + + static boolean nativeFlussFeatureBuilt() { + return Native.flussFeatureBuilt(); + } + + private static Map logSplitsByPartition( + org.apache.fluss.config.Configuration clientConfig, String tableName) throws Exception { + TablePath flussTablePath = TablePath.of(DATABASE, tableName); + try (Connection connection = ConnectionFactory.createConnection(clientConfig)) { + Admin admin = connection.getAdmin(); + long tableId = admin.getTableInfo(flussTablePath).get().getTableId(); + Map splits = new HashMap<>(); + for (PartitionInfo partition : admin.listPartitionInfos(flussTablePath).get()) { + TableBucket bucket = new TableBucket(tableId, partition.getPartitionId(), 0); + splits.put( + partition.getPartitionName(), new LogSplit(bucket, partition.getPartitionName(), 0L)); + } + return splits; + } + } + + private static int drainBatches( + RecordsWithSplitIds records, Set finishedSplitIds) { + int rows = 0; + String splitId = records.nextSplit(); + while (splitId != null) { + NativeFlussRecord record = records.nextRecordFromSplit(); + while (record != null) { + try (VectorSchemaRoot root = record.batch().root()) { + rows += root.getRowCount(); + } + record = records.nextRecordFromSplit(); + } + splitId = records.nextSplit(); + } + finishedSplitIds.addAll(records.finishedSplits()); + return rows; + } + + private static void writeRows(String bootstrapServers, String tablePath) throws Exception { + StreamTableEnvironment tEnv = environment(bootstrapServers); + tEnv.executeSql("DROP TABLE IF EXISTS " + tablePath); + tEnv.executeSql( + "CREATE TABLE " + + tablePath + + " (id BIGINT, name STRING, score INT) WITH ('bucket.num' = '1')"); + tEnv.executeSql( + "INSERT INTO " + + tablePath + + " VALUES (1, 'alice', 10), (2, 'bob', 20), (3, 'carol', 30)") + .await(); + } + + private static void writeMultiBucketRows(String bootstrapServers, String tablePath) + throws Exception { + StreamTableEnvironment tEnv = environment(bootstrapServers); + tEnv.executeSql("DROP TABLE IF EXISTS " + tablePath); + tEnv.executeSql( + "CREATE TABLE " + + tablePath + + " (id BIGINT, name STRING, score INT)" + + " WITH ('bucket.num' = '3', 'bucket.key' = 'name')"); + tEnv.executeSql( + "INSERT INTO " + + tablePath + + " VALUES (1, 'alice', 10), (2, 'bob', 20), (3, 'carol', 30)," + + " (4, 'dave', 40), (5, 'erin', 50), (6, 'frank', 60)," + + " (7, 'grace', 70), (8, 'heidi', 80), (9, 'ivan', 90)") + .await(); + } + + private static void writeRowsWithUnusedUnsupportedColumn(String bootstrapServers, String tablePath) + throws Exception { + StreamTableEnvironment tEnv = environment(bootstrapServers); + tEnv.executeSql("DROP TABLE IF EXISTS " + tablePath); + tEnv.executeSql( + "CREATE TABLE " + + tablePath + + " (id BIGINT, unsupported TIMESTAMP_LTZ(3)) WITH ('bucket.num' = '1')"); + tEnv.executeSql( + "INSERT INTO " + + tablePath + + " VALUES" + + " (1, CAST(TIMESTAMP '2024-01-01 00:00:00.001' AS TIMESTAMP_LTZ(3)))," + + " (2, CAST(TIMESTAMP '2024-01-01 00:00:00.002' AS TIMESTAMP_LTZ(3)))") + .await(); + } + + private static void writeNestedRows(String bootstrapServers, String tablePath) throws Exception { + StreamTableEnvironment tEnv = environment(bootstrapServers); + tEnv.executeSql("DROP TABLE IF EXISTS " + tablePath); + String bidType = + "ROW"; + tEnv.executeSql( + "CREATE TABLE " + + tablePath + + " (event_type INT, bid " + + bidType + + ", `dateTime` TIMESTAMP(3)) WITH ('bucket.num' = '1')"); + tEnv.executeSql( + "INSERT INTO " + + tablePath + + " VALUES" + + " (2, CAST(ROW(100, 7, 42, 'web', 'https://nexmark.test/100'," + + " TIMESTAMP '2024-01-01 00:00:00.123', 'bextra-1') AS " + + bidType + + "), TIMESTAMP '2024-01-01 00:00:00.123')," + + " (2, CAST(ROW(101, 8, 43, 'mobile', 'https://nexmark.test/101'," + + " TIMESTAMP '2024-01-01 00:00:00.456', 'bextra-2') AS " + + bidType + + "), TIMESTAMP '2024-01-01 00:00:00.456')") + .await(); + } + + private static void writePartitionedRows(String bootstrapServers, String tablePath) + throws Exception { + createPartitionedTable(bootstrapServers, tablePath, "0 ms"); + StreamTableEnvironment tEnv = environment(bootstrapServers); + tEnv.executeSql("ALTER TABLE " + tablePath + " ADD PARTITION (region = 'US')"); + tEnv.executeSql("ALTER TABLE " + tablePath + " ADD PARTITION (region = 'EU')"); + tEnv.executeSql( + "INSERT INTO " + + tablePath + + " VALUES (1, 'US', 10), (2, 'EU', 20), (3, 'US', 30)") + .await(); + } + + private static void createPartitionedTable( + String bootstrapServers, String tablePath, String discoveryInterval) { + StreamTableEnvironment tEnv = environment(bootstrapServers); + tEnv.executeSql("DROP TABLE IF EXISTS " + tablePath); + tEnv.executeSql( + "CREATE TABLE " + + tablePath + + " (id BIGINT, region STRING, score INT)" + + " PARTITIONED BY (region)" + + " WITH ('bucket.num' = '1', 'scan.partition.discovery.interval' = '" + + discoveryInterval + + "')"); + } + + private static void addPartitionedRow( + String bootstrapServers, String tablePath, String region, int id, int score) throws Exception { + StreamTableEnvironment tEnv = environment(bootstrapServers); + tEnv.executeSql("ALTER TABLE " + tablePath + " ADD PARTITION (region = '" + region + "')"); + tEnv.executeSql( + "INSERT INTO " + + tablePath + + " VALUES (" + + id + + ", '" + + region + + "', " + + score + + ")") + .await(); + } + + private static void dropPartition(String bootstrapServers, String tablePath, String region) { + environment(bootstrapServers) + .executeSql("ALTER TABLE " + tablePath + " DROP PARTITION (region = '" + region + "')"); + } + + private static List> readRows(String bootstrapServers, String sql, int targetRows) + throws Exception { + return readRows(bootstrapServers, sql, targetRows, () -> {}); + } + + private static List> readRows( + String bootstrapServers, String sql, int targetRows, ThrowingRunnable afterStart) + throws Exception { + StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); + StreamTableEnvironment tEnv = environment(env, bootstrapServers); + PhysicalPlanScan scan = NativePlanner.install(tEnv); + + rowsCollected = new CountDownLatch(1); + collectedRows = Collections.synchronizedList(new ArrayList<>()); + Table table = tEnv.sqlQuery(sql); + tEnv.toDataStream(table).addSink(new CollectingSink(targetRows)).name("collect-native-fluss-it"); + JobClient job = env.executeAsync("native-fluss-source-sql-harness"); + try { + assertTrue( + scan.substitutions() > 0, + "Fluss source did not route to native; reasons=" + scan.fallbackReasons()); + afterStart.run(); + if (!rowsCollected.await(30, TimeUnit.SECONDS)) { + throw new TimeoutException("timed out waiting for native Fluss rows: " + collectedRows); + } + List> rows; + synchronized (collectedRows) { + rows = new ArrayList<>(collectedRows); + } + rows.sort(Comparator.comparing(Object::toString)); + return rows; + } finally { + job.cancel().get(); + rowsCollected = null; + collectedRows = null; + } + } + + private static List> readRowsAfterFirstRow( + String bootstrapServers, String sql, int targetRows, ThrowingRunnable afterFirstRow) + throws Exception { + firstRowCollected = new CountDownLatch(1); + try { + return readRows( + bootstrapServers, + sql, + targetRows, + () -> { + if (!firstRowCollected.await(30, TimeUnit.SECONDS)) { + throw new TimeoutException( + "timed out waiting for first native Fluss row: " + collectedRows); + } + afterFirstRow.run(); + }); + } finally { + firstRowCollected = null; + } + } + + private interface ThrowingRunnable { + void run() throws Exception; + } + + private static StreamTableEnvironment environment(String bootstrapServers) { + StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); + return environment(env, bootstrapServers); + } + + private static StreamTableEnvironment environment( + StreamExecutionEnvironment env, String bootstrapServers) { + env.setParallelism(1); + env.getConfig().enableObjectReuse(); + StreamTableEnvironment tEnv = StreamTableEnvironment.create(env); + tEnv.getConfig().getConfiguration().setString("execution.runtime-mode", "streaming"); + tEnv.executeSql( + "CREATE CATALOG " + + CATALOG + + " WITH ('type' = 'fluss', 'bootstrap.servers' = '" + + bootstrapServers + + "')"); + return tEnv; + } + + private static void dropTable(String bootstrapServers, String tablePath) { + try { + environment(bootstrapServers).executeSql("DROP TABLE IF EXISTS " + tablePath); + } catch (Exception ignored) { + // Best-effort cleanup; the embedded cluster is closed immediately afterwards. + } + } + + private static final class CollectingSink extends RichSinkFunction { + private final int targetRows; + + private CollectingSink(int targetRows) { + this.targetRows = targetRows; + } + + @Override + public void invoke(Row value, Context context) { + List> rows = collectedRows; + CountDownLatch latch = rowsCollected; + CountDownLatch firstLatch = firstRowCollected; + if (rows == null || latch == null) { + return; + } + synchronized (rows) { + if (rows.size() < targetRows) { + List fields = new ArrayList<>(value.getArity()); + for (int i = 0; i < value.getArity(); i++) { + fields.add(value.getField(i)); + } + rows.add(fields); + if (rows.size() == 1 && firstLatch != null) { + firstLatch.countDown(); + } + } + if (rows.size() >= targetRows) { + latch.countDown(); + } + } + } + } +} diff --git a/src/test/java/io/github/jordepic/streamfusion/fluss/NativeFlussTypeParityTest.java b/src/test/java/io/github/jordepic/streamfusion/fluss/NativeFlussTypeParityTest.java new file mode 100644 index 00000000..7de3a2ef --- /dev/null +++ b/src/test/java/io/github/jordepic/streamfusion/fluss/NativeFlussTypeParityTest.java @@ -0,0 +1,244 @@ +package io.github.jordepic.streamfusion.fluss; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertTrue; + +import io.github.jordepic.streamfusion.Native; +import io.github.jordepic.streamfusion.planner.NativePlanner; +import io.github.jordepic.streamfusion.planner.PhysicalPlanScan; +import java.time.LocalDateTime; +import java.util.ArrayList; +import java.util.Collections; +import java.util.Comparator; +import java.util.List; +import java.util.concurrent.CountDownLatch; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.TimeoutException; +import org.apache.flink.core.execution.JobClient; +import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment; +import org.apache.flink.streaming.api.functions.sink.legacy.RichSinkFunction; +import org.apache.flink.table.api.Table; +import org.apache.flink.table.api.bridge.java.StreamTableEnvironment; +import org.apache.flink.types.Row; +import org.apache.fluss.server.testutils.FlussClusterExtension; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.condition.EnabledIf; + +/** + * Type-parity referee for the native Fluss source: one column per whitelisted logical type + * (FlussTables' column gate), rows with edge values (nulls, min/max, empty string, empty-ish + * binary), written once through the stock connector and read back both ways — stock Fluss connector + * and the native fluss-rs path — asserting row-for-row equality. The native run additionally + * asserts the optimizer substituted the native source, so fallback cannot masquerade as parity. + * + *

Run with a native library built using {@code -Dnative.cargo.args="build --features fluss"}. + */ +@EnabledIf("nativeFlussFeatureBuilt") +class NativeFlussTypeParityTest { + + private static final String CATALOG = "fluss_parity_catalog"; + private static final String DATABASE = "fluss"; + private static final String FLUSS_SOURCE_FLAG = "streamfusion.operator.flussSource.enabled"; + private static final int ROWS = 4; + private static volatile CountDownLatch rowsCollected; + private static volatile List> collectedRows; + + @Test + void nativeAndStockSourcesReadWhitelistedTypesIdentically() throws Exception { + FlussClusterExtension cluster = FlussClusterExtension.builder().setNumOfTabletServers(1).build(); + String bootstrapServers = null; + String tablePath = null; + try { + cluster.start(); + bootstrapServers = cluster.getBootstrapServers(); + tablePath = CATALOG + "." + DATABASE + ".native_fluss_type_parity_it_" + System.nanoTime(); + writeRows(bootstrapServers, tablePath); + + String query = + "SELECT id, b, ti, si, i, bi, f, d, c, s, dc, dt, tm, ts, vb, nested FROM " + tablePath; + List> stockRows = readRows(bootstrapServers, query, false); + List> nativeRows = readRows(bootstrapServers, query, true); + + assertEquals(ROWS, stockRows.size()); + assertEquals(stockRows, nativeRows); + } finally { + if (bootstrapServers != null && tablePath != null) { + dropTable(bootstrapServers, tablePath); + } + cluster.close(); + } + } + + static boolean nativeFlussFeatureBuilt() { + return Native.flussFeatureBuilt(); + } + + private static void writeRows(String bootstrapServers, String tablePath) throws Exception { + StreamTableEnvironment tEnv = environment(bootstrapServers); + tEnv.executeSql("DROP TABLE IF EXISTS " + tablePath); + tEnv.executeSql( + "CREATE TABLE " + + tablePath + + " (id BIGINT, b BOOLEAN, ti TINYINT, si SMALLINT, i INT, bi BIGINT, f FLOAT," + + " d DOUBLE, c CHAR(5), s STRING, dc DECIMAL(10, 2), dt DATE, tm TIME," + + " ts TIMESTAMP(3), vb BYTES," + + " nested ROW)" + + " WITH ('bucket.num' = '1')"); + tEnv.executeSql( + "INSERT INTO " + + tablePath + + " VALUES" + + " (1, TRUE, CAST(1 AS TINYINT), CAST(2 AS SMALLINT), 3, 4," + + " CAST(1.5 AS FLOAT), 2.5, CAST('ab' AS CHAR(5)), 'héllo, fluss', 12345.67," + + " DATE '2024-02-29', TIME '12:34:56', TIMESTAMP '2024-02-29 12:34:56.789'," + + " x'01af', ROW(10, 'nested-alice', TIMESTAMP '2024-02-29 12:35:00.123'))," + + " (2, FALSE, CAST(-128 AS TINYINT), CAST(-32768 AS SMALLINT), -2147483648," + + " -9223372036854775807, CAST(-3.4028235E38 AS FLOAT)," + + " -1.7976931348623157E308, CAST('' AS CHAR(5)), '', -99999999.99," + + " DATE '1970-01-01'," + + " TIME '00:00:00', TIMESTAMP '1970-01-01 00:00:00.000', x'00'," + + " ROW(20, '', TIMESTAMP '1970-01-01 00:00:00.001'))," + + " (3, TRUE, CAST(127 AS TINYINT), CAST(32767 AS SMALLINT), 2147483647," + + " 9223372036854775807, CAST(3.4028235E38 AS FLOAT), 1.7976931348623157E308," + + " CAST('abcde' AS CHAR(5)), 'a longer utf-8 string', 99999999.99," + + " DATE '9999-12-31'," + + " TIME '23:59:59', TIMESTAMP '2262-04-11 23:47:16.854', x'deadbeef'," + + " ROW(30, 'nested-carol', TIMESTAMP '2024-01-01 00:00:00.999'))," + + " (4, CAST(NULL AS BOOLEAN), CAST(NULL AS TINYINT), CAST(NULL AS SMALLINT)," + + " CAST(NULL AS INT), CAST(NULL AS BIGINT), CAST(NULL AS FLOAT)," + + " CAST(NULL AS DOUBLE), CAST(NULL AS CHAR(5)), CAST(NULL AS STRING)," + + " CAST(NULL AS DECIMAL(10, 2)), CAST(NULL AS DATE), CAST(NULL AS TIME)," + + " CAST(NULL AS TIMESTAMP(3)), CAST(NULL AS BYTES)," + + " CAST(NULL AS ROW))") + .await(); + } + + private static List> readRows( + String bootstrapServers, String sql, boolean nativeSource) throws Exception { + String previous = System.getProperty(FLUSS_SOURCE_FLAG); + System.setProperty(FLUSS_SOURCE_FLAG, Boolean.toString(nativeSource)); + try { + StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); + StreamTableEnvironment tEnv = environment(env, bootstrapServers); + PhysicalPlanScan scan = NativePlanner.install(tEnv); + + rowsCollected = new CountDownLatch(1); + collectedRows = Collections.synchronizedList(new ArrayList<>()); + Table table = tEnv.sqlQuery(sql); + tEnv.toDataStream(table).addSink(new CollectingSink(ROWS)).name("collect-fluss-type-parity"); + JobClient job = env.executeAsync("native-fluss-type-parity"); + try { + if (nativeSource) { + assertTrue( + scan.substitutions() > 0, + "Fluss source did not route to native; reasons=" + scan.fallbackReasons()); + } else { + assertEquals(0, scan.substitutions(), "stock run unexpectedly substituted natively"); + } + if (!rowsCollected.await(30, TimeUnit.SECONDS)) { + throw new TimeoutException("timed out waiting for Fluss rows: " + collectedRows); + } + List> rows; + synchronized (collectedRows) { + rows = new ArrayList<>(collectedRows); + } + rows.sort(Comparator.comparingLong(row -> (Long) row.get(0))); + return rows; + } finally { + job.cancel().get(); + rowsCollected = null; + collectedRows = null; + } + } finally { + if (previous == null) { + System.clearProperty(FLUSS_SOURCE_FLAG); + } else { + System.setProperty(FLUSS_SOURCE_FLAG, previous); + } + } + } + + private static StreamTableEnvironment environment(String bootstrapServers) { + StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); + return environment(env, bootstrapServers); + } + + private static StreamTableEnvironment environment( + StreamExecutionEnvironment env, String bootstrapServers) { + env.setParallelism(1); + env.getConfig().enableObjectReuse(); + StreamTableEnvironment tEnv = StreamTableEnvironment.create(env); + tEnv.getConfig().getConfiguration().setString("execution.runtime-mode", "streaming"); + tEnv.executeSql( + "CREATE CATALOG " + + CATALOG + + " WITH ('type' = 'fluss', 'bootstrap.servers' = '" + + bootstrapServers + + "')"); + return tEnv; + } + + private static void dropTable(String bootstrapServers, String tablePath) { + try { + environment(bootstrapServers).executeSql("DROP TABLE IF EXISTS " + tablePath); + } catch (Exception ignored) { + // Best-effort cleanup; the embedded cluster is closed immediately afterwards. + } + } + + /** byte[] compares by identity inside a List; carry binary columns as List<Byte> instead. */ + private static Object comparable(Object field) { + if (field instanceof byte[]) { + byte[] bytes = (byte[]) field; + List boxed = new ArrayList<>(bytes.length); + for (byte b : bytes) { + boxed.add(b); + } + return boxed; + } + if (field instanceof Row) { + Row row = (Row) field; + List values = new ArrayList<>(row.getArity()); + for (int i = 0; i < row.getArity(); i++) { + values.add(comparable(row.getField(i))); + } + return values; + } + if (field instanceof java.sql.Timestamp) { + return ((java.sql.Timestamp) field).toLocalDateTime(); + } + if (field instanceof LocalDateTime) { + return field; + } + return field; + } + + private static final class CollectingSink extends RichSinkFunction { + private final int targetRows; + + private CollectingSink(int targetRows) { + this.targetRows = targetRows; + } + + @Override + public void invoke(Row value, Context context) { + List> rows = collectedRows; + CountDownLatch latch = rowsCollected; + if (rows == null || latch == null) { + return; + } + synchronized (rows) { + if (rows.size() < targetRows) { + List fields = new ArrayList<>(value.getArity()); + for (int i = 0; i < value.getArity(); i++) { + fields.add(comparable(value.getField(i))); + } + rows.add(fields); + } + if (rows.size() >= targetRows) { + latch.countDown(); + } + } + } + } +}