diff --git a/GENERATION.md b/GENERATION.md index bc6e655..5721ccd 100644 --- a/GENERATION.md +++ b/GENERATION.md @@ -75,7 +75,6 @@ master the first time). All of that script's options pass through: ```bash tools/refresh-from-master.sh --mdb ~/src/MobilityDB # refresh against a local MobilityDB branch -tools/refresh-from-master.sh --skip-tests # regenerate + compile, skip the tests ``` This repo's last leg is `tools/refresh.conf`: the repo-root Maven module, the `org.jmeos:meos:1.0` diff --git a/README.md b/README.md index c39a206..138b4e6 100644 --- a/README.md +++ b/README.md @@ -1,82 +1,82 @@ - -MobilityKafka -=============== - -An open-source geospatial trajectory data streaming platform based on Apache [Kafka](https://kafka.apache.org/). - -MobilityDB Logo - -MobilityKafka explores the advantages of [MobilityDB](https://github.com/MobilityDB/MobilityDB) datatypes and functions in the Kafka environment, using the [JMEOS](https://github.com/MobilityDB/JMEOS) library as middleware. - -The MobilityDB project is developed by the Computer & Decision Engineering Department of the [Université libre de Bruxelles](https://www.ulb.be/) (ULB) under the direction of [Prof. Esteban Zimányi](http://cs.ulb.ac.be/members/esteban/). ULB is an OGC Associate Member and member of the OGC Moving Feature Standard Working Group ([MF-SWG](https://www.ogc.org/projects/groups/movfeatswg)). - -OGC Associate Member Logo - -More information about MobilityDB, including publications, presentations, etc., can be found in the MobilityDB [website](https://mobilitydb.com). - - -# BerlinMOD-9 × 3 streaming forms — the parity matrix on Kafka Streams - -The streaming-side parity matrix runs all nine BerlinMOD reference queries (Q1..Q9) in three streaming forms each on this runtime: **continuous** (always-on, per-event emission), **windowed** (tumbling 10-second aggregation), and **snapshot** (5-second tick — the parity-oracle form whose output at watermark T equals the batch BerlinMOD-Q result on data up to T). - -| Q | Topic | Continuous | Windowed | Snapshot | -|---|---|---|---|---| -| Q1 | "which vehicles have appeared in the stream?" | ✓ | ✓ | ✓ | -| Q2 | "where is vehicle X at time T?" | ✓ | ✓ | ✓ | -| Q3 | "vehicles within d of P at time T?" | ✓ | ✓ | ✓ | -| Q4 | "vehicles entered region R, and when?" | ✓ | ✓ | ✓ | -| Q5 | "pairs of vehicles meeting near P" | ✓ | ✓ | ✓ | -| Q6 | "cumulative distance per vehicle" | ✓ | ✓ | ✓ | -| Q7 | "first passage of vehicles through POIs" | ✓ | ✓ | ✓ | -| Q8 | "vehicles close to a road segment" | ✓ | ✓ | ✓ | -| Q9 | "distance between vehicles X and Y at time T" | ✓ | ✓ | ✓ | - -**27 / 27 cells** = the full MobilityKafka parity-matrix row. Each cell has a dedicated `Q{Continuous,Windowed,Snapshot}Processor` class in [`kafka-streams-app/src/main/java/berlinmod/`](kafka-streams-app/src/main/java/berlinmod/) and is locally verified via [`BerlinMODQ1LocalTest`](kafka-streams-app/src/main/java/berlinmod/BerlinMODQ1LocalTest.java) running on the Kafka-Streams `TopologyTestDriver` (no real broker required). - -## Module structure - -`kafka-streams-app/` is a Maven project (Java 21, Kafka Streams 3.6.0) holding: - -- 27 per-cell `Q{Continuous,Windowed,Snapshot}Processor` classes -- `BerlinMODTopology` — unified topology fanning input topic `berlinmod` to per-Q-form output topics -- `BerlinMODTrip` + `BerlinMODTripSerde` — shared data class + JSON Serde (byte-shape equivalent to MobilityFlink's `BerlinMODTrip`) -- `Haversine` + `SegmentDistance` + `PointOfInterest` — pure-Java geometry utilities used by the spatial-predicate cells -- `BerlinMODQ1LocalTest` — TopologyTestDriver-based local end-to-end driver - -The streaming snapshot form converges to the batch BerlinMOD result on the same scale-factor corpus, anchored against the cross-platform outputs in [MobilityDB-BerlinMOD](https://github.com/MobilityDB/MobilityDB-BerlinMOD). - -Spatial predicates today use pure-Java great-circle (`Haversine`) and planar segment-distance (`SegmentDistance`) utilities; each call site is marked `TODO(meos)` for JMEOS-bridge migration after [JMEOS#15](https://github.com/MobilityDB/JMEOS/pull/15) (the MEOS 1.4 regen) settles. - -## Build and run - -### MEOS native dependency - -The spatial predicates route through MEOS via the [JMEOS](https://github.com/MobilityDB/JMEOS) -bridge, so the build needs the JMEOS jar and the native `libmeos.so`. Neither is -committed to this repository — both are derived from source: derive `libmeos.so` -and the catalog from MobilityDB master, build the JMEOS jar and install it as -`org.jmeos:meos:1.0`, then build. `GENERATION.md` gives the exact commands, the same -ones MobilitySpark and MobilityFlink use. Once the jar is installed, JMEOS resolves -as an ordinary Maven dependency. - -### Build the app - -``` -cd kafka-streams-app -mvn -q clean package -DskipTests -java --add-opens java.base/java.lang=ALL-UNNAMED \ - --add-opens java.base/java.util=ALL-UNNAMED \ - --add-opens java.base/java.lang.reflect=ALL-UNNAMED \ - -cp target/mobility-kafka-streams-1.0-SNAPSHOT.jar \ - berlinmod.BerlinMODQ1LocalTest -``` - -The driver pipes a 21-event sorted-event-time corpus plus two sentinel records at `t = T0+15001` and `t = T0+20001` (to step the STREAM_TIME punctuator through the desired tick boundaries) and reads every per-Q-form output topic with the appropriate deserializer. Expected per-Q-form counts are in the PR body for the open scaffold PR. - -## Sibling parity work in the ecosystem - -- [MobilityFlink#3](https://github.com/MobilityDB/MobilityFlink/pull/3) — the same 27-cell row on Flink -- [MobilityNebula#15](https://github.com/MobilityDB/MobilityNebula/pull/15) — 15 of 27 cells on NebulaStream (Q1, Q2, Q3, Q4, Q7-via-POI-fanout) -- [MobilityDB-BerlinMOD#29](https://github.com/MobilityDB/MobilityDB-BerlinMOD/pull/29) — the batch BerlinMOD-9 cross-platform timings (the snapshot form's gold-answer source) -- [MobilityDB/.github#10](https://github.com/MobilityDB/.github/pull/10) — the ecosystem-profile description of the stream-layers tier - + +MobilityKafka +=============== + +An open-source geospatial trajectory data streaming platform based on Apache [Kafka](https://kafka.apache.org/). + +MobilityDB Logo + +MobilityKafka explores the advantages of [MobilityDB](https://github.com/MobilityDB/MobilityDB) datatypes and functions in the Kafka environment, using the [JMEOS](https://github.com/MobilityDB/JMEOS) library as middleware. + +The MobilityDB project is developed by the Computer & Decision Engineering Department of the [Université libre de Bruxelles](https://www.ulb.be/) (ULB) under the direction of [Prof. Esteban Zimányi](http://cs.ulb.ac.be/members/esteban/). ULB is an OGC Associate Member and member of the OGC Moving Feature Standard Working Group ([MF-SWG](https://www.ogc.org/projects/groups/movfeatswg)). + +OGC Associate Member Logo + +More information about MobilityDB, including publications, presentations, etc., can be found in the MobilityDB [website](https://mobilitydb.com). + + +# BerlinMOD-9 × 3 streaming forms — the parity matrix on Kafka Streams + +The streaming-side parity matrix runs all nine BerlinMOD reference queries (Q1..Q9) in three streaming forms each on this runtime: **continuous** (always-on, per-event emission), **windowed** (tumbling 10-second aggregation), and **snapshot** (5-second tick — the parity-oracle form whose output at watermark T equals the batch BerlinMOD-Q result on data up to T). + +| Q | Topic | Continuous | Windowed | Snapshot | +|---|---|---|---|---| +| Q1 | "which vehicles have appeared in the stream?" | ✓ | ✓ | ✓ | +| Q2 | "where is vehicle X at time T?" | ✓ | ✓ | ✓ | +| Q3 | "vehicles within d of P at time T?" | ✓ | ✓ | ✓ | +| Q4 | "vehicles entered region R, and when?" | ✓ | ✓ | ✓ | +| Q5 | "pairs of vehicles meeting near P" | ✓ | ✓ | ✓ | +| Q6 | "cumulative distance per vehicle" | ✓ | ✓ | ✓ | +| Q7 | "first passage of vehicles through POIs" | ✓ | ✓ | ✓ | +| Q8 | "vehicles close to a road segment" | ✓ | ✓ | ✓ | +| Q9 | "distance between vehicles X and Y at time T" | ✓ | ✓ | ✓ | + +**27 / 27 cells** = the full MobilityKafka parity-matrix row. Each cell has a dedicated `Q{Continuous,Windowed,Snapshot}Processor` class in [`kafka-streams-app/src/main/java/berlinmod/`](kafka-streams-app/src/main/java/berlinmod/) and is locally verified via [`BerlinMODQ1LocalTest`](kafka-streams-app/src/main/java/berlinmod/BerlinMODQ1LocalTest.java) running on the Kafka-Streams `TopologyTestDriver` (no real broker required). + +## Module structure + +`kafka-streams-app/` is a Maven project (Java 21, Kafka Streams 3.6.0) holding: + +- 27 per-cell `Q{Continuous,Windowed,Snapshot}Processor` classes +- `BerlinMODTopology` — unified topology fanning input topic `berlinmod` to per-Q-form output topics +- `BerlinMODTrip` + `BerlinMODTripSerde` — shared data class + JSON Serde (byte-shape equivalent to MobilityFlink's `BerlinMODTrip`) +- `Haversine` + `SegmentDistance` + `PointOfInterest` — pure-Java geometry utilities used by the spatial-predicate cells +- `BerlinMODQ1LocalTest` — TopologyTestDriver-based local end-to-end driver + +The streaming snapshot form converges to the batch BerlinMOD result on the same scale-factor corpus, anchored against the cross-platform outputs in [MobilityDB-BerlinMOD](https://github.com/MobilityDB/MobilityDB-BerlinMOD). + +Spatial predicates today use pure-Java great-circle (`Haversine`) and planar segment-distance (`SegmentDistance`) utilities; each call site is marked `TODO(meos)` for JMEOS-bridge migration after [JMEOS#15](https://github.com/MobilityDB/JMEOS/pull/15) (the MEOS 1.4 regen) settles. + +## Build and run + +### MEOS native dependency + +The spatial predicates route through MEOS via the [JMEOS](https://github.com/MobilityDB/JMEOS) +bridge, so the build needs the JMEOS jar and the native `libmeos.so`. Neither is +committed to this repository — both are derived from source: derive `libmeos.so` +and the catalog from MobilityDB master, build the JMEOS jar and install it as +`org.jmeos:meos:1.0`, then build. `GENERATION.md` gives the exact commands, the same +ones MobilitySpark and MobilityFlink use. Once the jar is installed, JMEOS resolves +as an ordinary Maven dependency. + +### Build the app + +``` +cd kafka-streams-app +mvn -q clean package +java --add-opens java.base/java.lang=ALL-UNNAMED \ + --add-opens java.base/java.util=ALL-UNNAMED \ + --add-opens java.base/java.lang.reflect=ALL-UNNAMED \ + -cp target/mobility-kafka-streams-1.0-SNAPSHOT.jar \ + berlinmod.BerlinMODQ1LocalTest +``` + +The driver pipes a 21-event sorted-event-time corpus plus two sentinel records at `t = T0+15001` and `t = T0+20001` (to step the STREAM_TIME punctuator through the desired tick boundaries) and reads every per-Q-form output topic with the appropriate deserializer. Expected per-Q-form counts are in the PR body for the open scaffold PR. + +## Sibling parity work in the ecosystem + +- [MobilityFlink#3](https://github.com/MobilityDB/MobilityFlink/pull/3) — the same 27-cell row on Flink +- [MobilityNebula#15](https://github.com/MobilityDB/MobilityNebula/pull/15) — 15 of 27 cells on NebulaStream (Q1, Q2, Q3, Q4, Q7-via-POI-fanout) +- [MobilityDB-BerlinMOD#29](https://github.com/MobilityDB/MobilityDB-BerlinMOD/pull/29) — the batch BerlinMOD-9 cross-platform timings (the snapshot form's gold-answer source) +- [MobilityDB/.github#10](https://github.com/MobilityDB/.github/pull/10) — the ecosystem-profile description of the stream-layers tier +