Add Pathway docs to Third Party Tools

[abhirajadhikary06]

Pathway is an open-source, Rust-based framework for real-time data processing and streaming analytics. It enables developers to build low-latency pipelines for ingesting, transforming, and querying data at scale, with built-in support for incremental computation and connectors to databases like QuestDB.

Adding Pathway integration docs is essential because it bridges QuestDB's time-series strengths with Pathway's real-time processing capabilities, enabling seamless bidirectional streaming (e.g., via InfluxDB Line Protocol). This undocumented gap limits adoption in modern ETL/analyst workflows—official coverage would provide setup guides, examples, and troubleshooting, accelerating community contributions and positioning QuestDB as a top choice for dynamic data pipelines.

Pathway docs - https://pathway.com/developers/api-docs/pathway-io/questdb

@javier Assign this issue to me

[javier]

Hi. Thanks for wanting to contribute.

I am not sure I understand "bidirectional streaming (e.g., via InfluxDB Line Protocol)", as ILP is for input only. not for data out.

Which type of real-life example are you thinking of documenting with this? Also, I would make sure to mention it is a python project, even if powered by a Rust engine. Otherwise we could be setting false expectations.

[abhirajadhikary06]

Yeah, that's true afterall it is provided a python library for the work.
I was going through the docs of questdb and found out that as a third party connector Pathway is missing although Pathway is having a docs for itself with questdb on their platform.

So I thought that adding Pathway
to the QuestDB side will be useful.

The project I haven't thought about yet but if you wish to proceed with adding this as a third party tool, then I would research and build an documentation of a real life project

[javier]

For sure it would be interesting. But I read you said bidirectional streaming, and on the pathway docs I saw it seems to be just for output of data into QuestDB, so I was curious, as I've never used Pathway myself. It would be good to understand how you would frame this integration, so I can understand how useful this would be for our user base

[abhirajadhikary06]

For sure it would be interesting. But I read you said bidirectional streaming, and on the pathway docs I saw it seems to be just for output of data into QuestDB, so I was curious, as I've never used Pathway myself. It would be good to understand how you would frame this integration, so I can understand how useful this would be for our user base

Oh my bad, actually it is unidirectional (where stream processing takes place from Pathway pipleine to QuestDB via InfluxDB line protocol over http).
For making it bidirectional - pw.io.questdb.write() this allows us to write the data to QuestDB which makes an ingestion process which is first written to QuestDB's WAL (Write ahead log). The QuestDB WAL logs every record which we will use to check the changed data using a CDC tool and connect the CDC in turn with a real time streaming tool eg. Kafka, if using Kafka, we use the pathway connector for Kafka pw.io.kafka to ingest the data to Pathway.

[javier]

I am still not clear what would be the use case specifically for QuestDB users. Are you a user of Parhway and QuestDB yourself? How are you using them together?

Thanks

[abhirajadhikary06]

I am still not clear what would be the use case specifically for QuestDB users. Are you a user of Parhway and QuestDB yourself? How are you using them together?

Thanks

A couple of months back I had a small IoT project where I was logging the sensor data to QuestDB and was using Pathway for a small ETL function. Next from QuestDB using Debezium I used the streaming data to build my real time dashboard. So this way I used but I never looked for the docs for Pathway on QuestDB's documentation I used the Pathway docs itself and used gpt to build my project stuff.
Now when I came to know that QuestDB is in hacktober list so I check out that in Third Party tools Pathway was missing.

Speaking from Data Engineering p.o.v where there is lot much ETL work to do Pathway is required there, now when the data is in streaming format using QuestDB's fast access and time series features users can easily handle streaming data.

Hope now I have given the full answer of it's practical usage and the need. Previously I forgot to answer about the needs so here are those.

[javier]

Thanks! So in your case QuestDB was a destination of data, right? Rather than bi-directional. I am assuming Debezium was capturing data from somewhere else, then Pathway was reading from Debezium/Kafka, and you were outputting to QuestDB finally.

I am a bit confused because you mentioned several times debezium reading from QuestDB, both in your previous comment and in the last one, so I am not sure if you are saying you were using Debezium to monitor changes from the QuestDB WAL files and do the ETL on top of that?

[abhirajadhikary06]

There are two cases:
Case 1: I was extracting the data using Pathway performing some cleaning and conversion ops and logging the data to QuestDB.
Case2: From QuestDB WAL files I was capturing the changing data using Debezium and then pulled the data using Pathway's connector and broadcasted it on a dashboard.

My project's flow was something like this

[javier]

Thanks. I am very interested in Case2. How were you capturing changed with Debezium from the QuestDB WAL files?