Skip to content

[Rust] Dynamic protobuf support#465

Open
andrijast-db wants to merge 7 commits into
mainfrom
andrijast-db/dynamic-proto-rust-dynamic-proto
Open

[Rust] Dynamic protobuf support#465
andrijast-db wants to merge 7 commits into
mainfrom
andrijast-db/dynamic-proto-rust-dynamic-proto

Conversation

@andrijast-db

@andrijast-db andrijast-db commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

What changes are proposed in this pull request?

Closes #372

Adds a native dynamic-protobuf path for ingesting records whose schema is known only at runtime, with no compiled prost::Message type.

New public API in the dynamic module:

  • StreamBuilder::dynamic_proto(descriptor) - select the format from a runtime DescriptorProto.
  • ZerobusStream::message_descriptor() / new_record() - resolve the schema (cached) and start a record.
  • dynamic::DynamicRecord - build records field-by-field with a validating, chainable .set(field, value)?.
  • dynamic::message_descriptor() standalone resolver, the IntoDynamicValue conversion trait, and re-exports of prost_reflect::{DynamicMessage, MessageDescriptor, Value}.

Follow-ups:

  • Add ability to fetch schema from UC

How is this tested?

Manually tested with custom Rust client

@andrijast-db andrijast-db force-pushed the andrijast-db/dynamic-proto-rust-dynamic-proto branch from 9bca843 to 10528f1 Compare July 2, 2026 09:56
@andrijast-db andrijast-db marked this pull request as ready for review July 2, 2026 12:31
@andrijast-db andrijast-db changed the title [Rust] Dynamic proto [Rust] Dynamic protobuf support Jul 2, 2026
@andrijast-db andrijast-db force-pushed the andrijast-db/dynamic-proto-rust-dynamic-proto branch from 10528f1 to df42e1f Compare July 2, 2026 13:32
Comment thread rust/sdk/src/dynamic.rs Outdated
@andrijast-db andrijast-db force-pushed the andrijast-db/dynamic-proto-rust-dynamic-proto branch from 8889215 to 5f9e3ae Compare July 6, 2026 09:29
Comment thread rust/examples/proto/dynamic/single.rs
Comment thread rust/examples/proto/README.md Outdated
Comment thread rust/sdk/src/dynamic_proto.rs
Comment thread rust/sdk/Cargo.toml Outdated
Comment thread rust/README.md Outdated
Comment thread rust/sdk/src/builder/stream_builder.rs Outdated
/// [`crate::dynamic::DynamicRecord`]. Wire-identical to
/// [`compiled_proto`](Self::compiled_proto); see the [`dynamic`](crate::dynamic)
/// module for the full pattern.
pub fn dynamic_proto(mut self, descriptor: prost_types::DescriptorProto) -> Self {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This now has the same signature as compiled_proto and collapses into the same (RecordType::Proto, Some(*desc)) build arm, the DynamicProto variant currently differs only in its Debug label, and message_descriptor()/new_record() work on compiled_proto streams too since they only check for a descriptor. So the second method adds a name but no distinct behavior.

I think we should look into dynamic_proto accepting the resolved descriptor instead:

pub fn dynamic_proto(mut self, descriptor: dynamic::MessageDescriptor) -> Self

The caller resolves once up front via dynamic::message_descriptor(&proto)? (or we add a descriptor_from_uc_columns to resolved-descriptor path), and the builder pulls the raw DescriptorProto back out for the CreateStream payload with MessageDescriptor::descriptor_proto().encode_to_vec(), wire-identical to today. This makes the method genuinely different from compiled_proto, fails fast on a bad descriptor at build() instead of at the first new_record(), and lets us drop the lazy OnceLock<MessageDescriptor> on ZerobusStream (the descriptor is now supplied, not lazily rebuilt).

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense. Switched to MessageDescriptor.

Comment thread rust/sdk/src/dynamic_proto.rs
Comment thread rust/sdk/src/dynamic_proto.rs
@andrijast-db andrijast-db force-pushed the andrijast-db/dynamic-proto-rust-dynamic-proto branch from b91a009 to d0f6aa6 Compare July 10, 2026 13:58
@andrijast-db andrijast-db force-pushed the andrijast-db/dynamic-proto-rust-dynamic-proto branch from 382d382 to ea6703f Compare July 10, 2026 15:11
@andrijast-db andrijast-db force-pushed the andrijast-db/dynamic-proto-rust-dynamic-proto branch from ea6703f to d973c74 Compare July 10, 2026 15:55
Comment thread rust/examples/proto/compiled/single.rs
Comment thread rust/sdk/src/dynamic_proto.rs
Comment thread rust/sdk/src/builder/stream_builder.rs Outdated
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Rust] Dynamic protobuf: runtime descriptor fetch + dynamic message ingestion

3 participants