[Enhancement] Support Wildcard Pattern Matching for LiteTopic Consumer Subscriptions

[leizhiyuan]

Before Creating the Enhancement Request

• I have confirmed that this should be classified as an enhancement rather than a bug/feature.

Summary

Support wildcard pattern matching for LiteTopic consumer subscriptions, enabling hierarchical topic organization and fine-grained subscriber-side filtering.

Motivation

PR #10204 introduced Lite wildcard groups, but the current implementation has several limitations:

1. No consumer-side pattern filtering — A wildcard consumer group receives all LiteTopics under the parent topic. Consumers cannot express which subset they want; they always get everything.

2. Full LMQ scan on every dispatch cycle — doFullDispatchForWildcardGroup calls forEachLiteTopic() which scans the entire ConsumeQueue table (O(N) over all LMQs), not just the topics under the bound parent topic.

3. No hierarchical topic organization — LiteTopic names are flat strings with no way to express logical grouping (e.g., "all payment-related subtopics").

This makes it difficult to build systems where different consumers within the same group handle different subsets of events.

Describe the Solution You'd Like

Wildcard pattern matching for LiteTopic consumer subscriptions, inspired by NATS subject matching.

Key Design Decisions

Separator: __ (double underscore)

• . is already used in RocketMQ topic names (e.g., %LMQ%$parent$child) and in ConsumeQueue file paths
• __ is visually distinctive, rarely appears in topic names, and avoids conflicts

Pattern Tokens

Token	Meaning	Position	NATS Equivalent
*	Matches exactly one __-delimited segment	Anywhere	*
**	Matches one or more segments	Must be at the end	>

Examples (parent topic = order_events)

Pattern	Matches	Does Not Match
pay__refund	pay__refund	pay__refund__notify, notify__refund
pay__*	pay__refund, pay__success	pay__refund__notify
*__refund	pay__refund, notify__refund	pay__refund__notify
pay__*__notify	pay__refund__notify, pay__success__notify	pay__refund
**	pay, pay__refund, pay__refund__notify	(matches all)
pay__**	pay__refund, pay__refund__notify	notify__refund

Implementation Plan

Core matching engine: LitePatternMatcher in common module, providing three static methods:

• matches(pattern, liteTopic) — recursive segment-by-segment matching, no regex
• expand(pattern, candidates) — expand a pattern against a set of candidate topics
• validate(pattern) — validate pattern syntax

Two-phase subscription expansion:

1. Eager expansion at COMPLETE_ADD time — expand all patterns against existing LiteTopics under the parent topic, register matched topics to liteTopic2Group
2. Lazy re-expansion when new LiteTopics are created via onRegister() — check if the new topic matches any client's patterns, add to subscription if so

Dispatch optimization: Replace full LMQ scan (forEachLiteTopic, O(N)) with targeted iteration via collectByParentTopic (O(M), where M = topics under the parent topic).

Multi-pattern support (v1): liteTopicSet is a Set<String>, so multiple patterns can be sent in a single COMPLETE_ADD request. The union of all matched topics becomes the client's effective subscription.

Protocol compatibility: Reuse LiteSubscriptionDTO.liteTopicSet to carry pattern strings for wildcard groups. No wire protocol changes needed — the broker distinguishes via isWildcardGroup().

Consumer API Example

DefaultLitePullConsumer consumer = new DefaultLitePullConsumer("order_consumer_group");
consumer.setLiteBindTopic("order_events");
consumer.subscribeLite("pay__*");       // single-level payment events
consumer.subscribeLite("notify__**");   // all notification events (multi-level)

Key Changes

Component	Change
LitePatternMatcher (new)	Core matching engine in common
LiteSubscription	+ wildcardPatterns: Set<String>
LiteSubscriptionRegistryImpl	Multi-pattern expansion in addCompleteSubscription
LiteEventDispatcher	Lazy pattern re-expansion in onRegister; dispatch optimization

Describe Alternatives You've Considered

Alternative	Why Rejected
Store compiled Trie per wildcard group	Over-engineering for v1; String.split + recursive matching is fast enough for < 10K topics
Support ** in the middle of a pattern (e.g., order__**__refund)	NATS does not support this; greedy vs. non-greedy makes matching ambiguous; deferred to v2
Use . as separator	Conflicts with existing topic naming conventions in RocketMQ
Per-message pattern matching (no eager expansion)	Would require checking patterns on every doDispatch call; eager expansion is more efficient
Add a dedicated pattern field to LiteSubscriptionDTO	Unnecessary; reusing liteTopicSet is simpler and avoids wire protocol changes

Additional Context

• Inspired by NATS subject-based addressing and MQTT topic filters (with + / #), but using __ separator to avoid conflicts with RocketMQ's existing naming conventions
• Aligned with FlowMQ's "unified topic routing" philosophy — one topic namespace, multiple protocol bindings, unified wildcard matching
• Performance target: matches() < 5μs per call, expand() < 10ms for 1,000 candidates
• Full backward compatibility: wildcard groups without patterns continue to receive all LiteTopics under the parent topic (existing behavior)
• Registers via LiteSubscriptionCtlRequestBody without any wire protocol changes
• Reuses the existing collectByParentTopic API for dispatch optimization

[RockteMQ-AI]

Issue Evaluation

Category: type/enhancement | Status: Evaluated

Feasibility: Feasible
Scope: Broker dispatch layer (LiteTopic/LMQ), consumer subscription protocol, common module (new LitePatternMatcher)
Compatibility: Backward-compatible — existing exact-match subscriptions continue to work; pattern matching is opt-in.

Assessment:

This is a well-structured proposal that addresses real limitations in the current LiteTopic wildcard implementation (introduced in PR #10204). The three identified gaps — no consumer-side filtering, full LMQ scan overhead, and flat topic naming — are legitimate pain points for hierarchical event architectures.

Key observations:

• The __ separator choice is pragmatic, avoiding conflicts with existing . usage in topic names and ConsumeQueue paths.
• The NATS-inspired * / ** pattern semantics are well-understood and have proven production viability in other messaging systems.
• The two-phase expansion approach (eager at COMPLETE_ADD, lazy at dispatch) is a reasonable design that balances freshness with performance.
• The LitePatternMatcher in common with recursive segment matching (no regex) is the right performance-conscious approach.

Potential concerns for discussion:

1. Pattern index maintenance — The trie-based index for fast pattern→topic resolution adds memory overhead per broker. Consider the upper bound when thousands of patterns are active.
2. Cross-module coordination — If rocketmq-clients also needs pattern expression support (for proxy-mode consumption), the pattern syntax should be formalized as a protocol-level contract, not just a broker-internal format.
3. Validation edge cases — Patterns like **__something (non-terminal **) should be explicitly rejected; the validate() method needs clear rules.

Overall, this is a solid enhancement with a clear implementation path. Recommend proceeding to spec/PR phase after community discussion on the concerns above.

---

Automated evaluation by RockteMQ-AI