Runtime platform environment
Proxy cluster mode with a Remoting transaction producer. The state transition can be reproduced deterministically in a service-level unit test, without a running broker or timing sleeps.
RocketMQ version
Current develop: 80e1ae55773c2330d2005b86f020ed028f649e94.
JDK Version
Local regression validation: Amazon Corretto 11.0.23, Maven 3.9.8, macOS arm64.
Describe the Bug
ClusterTransactionService.scanProducerHeartBeat() deletes a transaction subscription as soon as ProducerManager.groupOnline(group) returns false. A transaction send can create this subscription before the producer's first heartbeat registers its channel in the proxy.
If the scan runs in that interval, returning null from groupClusterData.computeIfPresent() removes the group-to-cluster mapping. A later producer heartbeat only updates ProducerManager; it does not rebuild the deleted transaction subscription. If the producer sends no further transaction messages, subsequent scans cannot register that group with the broker, so transaction checks cannot reach the producer.
This is a follow-up to #8316, which reports the same race and was closed by the stale bot on 2026-09-15. It remains reproducible in the current implementation. See also #5862 and its fix #5865: preventing indefinite heartbeats for producers that exit before their first heartbeat remains necessary.
The earlier fix proposal #8320 by @redlsz introduced a subscription timestamp and grace period for this race. It was closed as stale on 2026-05-18 without being merged, so those changes are not present in the current implementation.
Steps to Reproduce
Deterministic service-level sequence:
- Add a transaction subscription for a group and a topic with a valid broker-cluster route.
- Leave the group absent from
ProducerManager, representing a pending first producer heartbeat.
- Run
scanProducerHeartBeat().
- Register the group, or change the mocked
groupOnline(group) result to true.
- Run
scanProducerHeartBeat() again, without adding another transaction subscription.
The group is removed at step 3. Step 5 has no subscription from which to build a broker heartbeat.
An end-to-end trigger is to send a transaction whose local result is UNKNOWN before the first producer heartbeat, keep the producer alive, and send no additional transaction messages. The failure depends on the scan falling between subscription creation and producer registration; it does not happen on every run.
What Did You Expect to See?
A recently created transaction subscription should survive a delayed first producer heartbeat and resume broker heartbeats once the producer becomes online.
For the #5862 case, offline groups should receive no broker heartbeats, and subscriptions for producers that never register should eventually be reclaimed. Explicit group unregistration should still remove subscriptions immediately.
What Did You See Instead?
The subscription is removed immediately while the group is not yet registered. Registration alone does not restore it. The broker can then fail to find a channel for transaction checks, and the producer's transaction checker is not invoked.
Proposed Fix
Keep a millisecond lastActiveTimestamp, using System.currentTimeMillis(), on the existing ClusterData entries. Refresh it when subscriptions are added/replaced and while the group is observed online. When the group is offline, skip broker heartbeats immediately and expire subscriptions only after channelExpiredTimeout (currently 120 seconds by default). Preserve the existing explicit-unsubscribe path.
Regression coverage should include delayed first registration, never-registered expiration, renewed subscription activity, online activity, replacement, and explicit unsubscription during the grace period.
Runtime platform environment
Proxy cluster mode with a Remoting transaction producer. The state transition can be reproduced deterministically in a service-level unit test, without a running broker or timing sleeps.
RocketMQ version
Current
develop:80e1ae55773c2330d2005b86f020ed028f649e94.JDK Version
Local regression validation: Amazon Corretto 11.0.23, Maven 3.9.8, macOS arm64.
Describe the Bug
ClusterTransactionService.scanProducerHeartBeat()deletes a transaction subscription as soon asProducerManager.groupOnline(group)returns false. A transaction send can create this subscription before the producer's first heartbeat registers its channel in the proxy.If the scan runs in that interval, returning
nullfromgroupClusterData.computeIfPresent()removes the group-to-cluster mapping. A later producer heartbeat only updatesProducerManager; it does not rebuild the deleted transaction subscription. If the producer sends no further transaction messages, subsequent scans cannot register that group with the broker, so transaction checks cannot reach the producer.This is a follow-up to #8316, which reports the same race and was closed by the stale bot on 2026-09-15. It remains reproducible in the current implementation. See also #5862 and its fix #5865: preventing indefinite heartbeats for producers that exit before their first heartbeat remains necessary.
The earlier fix proposal #8320 by @redlsz introduced a subscription timestamp and grace period for this race. It was closed as stale on 2026-05-18 without being merged, so those changes are not present in the current implementation.
Steps to Reproduce
Deterministic service-level sequence:
ProducerManager, representing a pending first producer heartbeat.scanProducerHeartBeat().groupOnline(group)result totrue.scanProducerHeartBeat()again, without adding another transaction subscription.The group is removed at step 3. Step 5 has no subscription from which to build a broker heartbeat.
An end-to-end trigger is to send a transaction whose local result is
UNKNOWNbefore the first producer heartbeat, keep the producer alive, and send no additional transaction messages. The failure depends on the scan falling between subscription creation and producer registration; it does not happen on every run.What Did You Expect to See?
A recently created transaction subscription should survive a delayed first producer heartbeat and resume broker heartbeats once the producer becomes online.
For the #5862 case, offline groups should receive no broker heartbeats, and subscriptions for producers that never register should eventually be reclaimed. Explicit group unregistration should still remove subscriptions immediately.
What Did You See Instead?
The subscription is removed immediately while the group is not yet registered. Registration alone does not restore it. The broker can then fail to find a channel for transaction checks, and the producer's transaction checker is not invoked.
Proposed Fix
Keep a millisecond
lastActiveTimestamp, usingSystem.currentTimeMillis(), on the existingClusterDataentries. Refresh it when subscriptions are added/replaced and while the group is observed online. When the group is offline, skip broker heartbeats immediately and expire subscriptions only afterchannelExpiredTimeout(currently 120 seconds by default). Preserve the existing explicit-unsubscribe path.Regression coverage should include delayed first registration, never-registered expiration, renewed subscription activity, online activity, replacement, and explicit unsubscription during the grace period.