Skip to content

[Bug] Async send leaves requests unanswered when store future fails #11172

Description

@zmuxuny

Before Creating the Bug Report

  • I found a bug, not just asking a question.
  • I searched open/closed issues and pull requests for asyncSendEnable, asyncPutMessageFuture, exceptional completion, and send timeout, and found no equivalent report.
  • I confirmed the affected code is in Apache RocketMQ.

Runtime platform environment

Deterministic broker unit test on Windows; the control-flow defect is platform independent.

RocketMQ version

develop at bc33e8e4d7b25089af5f51bc669bdfedabfebe7d.

JDK Version

Amazon Corretto 8u432 for the regression test.

Describe the Bug

When asyncSendEnable=true, SendMessageProcessor returns null immediately and relies on the MessageStore future to send the response later. Both the single-message and batch paths attach only thenAcceptAsync(...) to that future.
If asyncPutMessage(...) / asyncPutMessages(...) completes exceptionally, the consumer passed to thenAcceptAsync is never invoked. Because the processor already returned null, no doResponse(...) path remains: the client receives no broker response and waits until an outer remoting timeout or connection teardown. The after-send hook is skipped as well.

The single-message branch and the batch branch have the same shape:

asyncPutMessageFuture.thenAcceptAsync(putMessageResult -> {
    RemotingCommand responseFuture = handlePutMessageResult(...);
    if (responseFuture != null) {
        doResponse(ctx, request, responseFuture);
    }
    sendMessageCallback.onComplete(sendMessageContext, response);
}, executor);
return null;

This is an observable asynchronous-completion gap, not just a theoretical null check. The store future contract can propagate exceptional completion from lower async stages; for example CommitLog#handleDiskFlushAndHA composes flush and HA futures with thenCombine.

Steps to Reproduce

  1. Enable async send in a SendMessageProcessorTest fixture.
  2. Make MessageStore.asyncPutMessage(...) return a CompletableFuture<PutMessageResult> completed exceptionally with RuntimeException("store write failed").
  3. Invoke processRequest with a normal SEND_MESSAGE request and capture channel.writeAndFlush.
  4. processRequest(...) returns null, as expected for async mode.
  5. Wait up to one second for a response.

On the unmodified baseline the regression fails deterministically:

Tests run: 1, Failures: 0, Errors: 1
ConditionTimeoutException: response was not fulfilled within 1 seconds

The preceding 10 reactor modules succeed; only this broker regression fails.

What Did You Expect to See?

An exceptional store future should complete the request promptly with SYSTEM_ERROR, log the original throwable on the broker, and execute the normal after-send hook exactly once. The client should not have to discover a completed store failure through a later request timeout.

What Did You See Instead?

No response is written at all. The client remains pending until external timeout/connection handling terminates the request.

Proposed scope

Handle both single-message and batch async-send futures symmetrically. Preserve all existing success/result-code behavior, synchronous-send behavior, metrics, and protocol formats. Add deterministic regressions for exceptional completion and retain existing success coverage.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions