Firestore `call was cancelled` crash occurs across multiple gRPC call paths

[anndressa-arruda]

[READ] Step 1: Are you in the right place?

Yes. This issue reports a bug in the Firestore code in this repository. It is not a general usage question or an application support request.

[REQUIRED] Step 2: Describe your environment

• Android Studio version: N/A
• Firebase Component: Firestore
• Component version: Firebase BoM 34.18.0; Firestore 26.6.0 (This crash was also observed in previous versions, including Firebase BoM 33.4.0; Firestore 25.1.2)

[REQUIRED] Step 3: Describe the problem

Steps to reproduce:

The crash is observed intermittently in production. We do not have a deterministic local reproduction. The crash is reported by Crashlytics and causes the application process to terminate.

The crash appears to be the same issue reported previously #6883. That issue was addressed by #6894, but the crash is still occurring for users.

The crash has been observed while different gRPC call paths are being closed. AsyncQueue is the common execution and error-reporting layer in all three traces, so the originating application-level call cannot be determined from these stack traces. The differences appear in the underlying Firestore/gRPC calls that reach the cancellation failure.

Observed stack traces:

1. Firestore stops listening to a target and sends an unwatch request:

Fatal Exception: java.lang.RuntimeException: Internal error in Cloud Firestore (26.6.0).
       at com.google.firebase.firestore.util.AsyncQueue.lambda$panic$3(AsyncQueue.java:546)
       at android.os.Handler.handleCallback(Handler.java:958)
       at android.os.Handler.dispatchMessage(Handler.java:99)
       at android.os.Looper.loopOnce(Looper.java:205)
       at android.os.Looper.loop(Looper.java:294)
       at android.app.ActivityThread.main(ActivityThread.java:8561)
       at java.lang.reflect.Method.invoke(Method.java)
       at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:652)
       at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1026)

Caused by java.lang.RuntimeException: java.lang.IllegalStateException: call was cancelled
       at com.google.firebase.firestore.util.AsyncQueue$SynchronizedShutdownAwareExecutor.lambda$executeAndReportResult$1(AsyncQueue.java:333)
       at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:520)
       at java.util.concurrent.FutureTask.run(FutureTask.java:328)
       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:323)
       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1100)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
       at com.google.firebase.firestore.util.AsyncQueue$SynchronizedShutdownAwareExecutor$DelayedStartFactory.run(AsyncQueue.java:235)
       at java.lang.Thread.run(Thread.java:1572)

Caused by java.lang.IllegalStateException: call was cancelled
       at com.google.common.base.Preconditions.checkState(Preconditions.java:513)
       at io.grpc.internal.ClientCallImpl.sendMessageInternal(ClientCallImpl.java:516)
       at io.grpc.internal.ClientCallImpl.sendMessage(ClientCallImpl.java:510)
       at io.grpc.internal.DelayedClientCall.sendMessage(DelayedClientCall.java:340)
       at io.grpc.ForwardingClientCall.sendMessage(ForwardingClientCall.java:37)
       at com.google.firebase.firestore.remote.AbstractStream.writeRequest(AbstractStream.java:412)
       at com.google.firebase.firestore.remote.WatchStream.unwatchTarget(WatchStream.java:99)
       at com.google.firebase.firestore.remote.RemoteStore.sendUnwatchRequest(RemoteStore.java:423)
       at com.google.firebase.firestore.remote.RemoteStore.stopListening(RemoteStore.java:406)
       at com.google.firebase.firestore.core.SyncEngine.stopListening(SyncEngine.java:291)
       at com.google.firebase.firestore.core.EventManager.removeQueryListener(EventManager.java:190)
       at com.google.firebase.firestore.core.FirestoreClient.lambda$stopListening$8(FirestoreClient.java:186)
       at com.google.firebase.firestore.util.AsyncQueue.lambda$enqueue$2(AsyncQueue.java:445)
       at com.google.firebase.firestore.util.AsyncQueue$SynchronizedShutdownAwareExecutor.lambda$executeAndReportResult$1(AsyncQueue.java:330)
       at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:520)
       at java.util.concurrent.FutureTask.run(FutureTask.java:328)
       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:323)
       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1100)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
       at com.google.firebase.firestore.util.AsyncQueue$SynchronizedShutdownAwareExecutor$DelayedStartFactory.run(AsyncQueue.java:235)
       at java.lang.Thread.run(Thread.java:1572)
       ...

2. Firestore closes a stream while restarting the network:

Fatal Exception: java.lang.RuntimeException: Internal error in Cloud Firestore (26.3.0).
       at com.google.firebase.firestore.util.AsyncQueue.lambda$panic$3(AsyncQueue.java:546)
       at android.os.Handler.handleCallback(Handler.java:958)
       at android.os.Handler.dispatchMessage(Handler.java:99)
       at android.os.Looper.loopOnce(Looper.java:205)
       at android.os.Looper.loop(Looper.java:294)
       at android.app.ActivityThread.main(ActivityThread.java:8236)
       at java.lang.reflect.Method.invoke(Method.java)
       at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:573)
       at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1058)

Caused by java.lang.IllegalStateException: call was cancelled
       at com.google.common.base.Preconditions.checkState(Preconditions.java:513)
       at io.grpc.internal.ClientCallImpl.halfCloseInternal(ClientCallImpl.java:500)
       at io.grpc.internal.ClientCallImpl.halfClose(ClientCallImpl.java:494)
       at io.grpc.internal.DelayedClientCall$7.run(DelayedClientCall.java:384)
       at io.grpc.internal.DelayedClientCall.drainPendingCalls(DelayedClientCall.java:302)
       at io.grpc.internal.DelayedClientCall.access$100(DelayedClientCall.java:50)
       at io.grpc.internal.DelayedClientCall$1.runInContext(DelayedClientCall.java:174)
       at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
       at io.grpc.internal.ManagedChannelImpl$RealChannel$PendingCall$1.run(ManagedChannelImpl.java:1025)
       at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:520)
       at java.util.concurrent.FutureTask.run(FutureTask.java:328)
       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:323)
       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1100)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
       at com.google.firebase.firestore.util.AsyncQueue$SynchronizedShutdownAwareExecutor$DelayedStartFactory.run(AsyncQueue.java:235)
       at java.lang.Thread.run(Thread.java:1572)
       ...

3. A pending gRPC call is started after cancellation:

Fatal Exception: java.lang.RuntimeException: Internal error in Cloud Firestore (26.3.0).
       at com.google.firebase.firestore.util.AsyncQueue.lambda$panic$3(AsyncQueue.java:546)
       at android.os.Handler.handleCallback(Handler.java:942)
       at android.os.Handler.dispatchMessage(Handler.java:99)
       at android.os.Looper.loopOnce(Looper.java:240)
       at android.os.Looper.loop(Looper.java:351)
       at android.app.ActivityThread.main(ActivityThread.java:8377)
       at java.lang.reflect.Method.invoke(Method.java)
       at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:584)
       at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1013)

Caused by java.lang.IllegalStateException: call was cancelled
       at com.google.common.base.Preconditions.checkState(Preconditions.java:513)
       at io.grpc.internal.ClientCallImpl.startInternal(ClientCallImpl.java:190)
       at io.grpc.internal.ClientCallImpl.start(ClientCallImpl.java:184)
       at io.grpc.internal.DelayedClientCall$2.run(DelayedClientCall.java:204)
       at io.grpc.internal.DelayedClientCall.drainPendingCalls(DelayedClientCall.java:302)
       at io.grpc.internal.DelayedClientCall.access$100(DelayedClientCall.java:50)
       at io.grpc.internal.DelayedClientCall$1.runInContext(DelayedClientCall.java:174)
       at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
       at io.grpc.internal.ManagedChannelImpl$RealChannel$PendingCall$1.run(ManagedChannelImpl.java:1025)
       at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:520)
       at java.util.concurrent.FutureTask.run(FutureTask.java:328)
       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:323)
       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1100)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
       at com.google.firebase.firestore.util.AsyncQueue$SynchronizedShutdownAwareExecutor$DelayedStartFactory.run(AsyncQueue.java:235)
       at java.lang.Thread.run(Thread.java:1572)
       ...

Expected behavior: Firestore and its gRPC call paths should safely handle cancellation, stream shutdown, and call startup, without propagating a fatal exception through AsyncQueue.panic().

Actual behavior: A cancelled gRPC client call reaches one of several gRPC operations after cancellation: ClientCallImpl.sendMessageInternal(), ClientCallImpl.halfCloseInternal(), or ClientCallImpl.startInternal(). Each path throws IllegalStateException: call was cancelled, which is propagated through Firestore's AsyncQueue as an internal Firestore error and crashes the application.