Repository navigation
Pub/Sub trigger: a redelivered message re-runs the agent in a new session, repeating tool side effects that already happened #7322
Description
🔴 Required Information
Describe the Bug:
Suppose a Pub/Sub trigger run fails after a tool has already caused an external
side effect, such as a payment, an email or a ticket.
- The endpoint returns HTTP 500 and Pub/Sub redelivers the message.
- ADK runs the agent again from the start, in a new session with no record of
the first run, so the side effect happens again.
Nothing inside the agent can prevent this. Pub/Sub keeps the messageId the
same across redeliveries ("A redelivered message retains the same message ID
between redelivery attempts"). ADK logs it but never passes it to the agent or
its tools, so a tool has nothing to deduplicate on.
In src/google/adk/cli/trigger_routes.py
(v2.10.0; byte-identical in 2.9.2):
- L419:
session_id = str(uuid.uuid4()), once per HTTP delivery. - L514-516:
the agent receives{"data": ..., "attributes": ...}. ThemessageIdis
not included. - L518-522:
themessageIdis only logged. - L530-541:
HTTP 500 for aTransientErrorafter retries, and for any other exception. - L486,
the endpoint's own description: "errors trigger Pub/Sub retry".
The ambient agents docs describe the
design: "Each redelivery creates a new session. Trigger workloads are stateless
by design." They don't say what this means for tools with side effects.
The in-process 429 retry is fine: it reuses the session, so the model can see
the tool call that already completed. The problem is the hand-off to Pub/Sub.
Steps to Reproduce:
pip install google-adk==2.10.0- Save the script under "Minimal Reproduction Code" below as
repro.py. - Run
python repro.py.
It needs no network access and no API key.
- The agent is a stock
LlmAgenton ADK'sGeminiclass, with
base_urlpointed at a local fake Gemini endpoint. - The fake endpoint answers the first request with a
pay_invoicecall.
It answers the request after the tool with HTTP 503 UNAVAILABLE ("The model
is overloaded") once, then behaves normally. - The loop at the bottom plays Pub/Sub. After a non-2xx response it posts
the same push envelope again, with the samemessageId.
Expected Behavior:
One payment for one message. Failing that, a way for the agent or its tools to
recognise a redelivery, so they can skip a side effect that already happened.
Observed Behavior:
google-adk 2.10.0
delivery 1 of messageId 1234567890: HTTP 500
delivery 2 of messageId 1234567890: HTTP 200
payments made for one message: 2
For delivery 1, ADK logs
Error processing Pub/Sub message: 503 UNAVAILABLE. {'error': {'code': 503, 'status': 'UNAVAILABLE', 'message': 'The model is overloaded. Please try again later.'}}.
Environment Details:
- ADK Library Version (pip show google-adk): 2.10.0. Also 2.9.2, whose
trigger_routes.pyis byte-identical. - Desktop OS: Windows 11
- Python Version (python -V): 3.13.9
Model Information:
- Are you using LiteLLM: No
- Which model is being used:
gemini-2.5-flashthrough ADK'sGeminiclass,
answered by a local fake endpoint. No real model was called.
🟡 Optional Information
Regression:
No. The same code is in the commit that added the trigger endpoints
(e2d970f).
Additional Context:
The same duplicate has other routes. They were measured with the same kind of
local harness: a fake Gemini endpoint, and Pub/Sub emulated from its documented
redelivery rule.
| What happens after the tool's side effect | Deliveries | Side effects |
|---|---|---|
| the next model call returns 503 | 500, 200 | 2 |
| the model returns 429 through all 3 in-process retries | 500, 200 | 2 |
| the tool's own HTTP call times out after the provider committed | 500, 200 | 2 |
| no error: the run outlasts the subscription's acknowledgement deadline | resent while the first run is still running | 2 (both runs complete) |
| the failure repeats on every delivery | 500 ×5 | 5 |
Two rows deserve a note:
- The acknowledgement-deadline row needs no failure at all. A push
subscription's acknowledgement deadline defaults to 10 seconds, and "if you
send a negative acknowledgment or the acknowledgment deadline expires,
Pub/Sub resends the message". The docs give "10 minutes (ack deadline)" as
the maximum processing time, but don't mention that a subscription created
with defaults gets 10 seconds. - The last row has no cap. Without a dead-letter policy, a failure that
repeats after the side effect repeats the side effect on every redelivery,
until the message expires. The docs recommend a dead-letter queue but don't
say why it matters here.
Two related observations:
-
Eventarc (Python): direct CloudEvents do put
ce-idin the agent's
input (L617-627); Pub/Sub-wrapped events don't. -
google/adk-goappears to share the design. I read its source but did
not run it:RetriableRunner.RunAgentcreates one session per delivery (its comment:
"One session per delivery").messageContentFromPubSubleaves out the message ID.- A failed run returns 500.
I'm happy to open a companion issue there.
Possible changes, roughly by cost:
- Pass the delivery's identity into the run. For example, put
messageId
(andce-id) in session state or in the attributes the agent receives, so
a tool can derive an idempotency key for its provider. This is small, and
lets applications fix the problem themselves. - Derive
session_idfrom the delivery (e.g.uuid5of subscription and
messageId) instead ofuuid4().- A redelivery then lands in the session that holds the completed tool
calls, as the in-process retry already does. - ADK could also acknowledge a redelivery whose session already holds a
final response, which covers the acknowledgement-deadline case. - A redelivery that arrives while the first run is still going would need
a guard.
- A redelivery then lands in the session that holds the completed tool
- Document it. Say that a redelivery repeats side effects that already
happened. Recommend an acknowledgement deadline above the expected run time
(up to 600 s), a dead-letter policy, and idempotent tools. Show how a
publisher can carry an idempotency key in the message attributes, which do
reach the agent today.
I'm happy to be told this is intended. In that case, (3) alone would still help.
Found while auditing retry behaviour across agent frameworks.
Minimal Reproduction Code:
"""Pub/Sub trigger: one message, two payments.
Self-contained: `pip install google-adk` then `python issue_repro.py`. No network
access and no API key; nothing leaves 127.0.0.1.
- The agent is a stock LlmAgent on ADK's stock Gemini class. `base_url` points
at a local fake Gemini endpoint.
- The fake endpoint answers the first request with a `pay_invoice` call. It
answers the request after the payment with HTTP 503 UNAVAILABLE ("The model
is overloaded"), once, and then behaves normally.
- The loop at the bottom plays Pub/Sub: after a non-2xx response it posts the
same push envelope again, with the same messageId.
"""
import base64
import json
import os
import sys
import tempfile
import textwrap
import threading
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from pathlib import Path
GEMINI = ("127.0.0.1", 8944)
PAYMENTS = [] # the payment provider's ledger: one entry per irreversible charge
_calls = {"n": 0}
class FakeGemini(BaseHTTPRequestHandler):
def do_POST(self): # noqa: N802
body = json.loads(self.rfile.read(int(self.headers["Content-Length"])))
_calls["n"] += 1
after_payment = "functionResponse" in body["contents"][-1]["parts"][-1]
if after_payment and _calls["n"] == 2:
return self._send(503, {"error": {"code": 503, "status": "UNAVAILABLE",
"message": "The model is overloaded. Please try again later."}})
part = ({"text": "Paid."} if after_payment else
{"functionCall": {"name": "pay_invoice",
"args": {"invoice": "INV-1", "amount": "100.00"}}})
self._send(200, {"candidates": [{"content": {"role": "model", "parts": [part]},
"finishReason": "STOP"}]})
def _send(self, status, obj):
data = json.dumps(obj).encode()
self.send_response(status)
self.send_header("Content-Type", "application/json")
self.send_header("Content-Length", str(len(data)))
self.end_headers()
self.wfile.write(data)
def log_message(self, *args):
pass
AGENT = f'''
import __main__
from google.adk.agents import LlmAgent
from google.adk.models import Gemini
def pay_invoice(invoice: str, amount: str) -> dict:
"""Pay an invoice. Irreversible."""
__main__.PAYMENTS.append((invoice, amount))
return {{"status": "PAID", "payment_number": len(__main__.PAYMENTS)}}
root_agent = LlmAgent(
name="payer",
model=Gemini(model="gemini-2.5-flash", base_url="http://{GEMINI[0]}:{GEMINI[1]}"),
instruction="Pay the invoice in the message once.",
tools=[pay_invoice],
)
'''
def main() -> int:
os.environ["GOOGLE_API_KEY"] = "placeholder" # never sent anywhere but the fake
os.environ.pop("GOOGLE_GENAI_USE_VERTEXAI", None)
server = ThreadingHTTPServer(GEMINI, FakeGemini)
threading.Thread(target=server.serve_forever, daemon=True).start()
agents = Path(tempfile.mkdtemp())
(agents / "payer").mkdir()
(agents / "payer" / "__init__.py").write_text("from . import agent\n")
(agents / "payer" / "agent.py").write_text(textwrap.dedent(AGENT))
import google.adk
from fastapi.testclient import TestClient
from google.adk.cli.fast_api import get_fast_api_app
app = get_fast_api_app(agents_dir=str(agents), web=False, trigger_sources=["pubsub"])
client = TestClient(app)
envelope = {"message": {"data": base64.b64encode(b'{"invoice": "INV-1", "amount": "100.00"}').decode(),
"messageId": "1234567890"},
"subscription": "projects/p/subscriptions/invoices"}
print(f"google-adk {google.adk.__version__}")
for delivery in range(1, 4):
r = client.post("/apps/payer/trigger/pubsub", json=envelope)
print(f"delivery {delivery} of messageId 1234567890: HTTP {r.status_code}")
if r.status_code in (102, 200, 201, 202, 204):
break # acknowledged; otherwise Pub/Sub redelivers the same message
print(f"payments made for one message: {len(PAYMENTS)}")
server.shutdown()
return 0 if len(PAYMENTS) > 1 else 1
if __name__ == "__main__":
sys.exit(main())How often has this issue occurred?:
Every time, under the conditions above. The reproduction is deterministic.
Activity
surajksharma07 commentedon Sep 28, 2026
@FurkhanShaikh. Reproduced on main with your script (500 then 200, two payments) and it's the same on every release since the trigger endpoints landed in v1.29.0.
Until #7325 is in a workaround that worked: a small ASGI middleware copies subscription:messageId into message.attributes, a before_agent_callback puts it in state and the tool passes it to the provider as an idempotency key. Two deliveries, one charge and a new messageId still gets charged. Could you try that in your harness especially the ack-deadline case?
@devansh173 #7325 looks good. One thing for the docstring: state is new on every delivery so the dedupe has to happen at the provider or in an external store not in state. Since messageId is only unique per topic maybe also suggest including the subscription in the key.
Please re-run the trigger tests and your E2E once more after that before asking for a review. The docs part (ack deadline, dead-letter policy) would be great as a follow-up in adk-docs.
devansh173 commentedon Sep 28, 2026
Thanks for reproducing it and for the review. I pushed the changes to #7325:
TRIGGER_DELIVERY_STATE_KEYdocstring now says session state is fresh on every delivery, so deduplication has to happen at the provider or in an external store. It recommends keying onsubscription+idfor Pub/Sub andevent_source+idfor Eventarc.Re-ran everything: all 78 trigger tests pass. In the E2E, the idempotent tool pays twice for one messageId on
mainand once on this branch, and a new messageId is still charged. The logs are in the PR description.I'll follow up with an adk-docs PR covering the ack deadline, dead-letter policy and idempotent tools.
surajksharma07 commentedon Sep 28, 2026
0db8ecb reads well @devansh173. The docstring now says clearly where the dedupe has to happen and how to build the key. Re-checked: the 78 trigger tests pass and an idempotent tool keyed on trigger_delivery (subscription + id, ledger outside the session) pays once across the 500/200 redelivery while a new messageId is still charged. The same tool on main pays twice.
From my side the PR is ready for maintainer review. Will ask for the workflow approvals. Please link the adk-docs PR here once it's up.
@FurkhanShaikh whenever you get a chance it would be good to hear whether this (or the middleware workaround on current releases) holds up in your harness especially the ack-deadline case where the first run is still in flight.
Option 2 (deterministic session id) can go to a separate follow-up issue if maintainers want it so it doesn't hold this one up.
devansh173 commentedon Sep 28, 2026
The adk-docs follow-up is up: google/adk-docs#2274. It covers the ack deadline, dead-letter topic and idempotent tools using
trigger_delivery, and asks to hold merging until #7325 is released.surajksharma07 commentedon Sep 28, 2026
adk-docs#2274 reads well @devansh173 and matches #7325 (fields, key format and the "dedupe outside session state" point). Would take up your offer to split out the ack-deadline + dead-letter part. It applies to every release since v1.29.0 today while trigger_delivery waits on #7325 (still at 0db8ecb, not merged, main unchanged).
Two small additions while you're there. The ack deadline only helps if the Cloud Run request timeout is at least as long: the default is 300s so a 10-minute run gets cut at 5 min and redelivered anyway. Worth adding gcloud run services update --timeout=600 next to the ack-deadline step. For the dead-letter step the Pub/Sub service agent needs publisher on the dead-letter topic and subscriber on the subscription or nothing gets forwarded.
Might also be worth a line that for Eventarc the same settings go on the trigger's underlying Pub/Sub subscription. Please re-run mkdocs build --strict after the changes.
@FurkhanShaikh still keen to hear how the workaround or #7325 does in your harness especially the ack-deadline case.
docs(runtime): show deduplicating trigger redeliveries by delivery ID
devansh173 commentedon Sep 28, 2026
Thanks! I split the ack-deadline and dead-letter part out as google/adk-docs#2275, which applies to current releases. While there I added the Cloud Run
--timeout=600step next to the ack deadline, the service agent's Publisher and Subscriber roles for the dead-letter topic, and a note for Eventarc Standard with a command to find the trigger's subscription. google/adk-docs#2274 now builds on it and only adds thetrigger_deliverypart, which waits on #7325.mkdocs build --strictpasses on both with no warnings.surajksharma07 commentedon Sep 28, 2026
The split is exactly what I had in mind @devansh173. adk-docs#2275 covers the Cloud Run timeout, the service agent roles and the Eventarc subscription lookup and #2274 now only adds the trigger_delivery part on top of it.
One thing I noticed in the raw file (ecf96ea, also in b517386): the multi-line gcloud commands lost their trailing \ continuations so the dead-letter update, the two add-iam-policy-binding commands and the eventarc describe each render as one long line with runs of spaces. They still work when pasted which is probably why mkdocs --strict stays quiet but restoring the \ line breaks would match the rest of the page.
Once that's fixed #2275 is ready for docs review from my side and can go out independently. #2274 waits for #7325 (still 0db8ecb, not merged; main unchanged so the redelivery issue still applies from v1.29.0 through v2.10.0).
@FurkhanShaikh the offer still stands to try the workaround or #7325 in your harness whenever you get a chance.
docs(runtime): show deduplicating trigger redeliveries by delivery ID
devansh173 commentedon Sep 28, 2026
Thanks for spotting this, @surajksharma07. The trailing
\continuations got lost when I split the section. Fixed in google/adk-docs#2275 (ba0c416): the dead-letter update, bothadd-iam-policy-bindingcommands and the Eventarcdescribeare split across lines again, matching the rest of the page. google/adk-docs#2274 is rebased on top of it. I re-ranmkdocs build --stricton both with no warnings, and checked the raw file: all 8 continuation lines end in\again.surajksharma07 commentedon Sep 29, 2026
Checked ba0c416 @devansh173. All 8 continuations are back and no gcloud command is left on one long line in either #2275 or #2274. #2274 is cleanly on top of it and its trigger_delivery table and example still match #7325.
With that adk-docs#2275 is good to go from my side. Will ask for a docs reviewer and the workflow approvals so it can ship independently for current releases.
On the code side #7325 (0db8ecb) still merges cleanly onto today's main (e4c0d94) and the 78 trigger tests pass on the merged tree. Just waiting on a maintainer review now, nothing more needed from you there.
#2274 stays on hold until #7325 is released. Until then the redelivery behaviour applies to v1.29.0 through v2.10.0.
FurkhanShaikh commentedon Sep 29, 2026
Thanks both, and sorry for the delay.
I ran #7325 (
0db8ecb) and the middleware workaround (on 2.10.0)through the harness, including the ack-deadline case.
Short version: the key works on both paths, and it prevents the duplicate
when the dedupe is atomic at the effect.
A provider idempotency key held in every case, including a real overlap.
One common reading of "check it against an external store" still pays twice.
Setup.
LlmAgenton the stockGeminiclass, pointed at a local fake Geminiendpoint.
scaled to 4 s (60 s elsewhere), and there was a 2 s wait before resending
after a nack.
with a key charges; a concurrent request with the same key gets 409; a later
one gets the stored reply.
The tool builds
f"{subscription}:{id}"fromtrigger_delivery(#7325), orfrom the attribute the middleware adds (workaround). It then dedupes one of
three ways:
provider_key: passes the key as the provider's idempotency key;check_then_act: checks an external store, pays, then records the key;reserve: atomically inserts the key into the store before paying, andkeeps it on failure.
Charges, identical on #7325 and on the workaround:
provider_keycheck_then_actreserveasynctoolIn the overlap cases with
provider_key, the provider sawcharged, then409for each redelivery that arrived while payment 1 was in flight. The toolraised on the 409, so the run failed and the message was nacked; a later
delivery got the original reply replayed, and was acked with one charge.
Things that might be worth a line in the #7325 docstring or adk-docs#2274:
held in every row, because the provider is the one place that knows
whether the effect happened.
effect. Check → act → record pays twice after an ambiguous failure (a
timeout after the provider committed), and again when a redelivery
overlaps a run still in flight.
failure before commit into a skipped effect. In the last-but-one row the
message was acked and nothing was paid. So a store-based dedupe needs a way
to reconcile an ambiguous attempt, not just a flag.
model as a tool result. That is what our tool did; we didn't test the
alternative. The reasoning: the nack keeps the message alive until a
delivery sees the original result, whereas acking on a 409 would lose the
payment if the in-flight attempt then failed before committing.
One side observation on the ack-deadline case. With a synchronous tool,
the trigger path (
runner.run_async) calls the tool inline on the event loop(
FunctionTool._invoke_callable, 2.10.0 lines 457–460).RunConfig.tool_thread_pool_configdoesn't change this: it is only read inthe live path (
_process_function_live_helper). While the first payment'sreply was held, that instance could not start the redelivered run at all:
It does happen across two instances, or with an async tool, which is why
those two rows are there. Probably a separate topic, but it seemed relevant
to the ack-deadline advice in adk-docs#2275: one slow sync tool call can stall
every trigger request on that instance.
Limits:
a real payment API. A provider whose keys don't cover concurrent in-flight
requests would behave like
check_then_actin the overlap rows.arguments on every delivery, so the key was
subscription:idalone.adk-docs#2274's example also appends
invoice_id, which a message withseveral payments needs. We didn't test a model that changes that argument
between deliveries; if it did, the key would change with it.
surajksharma07 commentedon Oct 1, 2026
Really useful matrix @FurkhanShaikh. Good to see the key behaves the same on #7325 and the middleware and that provider_key was the only one that held in every row, overlap cases included.
@devansh173 could you fold that into the TRIGGER_DELIVERY_STATE_KEY docstring and the adk-docs#2274 example? Prefer the provider's idempotency key. If you use a store reserve the key atomically before the effect and plan for reconciling an ambiguous attempt since check → act → record pays twice. And raise on a "key in use" 409 so the message is nacked rather than handing it back to the model. Worth noting the PR's E2E ledger is check-and-add in memory so it shouldn't read as the recommended pattern.
On the sync tool point: in 2.10.0 non-live runs do honor tool_thread_pool_config (_caller.py:924-950) but the trigger route calls runner.run_async() without a run_config (trigger_routes.py:382) so a sync tool runs on the event loop and blocks that instance. Until that's configurable an async tool (or asyncio.to_thread around the blocking call) avoids it. Opening a separate issue for that so it doesn't hold this one up.
#7325 (0db8ecb) is still unmerged and 2.10.0 is unchanged so it's still with the maintainers for review.
devansh173 commentedon Oct 1, 2026
Thanks @surajksharma07 and @FurkhanShaikh. I've folded this in:
c937675): theTRIGGER_DELIVERY_STATE_KEYdocstring now prefers the provider's idempotency key. It says a store has to reserve the key atomically before the effect, with a plan for reconciling an ambiguous attempt, since check → act → record pays twice. It also says to raise on a key-in-use 409 so the message is nacked. The PR description now notes that the E2EDONEledger is an in-memory check-and-add stand-in, not the recommended pattern. The change is docstring only, and the 78 trigger tests pass.2114346): the example says the same, and lets a 409 propagate instead of returning it to the model.mkdocs build --strictpasses.google/adk-docs#2275 is unchanged.
surajksharma07 commentedon Oct 5, 2026
Checked c937675 and adk-docs#2274 (2114346) @devansh173. The docstring and the example now say the same thing as FurkhanShaikh's matrix: provider key first, atomic reservation if it's a store and a 409 is raised rather than handed back to the model. The change is docstring only so behaviour is the same as 0db8ecb.
One thing to sort before review: main has moved under #7325. 14f4154 (max_llm_calls, in 2.11.0) added test_eventarc_passes_max_llm_calls at the exact spot where TestTriggerDeliveryIdentity goes so test_trigger_routes.py no longer applies cleanly. The source side still applies and after resolving that by hand I get 86 passed on the merged tree. Could you rebase onto current main and re-run the trigger tests?
On versions: 2.11.0 still creates a uuid4() session per delivery and doesn't pass messageId through so the redelivery behaviour still applies from v1.29.0 through v2.11.0. The trigger route now passes a RunConfig but it only sets max_llm_calls so the sync-tool-on-the-event-loop point from before still holds.
After the rebase it's back with the maintainers. adk-docs#2275 can still go out on its own and #2274 stays on hold until #7325 ships.
fix(cli): expose trigger delivery id so tools can detect redeliveries
devansh173 commentedon Oct 5, 2026
Thanks @surajksharma07. I rebased #7325 onto the current main (
ab9fffc), and it's now at54e1f19. All three commits applied without conflicts:test_eventarc_passes_max_llm_callsstays inTestTriggerEventarc, withTestTriggerDeliveryIdentityafter it, and_run_agentpasses both the newRunConfigand the session state. The trigger tests give 86 passed on the rebased branch, and the E2E repro still charges once across the 500 → 200 redelivery on 2.11.0.surajksharma07 commentedon Oct 6, 2026
Checked 54e1f19 @devansh173. The rebase is clean on ab9fffc and main has moved 23 commits since but none of them touch trigger_routes.py or its tests so it still merges without conflicts. On the merged tree I get 86 passed including the 7 TestTriggerDeliveryIdentity cases.
Also ran the repro from this issue against it. A tool that ignores trigger_delivery still charges twice across 500 → 200 which is expected since the PR exposes the id and the dedup stays in the tool as the docstring now says. The middleware workaround from earlier still passes on the same tree so anyone on v1.29.0 through 2.11.0 can keep using it until this ships.
Nothing else pending from my side. #7325 is ready for maintainer review (the 2 workflows still need approval) and adk-docs#2274 stays on hold until it's released.
fix: expose trigger delivery id so tools can detect redeliveries