Skip to content

gmail: no retry on 429/5xx, and reads never use the local store #79

Description

@jqueguiner

Summary

The Gmail connector has no retry on transient API failures and no way to read from the
local store. Both are cheap to fix and both matter a lot when several void processes
share one Google account.

Gmail quota (Total Query Cost) is counted in units per minute per user, not per
process. N concurrent void clients on the same account share one budget. Today nothing
in void is aware of that.

Reproduction (observed in production)

Setup: ~10 agent sessions and cron jobs running against a single Google account, all
shelling out to void gmail. Measured on 2026-09-11.

Over the whole day:

Metric Value
Gmail-touching agent sessions 31
Failed Gmail calls returning 403 rateLimitExceeded 76
Gmail messages already in the local void store 1969
...of those, with a body of more than 200 characters stored 1320

Within the two-hour peak (15:30 to 17:30), for one profile:

Metric Value
gmail thread get calls 41
Distinct threads behind those 41 calls 16
Most re-fetches of a single thread 5

The error count is the number of tool results carrying a real Google quota error
payload, counted once per failed call. The scope of each block is stated because the
two are measured differently, and the numbers are not comparable across them.

At the peak, a hard 403 rateLimitExceeded took out every Gmail read for minutes, in
every process at once. The same thread was fetched up to 5 times, each time only to read
a different part of it. 1320 of those messages had a usable body sitting in the local
SQLite already.

Root cause

1. No retry anywhere in the Gmail client.

crates/void-gmail/src/api/client.rs has 14 call sites and all of them end the same way:

let resp = self
    .http
    .get(format!("{}/gmail/v1/users/me/messages", self.base_url))
    .bearer_auth(&self.access_token)
    .query(&params)
    .send()
    .await?
    .error_for_status()?;   // 429 -> fatal GmailError::Http, right here

grep -rn "retry\|backoff\|Retry-After" crates/void-gmail/src/ returns nothing except a
doc comment. A 429 is transient and retryable, and Retry-After tells you exactly how
long to wait. Void turns it into a fatal error instead.

The current behaviour is locked in by a test, api/tests.rs:225:

async fn create_draft_429_preserves_status() {
    // ... asserts the 429 propagates as GmailError::Http

That test has to change deliberately, which is part of why I am opening an issue before
a PR.

2. Reads never consult the local store.

crates/void-gmail/src/connector/api_methods.rs::search_api always goes to the network:

pub async fn search_api(&self, query: &str, max_results: u32)
    -> anyhow::Result<Vec<crate::api::GmailMessage>> {
    let api = self.get_client().await?;
    let resp = api.list_messages(max_results, None, None, Some(query)).await?;
    // ... then one get_message() per hit

That file contains zero references to the store or the db. So void gmail search costs
1 + N API calls even when every one of those messages is already synced locally. This
is the part that reads oddly for a tool whose premise is a local-first mirror.

Expected behaviour

  1. Retry with exponential backoff and jitter on 429 and 5xx, honouring Retry-After
    when present, bounded attempts, visible in logs. Non-retryable statuses (401, 403
    scope errors, 404) keep failing fast as they do now.
  2. Store-first reads. void gmail search and void gmail thread answer from the
    local store when the data is there, with an explicit --live escape hatch to force
    the network. Semantics stated plainly: the store is INBOX-only (see
    connector/sync.rs, INBOX_PAGE_SIZE / INBOX_MAX_PAGES), so in:sent, drafts and
    label queries still have to hit the API.
  3. Optional, only if 1 and 2 land: cross-process rate limiting. A shared token bucket
    per account, in the existing SQLite store or behind a file lock, so N concurrent void
    clients cannot collectively blow the per-user quota. This is the piece that actually
    fixes the multi-agent case, and it is also the most invasive, so it should come last
    and only if you like the approach.

Impact

Anyone running a single void process will mostly notice item 1, on a bad network or a
busy mailbox. Item 2 cuts API calls for the common "read the same thread again" pattern
regardless of setup. Item 3 only matters for multi-process users, which is the case I am
hitting.

Happy to do the work as separate PRs, smallest first, if you agree with the direction.
I would rather agree the scope here than send you a large PR you did not ask for.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions