Skip to content

Don't cache failed Bedrock Mantle model listings - #183

Merged
wch merged 1 commit into
mainfrom
bedrock-mantle-failure-caching
Nov 9, 2026
Merged

wch merged 1 commit into
mainfrom
bedrock-mantle-failure-caching

Conversation

@wch

@wch wch commented Nov 8, 2026

Copy link
Copy Markdown
Collaborator

When a user's AWS session expires and they sign in again outside the app (for example with aws sso login), the direct Bedrock provider lists Claude models on the next refresh but no OpenAI (Mantle) models for up to an hour. The Mantle listing turned every failure into an empty list, and the Bedrock model fetcher cached that empty list for the full one-hour cache lifetime. The Converse listing doesn't cache failures, so it recovered on the next refresh while Mantle didn't.

This PR makes the Mantle listing distinguish a stable empty answer from a failure:

  1. It returns an empty list only when a retry would give the same answer: a 404 (the region has no Mantle) or a 401/403 whose body is a recognized access-denied message. These stay cached for an hour, so accounts without Mantle access still send one listing request per hour.
  2. Every other outcome returns null: credential resolution failures (expired SSO), network errors, signature or expired-token rejections, unrecognized 401/403 bodies, other non-2xx responses, and malformed or unexpected bodies. The Bedrock fetcher never caches null, so the next refresh retries.
  3. An abort still rejects, as before.

A failure still lists no Mantle models for that refresh and never affects the Converse models. The Connect gateway consumer treats null as an empty list, so its behavior is unchanged.

Not included

  • The Connect gateway caches its combined model list separately, so a transient Mantle failure there is still cached. Users don't re-authenticate that path.
  • When a stale Mantle cache's refresh fails, Mantle returns no models instead of the stale list. The next successful refresh restores them.

listMantleModels turned every failure into [], and the direct Bedrock
fetcher cached that [] for the one-hour TTL. After an expired AWS session
and a re-auth outside the in-app flow (for example `aws sso login`),
Bedrock listed Claude models but no OpenAI (Mantle) models for up to an
hour.

listMantleModels now returns [] only when a retry would give the same
answer: a 404 (no Mantle in the region) or a recognized access-denied
body. Every other failure (credential resolution, signature or token
rejection, unrecognized 401/403, 5xx, malformed or unexpected bodies)
returns null. getMantleModels caches arrays and never caches null, so
the next listing retries. The Connect gateway consumer treats null as
[], as before.
@wch
wch merged commit f165bd6 into main Nov 9, 2026
4 checks passed
@wch
wch deleted the bedrock-mantle-failure-caching branch November 9, 2026 17:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant