fix(app): recover live-listen expired tokens - #12306
Conversation
Failure-Class: none
Git-on-my-level
left a comment
There was a problem hiding this comment.
Thanks @arhxam — the recovery module itself is cleanly built, but I found a wire-level gap that I believe prevents this from actually fixing #11694, so I'm requesting changes for now.
Core problem: the 4001/4004 close codes never reach onClosed in the failure mode this PR targets.
backend/utils/other/endpoints.pyraisesWebSocketException(code=4001/4004)inside_verify_ws_auth, which runs as a FastAPI dependency ofget_current_user_uid_ws_listen— i.e. beforewebsocket.accept().- With the pinned server stack (
uvicorn==0.30.5,websockets==12.0inbackend/requirements.txt), a close sent before the handshake completes is answered as HTTP 403 with the close code dropped (uvicorn'swebsockets_impl.pymaps a pre-acceptwebsocket.closetoHTTPStatus.FORBIDDEN). That matches the prod logs quoted in #11694 exactly:"WebSocket /v4/listen?... 403". - On the client, a rejected upgrade never produces a close frame:
channel.readyfails with aWebSocketChannelException,PureSocket.connect()(app/lib/services/sockets/pure_socket.dart) catches it and returnsfalse—onClosedis never called. So_scheduleLiveListenAuthRecoveryincapture_controller.dartnever fires for handshake rejections, and the retry storm in #11694 continues unchanged. - I also searched the backend for any post-accept producer of 4001/4004 and found none:
WS_AUTH_CODE_TOKEN_REFRESH/WS_AUTH_CODE_RELOGIN_REQUIREDare only used on the pre-accept path; accepted-socket closes use 1008/1003/1011 (listen runtime) or 4005/4006 (deletion/cutover fence). The backend unit tests (backend/tests/unit/test_ws_auth_handshake.py) pass because Starlette's TestClient preserves close-before-accept codes, while a real uvicorn websockets-protocol server does not.
Net: live_listen_auth_recovery.dart handles codes the current client+server pair cannot deliver for this bug.
Per file:
app/lib/services/capture/capture_controller.dart— the single-flight guard (_liveListenAuthRecoveryInFlight+identicalcheck) and the disposition switch are well done, and re-arming_startKeepAliveServices()onretryLater/notAuthRelatedpreserves the existing backoff;closeCode!is safe after the equality checks. The gap is only trigger reachability. One behavior worth a maintainer eye: 4004 now signs the user out immediately (expireSession) — reasonable for a revoked token, but it's a product behavior change on an auth surface.app/lib/services/sockets/live_listen_auth_recovery.dart— good pure design (DI ofrefreshToken/expireSession, exhaustive sealed-class switch overAuthTokenResult). If/when the codes actually arrive, this is the right shape — it's just unreachable today for the handshake-rejection scenario.app/test/unit/live_listen_auth_recovery_test.dart— solid matrix (success/transient/terminal/missing-user/4004/ordinary close, including asserting 4004 skips refresh). Missing the case that matters for #11694: an upgrade rejected with HTTP 403 delivers no close code at all and bypasses this module entirely.
Directions that would actually close #11694 (either, or coordinated):
- Client: handle the real signal — force
AuthService.instance.refreshIdToken()after a bounded number of consecutivePureSocket.connect()failures on an authed socket, or fix the stale-expiry gate the issue names as root cause:getAuthHeader()inapp/lib/backend/http/shared.darttrusts the cachedtokenExpirationTimepref and skips refresh when it is stale-but-in-the-future. - Backend: make
/v4/listendeliver the codes — accept first, then close with 4001/4004 (or use the websocket denial-response extension). That's a coordinated protocol change, andtest_ws_auth_handshake.pyshould keep modeling the real wire behavior.
Your recovery module slots straight into option 2 (and partly option 1), so this work isn't wasted — it's just ahead of the transport.
Maintainer note: beyond the contributor fix, this needs a maintainer decision on fix layering for #11694 (client refresh gate vs backend accept-then-close), and a quick sign-off on the new sign-out-on-4004 behavior since it changes an auth/session-expiry flow.
by AI on behalf of David — if you need David’s attention urgently, please @Git-on-my-level and escalate with need human response.
Failure-Class: none
What changed and why
Fix both sides of the live-listen expired-token loop:
getAuthHeader()now derives expiry from the cached Firebase JWT it is about to send. The token'sexpclaim wins over the separately persisted timestamp, so a dead token paired with stale future metadata is refreshed before the WebSocket upgrade instead of being rejected as HTTP 403 forever.Malformed or legacy non-JWT cached values retain the existing persisted-expiry fallback.
Closes #11694
Product invariants affected
none
How it was verified
flutter test test/unit/auth_token_expiration_test.dart test/unit/live_listen_auth_recovery_test.dart test/unit/backend/http/shared_test.dart test/unit/auth_service_token_result_test.dart test/unit/token_refresh_loop_test.dart— all 58 auth/header/recovery tests passed.flutter analyzeover the changed auth, capture, and test files — no issues.Tests
Failure-Class: none