fix(ws): make /ws credential revocation fail closed - #3336
vpetersson-bot wants to merge 9 commits into
Conversation
Follow-up Copilot review on PR #3324 found three ways a socket can still outlive the credentials it was accepted under. - Track an auth generation with each socket. A password or username rotation doesn't change the already-resolved scope['user'] — is_authenticated stays True for any real User row — so the per-frame re-check kept passing for a socket whose force_disconnect was lost (a transient Redis blip during the save is enough, since _broadcast deliberately swallows channel-layer failures). disconnect_all() now bumps a process-global counter before fanning the close out, and a socket whose recorded generation is stale goes silent. No DB hit per frame, and no dependence on the channel layer being healthy. - Skip that check while auth is disabled, so a socket that missed its close on an auth-off save isn't stranded on the 5s poll for good. - Reap sockets before publishing the viewer reload, on both settings-save surfaces. send_to_viewer() goes over Redis and can raise, which left the request exiting through its error handler with the new credentials persisted and every old socket still attached. Regression tests at both levels, each verified to fail without its fix. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: multica-agent <github@multica.ai>
There was a problem hiding this comment.
🟡 Changes recommended
Critical revocation gaps remain for settings-save failures and Django admin credential changes.
Get a fresh assessment by requesting another Copilot review.
Pull request overview
Hardens /ws credential revocation against lost Redis fan-out messages using process-local auth generations and earlier socket reaping.
Changes:
- Silences stale sockets after credential rotation.
- Reaps sockets before viewer reload publishing.
- Adds regression tests and documentation updates.
File summaries
| File | Summary / final comments |
|---|---|
website/content/docs/qa-checklist.md |
Adds credential-rotation WebSocket QA coverage. |
website/content/docs/developer-documentation.md |
Documents socket revocation. Nit (2 votes): clarify behavior when authentication is disabled. |
tests/test_template_views.py |
Tests HTML revocation when publishing fails. |
tests/test_consumers.py |
Tests generation-based stale-socket suppression. Critical (1 vote): Django admin credential changes bypass revocation. Nit (1 vote): qualify generation silencing while authentication remains enabled. |
src/anthias_server/app/views.py |
Reorders HTML settings revocation. Critical (3 votes): save failures after password mutation can bypass disconnect_all(). |
src/anthias_server/app/consumers.py |
Implements auth generations. |
src/anthias_server/api/views/v2.py |
Reorders API settings revocation. Critical (3 votes): save failures after password mutation can bypass disconnect_all(). |
src/anthias_server/api/tests/test_v2_endpoints.py |
Tests API revocation when publishing fails. |
Review details
Suppressed comments (1)
src/anthias_server/app/consumers.py:267
_is_authorized()returns before comparing generations whensettings['auth_backend']is empty (lines 112-115), so a socket that misses the close while authentication is being disabled is intentionally not silenced and continues receiving updates under the open-device contract. This paragraph overstates the guarantee; qualify the generation-based silencing as applying while authentication remains enabled.
The generation bump comes first and is the part that cannot fail:
the close below rides the channel layer and is swallowed if that is
down, whereas bumping the counter takes effect immediately and
silences every already-open socket from its next frame on. The
- Files reviewed: 8/8 changed files
- Comments generated: 4
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Second Copilot review on this PR found the generation bump was reachable from too few places, and reachable too late. - Revoke from a User post_save/post_delete receiver. /admin is a routed URL and the stock UserAdmin ships a change-password form, so a rotation there — or from `manage.py changepassword`, or a shell — never reached disconnect_all(). Hooking the model's own save is the one place that sees every credential mutation, including ones added later. last_login is exempt (Django writes it on each login through update_fields); anything else revokes, so is_active/is_staff changes and a deleted operator are covered too. - Reap in a finally around the settings write. apply_auth_settings() has already persisted the rotated User row by then, so a conf write that fails on a full or read-only /data volume jumped to the error handler with the new password live and every old socket attached. The signal now covers the rotation case on its own; the finally is what also covers a bare auth_backend toggle, which touches no User. - Stop the docs and the disconnect_all() docstring claiming a surviving socket always goes quiet — while auth is being disabled it keeps working by design, which is the open-device contract. Regression tests for each, verified to fail with the fix backed out. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: multica-agent <github@multica.ai>
There was a problem hiding this comment.
🟡 Changes recommended
Critical cross-process revocation gaps and masked regression tests remain unresolved.
Get a fresh assessment by requesting another Copilot review.
Review details
Suppressed comments (2)
src/anthias_server/api/tests/test_v2_endpoints.py:1024
- This test does not actually prove that the
finallyreap runs whensettings.save()fails: the User mutation invokes the new post-save receiver, which calls the same patcheddisconnect_allbeforesettings.save()is attempted. Removing the view'sfinallywould still leavedisconnect.calledtrue. Use an existing user and an auth-backend-only change, or otherwise isolate the signal, to make this regression assertion meaningful.
"""apply_auth_settings() persists the rotated User row before
settings.save() runs, so a conf write failure used to return through
the error handler with the new password live and every old socket
still attached. The reap is in a finally now."""
_device_settings_mock(settings_mock, auth_backend='')
settings_mock.save.side_effect = OSError('read-only file system')
tests/test_template_views.py:5161
- This test does not actually prove that the
finallyreap runs whensettings.save()fails: the User mutation invokes the new post-save receiver, which calls the same patcheddisconnect_allbeforesettings.save()is attempted. Removing the view'sfinallywould still leavedisconnect.calledtrue. Use an existing user and an auth-backend-only change, or otherwise isolate the signal, to make this regression assertion meaningful.
"""apply_auth_settings() persists the rotated User row before
settings.save() runs, so a conf write that fails — a full or
read-only /data volume — used to jump straight to the error handler
with the new password live and every old socket still attached.
The reap is in a finally now, so it happens either way."""
- Files reviewed: 10/10 changed files
- Comments generated: 5
- Review effort level: Lite
Third Copilot review on this PR. All five stand up. - Stop claiming out-of-process credential changes fail closed. The receiver runs in whichever process made the change, so `manage.py changepassword` or a shell bumps a counter nobody reads; only the best-effort Redis fan-out reaches uvicorn. What does fail closed is every surface served by uvicorn — the settings page, the v2 API and /admin — which is the gap that prompted the receiver. Narrowed the docstrings and the docs to say exactly that; making the rest fail closed would need the serving process re-reading credential state from the DB on a timer, which is the cost the counter exists to avoid. - Rewrite the settings-save reap tests. Adding the receiver hollowed them out: they enable auth by creating the operator, which fires the receiver, so `disconnect.called` stayed true with the view's reap deleted. They now toggle auth_backend against an operator row that already exists and send no new credentials, so no User is written, the receiver stays silent and the view is the only possible caller — and they assert the reap precedes the viewer publish rather than merely happening. Six tests, all verified to fail against the pre-fix arrangement; before this they could not. - Register the receiver through an explicit `signals.register()` call instead of a side-effect import needing `# noqa: F401`, which CLAUDE.md rules out where an idiom fixes the root cause. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: multica-agent <github@multica.ai>
There was a problem hiding this comment.
🔵 Needs a closer look
Unresolved moderate findings remain around save-failure handling, duplicate revocation, and transaction rollback safety.
Review details
Suppressed comments (5)
src/anthias_server/api/views/v2.py:714
- This
finallycovers every mutation beforesettings.save(), so a backend-only toggle can still bump the generation and close/silence valid sockets when an earlier setting operation fails even though the auth change was never persisted. Keep thefinallyaroundsettings.save()only (theUserreceiver already handles a password/username mutation), or otherwise distinguish a committed toggle from a pre-save exception; add a regression for that path.
finally:
# After settings.save() on the happy path, so a socket that
# reconnects immediately is judged against the new
# auth_backend rather than the old one. Before the viewer
# publish below, which goes over Redis and can raise.
src/anthias_server/api/views/v2.py:718
auth_changedis also true when only the username/password changed, but those changes already invokedisconnect_all()from the newUser.post_savereceiver. Thisfinallytherefore sends a second Redisforce_disconnectbroadcast and bumps the generation twice for every API credential rotation (and likewise when a toggle and rotation are combined). Please expose whetherapply_auth_settings()changed the backend versus the User row, and only perform this explicit reap for the backend-toggle case.
if auth_changed:
from anthias_server.app.consumers import disconnect_all
disconnect_all()
src/anthias_server/app/signals.py:81
post_save/post_deletefire before an enclosing database transaction commits, but this immediately advances the process-global generation and that bump cannot roll back. If an admin password change (or another caller usingtransaction.atomic()) later rolls back and Redis has dropped the close, otherwise-valid sockets remain permanently silent even though the credentials never changed. Schedule the revoke withtransaction.on_commitso the generation changes only after the User mutation is durable, and cover a rollback case.
# Imported lazily: consumers.py pulls in Channels, which the viewer
# image deliberately doesn't ship (see INSTALLED_APPS in
# django_project/settings.py) even though this app — and therefore
# this receiver — is installed there too.
from anthias_server.app.consumers import disconnect_all
logger.debug(
'Revoking /ws authorization after a change to user %r', instance.pk
)
disconnect_all()
src/anthias_server/app/views.py:1996
- This
finallycovers every mutation beforesettings.save(), so a backend-only toggle can still bump the generation and close/silence valid sockets when an earlier setting operation fails even though the auth change was never persisted. Keep thefinallyaroundsettings.save()only (theUserreceiver already handles a password/username mutation), or otherwise distinguish a committed toggle from a pre-save exception; add a regression for that path.
finally:
# After settings.save() on the happy path, so a socket that
# reconnects immediately is judged against the new
# auth_backend rather than the old one. Before the viewer
# publish below, which goes over Redis and can raise.
src/anthias_server/app/views.py:2000
auth_changedis also true when only the username/password changed, but those changes already invokedisconnect_all()from the newUser.post_savereceiver. Thisfinallytherefore sends a second Redisforce_disconnectbroadcast and bumps the generation twice for every settings-page credential rotation (and likewise when a toggle and rotation are combined). Please expose whetherapply_auth_settings()changed the backend versus the User row, and only perform this explicit reap for the backend-toggle case.
if auth_changed:
from anthias_server.app.consumers import disconnect_all
disconnect_all()
- Files reviewed: 10/10 changed files
- Comments generated: 0 new
- Review effort level: Lite
Fourth Copilot review on this PR — three findings, all of them fair. - Report backend and credential changes separately. apply_auth_settings returned one "something changed" bool, so the settings views reaped for a password rotation that the User receiver had already reaped, fanning a second force_disconnect over Redis and bumping the generation twice on every credential change. It now returns an AuthChange(backend_changed, credentials_rotated) and the views act only on the backend half — the half nothing else can see, since a toggle writes no User row. - Narrow the finally to settings.save(). Wrapping the whole mutation block meant an exception before the conf write also reaped; the write is the only thing there that can fail after apply_auth_settings() has committed its half. This also reverts the reindentation the wide version forced on ~8 unrelated lines. - Defer the receiver's revoke to transaction.on_commit. post_save fires inside the caller's transaction and Django's admin wraps its change form in one, so the write can still roll back — while the generation bump cannot, leaving a socket that also missed the close silent for good over a change that never happened. Outside a transaction (no ATOMIC_REQUESTS here) on_commit runs inline, so nothing is delayed on the paths that matter. Regression tests for each, plus a rolled-back admin-style change; all verified to fail with their fix backed out. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: multica-agent <github@multica.ai>
|
The latest Copilot pass ("needs a closer look") raised three findings as suppressed comments rather than threads, so there's nothing to resolve inline — answering them here. All three were fair; fixed in 7e4769e. Duplicate revocation on every credential rotation. The
Each fix was verified by backing it out: un-gating the reap fails the two rotation tests, replacing |
There was a problem hiding this comment.
🟡 Changes recommended
Resolve the duplicate revocation paths in both settings views and during initial authentication setup.
Get a fresh assessment by requesting another Copilot review.
Review details
Suppressed comments (1)
src/anthias_server/app/signals.py:81
- First-time auth setup currently causes multiple revocations for one operation:
_create_initial_operator()usesupdate_or_create()and then saves the same user again after hashing the password, so this receiver schedulesdisconnect_all()for both User writes;AuthChange.backend_changedthen makes each settings view broadcast a third time. On a Redis outage this multiplies the synchronous channel-layer timeout, and healthy clients receive redundant close events. Avoid emitting the receiver for the intermediatecreated=Truerow and create/save the initial user once, or otherwise coalesce this bootstrap revocation.
update_fields = kwargs.get('update_fields')
if (
update_fields is not None
and set(update_fields) <= _SESSION_NEUTRAL_FIELDS
):
return
- Files reviewed: 12/12 changed files
- Comments generated: 2
- Review effort level: Lite
…edentials The view-level reap keys on backend_changed alone, but one save can do both: enabling auth *and* supplying the username/password returns AuthChange(backend_changed=True, credentials_rotated=True), which tests/test_auth.py already pins. apply_auth_settings() writes the User, the post_save receiver bumps the generation and reaps — and then this finally block fans a second force_disconnect over Redis and bumps the generation again, for one operator action. That duplicate is the thing splitting the signal was meant to prevent. Guarded on `and not auth_change.credentials_rotated` in both settings views, with the rationale extended rather than left half-true. The regression test drives the real view and asserts the view-level disconnect_all is not called for that save. Verified it catches the bug: with the guards reverted it fails with "Expected 'disconnect_all' to not have been called. Called 1 times." Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: multica-agent <github@multica.ai>
There was a problem hiding this comment.
🔵 Needs a closer look
A moderate issue remains in signals.py: first-time setup can trigger duplicate revocations for one auth enable.
Review details
Suppressed comments (1)
Previously missed (1) — in code that hasn't changed since the last review.
src/anthias_server/app/signals.py:92
transaction.on_commitis registered once perUsersave, but first-time setup in_create_initial_operator()callsupdate_or_create()(which firespost_savefor the new row) and then callsuser.save()again after setting the password. Outside an outer transaction, both callbacks run immediately, so one initial auth enable sends twoforce_disconnectbroadcasts and bumps_auth_generationtwice. Consolidate creation into one User save or coalesce revocations per transaction so the one-revocation invariant also holds for first-time setup.
- Files reviewed: 12/12 changed files
- Comments generated: 0 new
- Review effort level: Lite
_create_initial_operator() called update_or_create() and then saved the same user again after hashing the password. The /ws revocation receiver is on post_save, so one operator action scheduled two disconnect_all() broadcasts — each paying the channel-layer timeout in full when Redis is down, and each sending healthy clients a redundant close. Build (or load) the row in memory, hash onto it, save once. The test pins the invariant by counting post_save: [True] now, [True, False] before. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: multica-agent <github@multica.ai>
There was a problem hiding this comment.
🟡 Changes recommended
A critical handshake race, a moderate atomic-upsert issue, and two test-credential nits remain.
Get a fresh assessment by requesting another Copilot review.
Review details
Suppressed comments (2)
src/anthias_server/api/tests/test_v2_endpoints.py:921
- These newly added test credentials are not covered by the test-file Sonar convention: the existing
_FIXTURE_PASSWORDandUserfactory in this file are centralized/suppressed because Sonar's S2068/S6437 scans test code. The added_OPERATOR_PWDand literal rotated passwords can fail that gate; reuse the existing fixture/factory pattern or add narrowly justifiedNOSONARsuppressions for the test-only values.
_OPERATOR_PWD = 'a-str0ng-QA-passphrase'
tests/test_template_views.py:5052
- These newly added test credentials are not covered by the test-file Sonar convention: existing fixtures in
tests/test_auth.py:48-56andsrc/anthias_server/api/tests/test_v2_endpoints.py:21are centralized and explicitly suppressed because Sonar's S2068/S6437 scans test code. The added_OPERATOR_PWDand literal password values in this block can fail that gate; centralize the fixtures and add a narrowly justifiedNOSONARsuppression (or otherwise follow the existing test helper pattern).
_OPERATOR_PWD = 'a-str0ng-QA-passphrase'
- Files reviewed: 12/12 changed files
- Comments generated: 2
- Review effort level: Lite
AuthMiddlewareStack resolves scope['user'] and only then dispatches to the consumer, so a rotation committing inside that window was missed: the user came from a session that was valid when it was read, while the consumer — constructed afterwards — read the already-bumped counter, found it current, and kept the socket indefinitely. An ASGI middleware outside AuthMiddlewareStack records the generation on the way in; the consumer compares that instead. A handshake that straddles a rotation now carries the pre-rotation value and is refused. Also keeps the initial-operator upsert atomic: update_or_create() with a pre-hashed password is one write *and* one statement, where the read-then-save it briefly became would have raised IntegrityError for the loser of two racing first-time enables. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: multica-agent <github@multica.ai>
There was a problem hiding this comment.
🟡 Changes recommended
Synchronize auth-generation increments and snapshots to prevent concurrent revocations from being lost.
Get a fresh assessment by requesting another Copilot review.
Review details
- Files reviewed: 13/13 changed files
- Comments generated: 1
- Review effort level: Lite
disconnect_all() can run on two threadpool workers at once (uvicorn runs sync views there), and `_auth_generation += 1` is load-add-store. A lost update would leave a socket stamped between the two changes matching the final generation — authorized after a revocation. Measured: no lost update in 480k racing increments at a 1 us switch interval, so the GIL hides this today and the test pins the invariant rather than demonstrating the race. The lock is for the build that doesn't have a GIL, and costs nothing uncontended. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: multica-agent <github@multica.ai>
There was a problem hiding this comment.
🔵 Needs a closer look
A moderate concurrency issue remains in WebSocket revocation handling and must be addressed before approval.
Review details
Suppressed comments (2)
src/anthias_server/app/consumers.py:363
- Because the generation is bumped before
_broadcast, a concurrent handshake can joinWS_GROUPafter this increment but before theforce_disconnectpublish. That socket carries the current generation and is otherwise valid, yet it still receives the in-flight revocation and is closed, so an immediate operator reconnect can be spuriously dropped. Include a revocation token/process identity in the event and makeforce_disconnectignore events for sockets stamped at or after that local bump, while retaining unconditional closes for events from other processes, or otherwise synchronize the membership/reap sequence.
with _auth_generation_lock:
_auth_generation += 1
website/content/docs/developer-documentation.md:17
- This sentence overstates the fail-closed guarantee for credential rotation:
_is_authorized()returnsTruebefore the generation check wheneverauth_backendis empty, so an admin username/password rotation made while authentication is disabled can leave a socket that missed the Redis close receiving updates. Qualify the rotation guarantee by authentication being enabled, or explicitly state that disabled mode remains open for these changes.
* The **web app** component (`anthias-server`) is the single HTTP entrypoint, served by uvicorn (ASGI). It runs the Django front-end + REST API, serves static assets via WhiteNoise, streams uploaded media at `/anthias_assets/`, and exposes the WebSocket endpoint at `/ws` via Django Channels. `/ws` follows the same authentication switch as the rest of the app: with **Authentication** set to **Basic** in Settings, the handshake must carry a logged-in browser session or it is refused; with authentication disabled the endpoint is open, like every other surface on the device. Authorization does not outlive the handshake. Turning authentication on, or rotating the operator's username or password (from the settings page, the API, or the Django admin), closes every open socket; any socket that survives the close — the close is best-effort, since it travels over Redis — stops receiving updates from that moment on. (That last guarantee covers changes made through the web app, which is the process holding the sockets; a password changed by `manage.py changepassword` on the device relies on the close alone.) Turning authentication *off* also closes them, but a socket that survives that close keeps working, because the device is open by contract in that mode and there is nothing left to revoke. Always plain HTTP, TLS is opt-in via the **anthias-caddy** sidecar that `bin/enable_ssl.sh` installs (Caddy local CA by default, or Let's Encrypt with `--domain`).
- Files reviewed: 13/13 changed files
- Comments generated: 0 new
- Review effort level: Lite
A handshake completing between the generation bump and the force_disconnect publish joins the group already stamped with the new generation — it was accepted under the new credentials — and was then closed by the revocation it postdates, dropping the operator's reconnect for nothing. The event now carries the bump it belongs to and the process that made it; a socket stamped at or after that bump ignores it. Events from any other process still close unconditionally, since their counter says nothing about this one's, as does an event with neither field (an older build's). Also qualifies the developer docs: with authentication disabled a rotation closes sockets but a survivor keeps working, because _is_authorized() returns True before the generation check in that mode. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: multica-agent <github@multica.ai>
|



Issues Fixed
Follow-up to #3324. Three review comments landed on it after it merged — one
critical, two moderate — all three real.
Description
Credential rotation could still outlive a socket.
asset_update()re-checks authorization per frame, but for a rotation that check kept
passing:
AuthMiddlewareStackresolvesscope['user']once at handshake, andis_authenticatedis True for any realUserrow whatever its password nowis. So the invariant rested entirely on the
force_disconnectfan-out landing— and that is best-effort by construction (
_broadcastswallows channel-layerfailures). A transient Redis blip during the settings save drops the close,
Redis recovers, and the socket accepted under the old password keeps receiving
every subsequent asset update.
disconnect_all()now bumps a process-global auth generation beforebroadcasting, and each socket records the generation it was accepted under. A
stale socket goes silent from its next frame on. The bump is the half that
cannot fail; the close stays the courteous half that lets the operator's own
tab re-handshake immediately and keep its live refresh.
In-process is the right scope here for the same reason the existing
settings['auth_backend']re-read is: uvicorn serves this app single-worker(
bin/start_server.sh), so the process handling the settings save is the oneholding every open socket. Deliberately not a per-frame credential lookup —
that would be a DB hit per socket per write on an SQLite/SBC device.
The check is skipped while auth is disabled. Turning auth off also bumps the
generation, but there are no credentials left to revoke and the documented
contract is that the device is open — a socket that missed its close shouldn't
be stranded on the 5s poll for the rest of its life.
The reap could be skipped entirely. Both settings-save surfaces ran
disconnect_all()aftersend_to_viewer('reload'), which publishes overRedis and can raise. The request then exited through its error handler with the
new credentials already persisted and every socket still attached under the old
ones. Both now reap immediately after
settings.save()— still after the save,so a socket reconnecting at once is judged against the new
auth_backend.Together these mean revocation no longer depends on Redis being up at the
moment the operator saves.
Checklist
Full suite green locally (2122 passed, 3 skipped), plus
ruff check,ruff format --checkandmypyclean. Each new test was verified to fail againstthe unfixed code: reverting the generation comparison fails the two rotation
tests, and restoring the original publish-then-reap order fails the two
settings-save tests. No hardware QA on this one — the change is a
process-local counter and a statement reorder, both covered end-to-end through
the real ASGI stack with a genuine session cookie; the QA checklist entry from
#3324 is extended with the rotation case for the next hardware pass.
🤖 Generated with Claude Code