From 9dbc46d6fabc04b65cb230d896d5f1ba11c0388b Mon Sep 17 00:00:00 2001 From: James Chapman Date: Fri, 13 Nov 2026 16:16:35 +0000 Subject: [PATCH 1/5] Record the 1.0.1 Complement waiver and the withdrawn 1.0.0 release (ADR-0744) 1.0.1 is tagged without a green Complement run because it fixes the vulnerabilities the 1.0.0 release ships; the 1.0.0 GitHub release is deleted and its tag kept. The waiver covers 1.0.1 only. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 3 +- ...lement-blocker-is-also-waived-for-1-0-1.md | 55 +++++++++++++++++++ docs/adr/index.md | 3 +- docs/release-process.md | 10 +++- docs/todos/production-milestone.md | 3 +- 5 files changed, 68 insertions(+), 6 deletions(-) create mode 100644 docs/adr/0744-the-complement-blocker-is-also-waived-for-1-0-1.md diff --git a/CHANGELOG.md b/CHANGELOG.md index b7732b7db..0c71b98c6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -192,7 +192,8 @@ high, all availability) and hardens two of the four findings that needed validat `federation.worker.database_uri_file` and `federation.worker.allow_shared_database_credentials` are now refused at start-up, so remove them from a PostgreSQL deployment first; the worker's database role can then be dropped (ADR-0736). The changelog now starts at 1.0.0 and uses the format in `docs/versioning.md` (ADR-0740, ADR-0741). A -tag now ships exactly the `latest` package set (ADR-0742). +tag now ships exactly the `latest` package set (ADR-0742). Like 1.0.0, 1.0.1 was tagged without a green Complement +run (ADR-0744). The 1.0.0 GitHub release is withdrawn because it ships the vulnerabilities fixed here; upgrade to 1.0.1. # Version 1.0.0 diff --git a/docs/adr/0744-the-complement-blocker-is-also-waived-for-1-0-1.md b/docs/adr/0744-the-complement-blocker-is-also-waived-for-1-0-1.md new file mode 100644 index 000000000..4b6693570 --- /dev/null +++ b/docs/adr/0744-the-complement-blocker-is-also-waived-for-1-0-1.md @@ -0,0 +1,55 @@ +# The Complement blocker is also waived for 1.0.1 + +* Status: accepted +* Date: 2026-11-13 + +## Context and Problem Statement + +[ADR-0648](0648-the-complement-blocker-is-waived-for-1-0-0-only.md) waived the green Complement run for 1.0.0 only, and +`docs/release-process.md` said the waiver applies to no later release. + +1.0.1 fixes the 49 findings of the run-3 security audit of the 1.0.0 release candidate, 9 of them high. The 1.0.0 release +ships those vulnerabilities, so the owner decided on 2026-11-13 to tag 1.0.1 at once and remove the 1.0.0 release. No +Complement run exists for 1.0.1. Can 1.0.1 be tagged without one? + +## Considered Options + +* Waive the blocker for 1.0.1 as well; it stays open for every later release +* Hold the 1.0.1 tag until a green Complement run exists + +## Decision Outcome + +Chosen option: "Waive the blocker for 1.0.1 as well". + +What is waived: the green Complement run before the 1.0.1 tag. 1.0.1 carries no Complement evidence and no signed +Complement record is attached to it. + +Why: the only published release, 1.0.0, has known vulnerabilities and 1.0.1 fixes them. Holding the fix for a Complement +run would leave 1.0.0 as the only release to install. The run-3 audit and this project's own suites gate 1.0.1, as the +run-2 audit gated 1.0.0. + +What still applies: + +* The blocker stays open in `docs/todos/production-milestone.md`. +* 1.0.2 and every later release follow `docs/release-process.md` as written, including a green Complement run before the + tag. This waiver is no precedent for them. +* The 1.0.0 GitHub release is deleted, because it ships the vulnerabilities 1.0.1 fixes. The `v1.0.0` tag stays: a tag is + never moved or removed once something was published under it, and ADR-0651 cites it. + +### Negative Consequences + +* Federation conformance for 1.0.1 rests on this project's own tests and the audit. No external check supports it. + +## Pros and Cons of the Options + +### Hold the 1.0.1 tag until a green Complement run exists + +* Good, because it follows the release process as written. +* Bad, because the vulnerable 1.0.0 stays the only release until a run is green, and no green run has ever been recorded. + +## Links + +* [ADR-0648](0648-the-complement-blocker-is-waived-for-1-0-0-only.md), + [ADR-0651](0651-the-v1-0-0-tag-moves-to-the-release-workflow-fix-without-a-version-bump.md), + [ADR-0235](0235-complement-evidence-is-a-maintainer-signed-record-tied-to-the-shipped-binary.md) +* [docs/release-process.md](../release-process.md), [docs/todos/production-milestone.md](../todos/production-milestone.md) diff --git a/docs/adr/index.md b/docs/adr/index.md index 5791492bc..959e952ee 100644 --- a/docs/adr/index.md +++ b/docs/adr/index.md @@ -560,7 +560,7 @@ * [0635 - An inbound federation origin is asked about once at a time, and its decision is kept for a short time](0635-an-inbound-federation-origin-is-asked-about-once-at-a-time-and-its-decision-is-kept-for-a-short-time.md) * [0636 - A blob row is deleted with its last reference, and a removed record holds no reference](0636-a-blob-row-is-deleted-with-its-last-reference-and-a-removed-record-holds-none.md) * [0647 - Versions from 1.0.0 are MAJOR.MINOR.BUILD](0647-versions-from-1-0-0-are-major-minor-build.md) -* [0648 - The Complement blocker is waived for 1.0.0 only](0648-the-complement-blocker-is-waived-for-1-0-0-only.md) +* [0648 - The Complement blocker is waived for 1.0.0 only](0648-the-complement-blocker-is-waived-for-1-0-0-only.md) — Extended by ADR-0744: 1.0.1 is waived too * [0731 - A spawned worker gets no terminal, and no sandbox profile allows TIOCSTI or TIOCLINUX](0731-a-spawned-worker-gets-no-terminal-and-no-sandbox-profile-allows-tiocsti.md) * [0732 - Every published attestation check pins the signing workflow and the ref](0732-every-published-attestation-check-pins-the-signing-workflow-and-the-ref.md) — Partly superseded by ADR-0742: every published file is attested by packages.yml * [0719 - One client key holds a sixteenth of the parked-connection budgets, and a readable connection waits a bounded time](0719-one-client-key-holds-a-sixteenth-of-the-parked-connection-budgets-and-a-readable-connection-waits-a-bounded-time.md) @@ -641,3 +641,4 @@ * [0757 - Hardened BSD builds link -pie, and the ELF gate reads elftoolchain readelf](0757-hardened-bsd-builds-link-pie-and-the-elf-gate-reads-elftoolchain-readelf.md) * [0758 - Every worker shard reloads every changed room, and main tracks each shard's copy](0758-every-worker-shard-reloads-every-changed-room-and-main-tracks-each-shards-copy.md) * [0759 - Main builds make_leave and make_knock templates while the worker's reload of the room is pending](0759-main-builds-make-leave-and-make-knock-templates-while-the-workers-reload-is-pending.md) +* [0744 - The Complement blocker is also waived for 1.0.1](0744-the-complement-blocker-is-also-waived-for-1-0-1.md) diff --git a/docs/release-process.md b/docs/release-process.md index ab9befeba..f76f16983 100644 --- a/docs/release-process.md +++ b/docs/release-process.md @@ -189,7 +189,9 @@ before the tag is created. The one exception is 1.0.0: the owner waived this requirement for that release only, because it was tagged without a green run and the release-gate security audit was its gate instead ([ADR-0648](adr/0648-the-complement-blocker-is-waived-for-1-0-0-only.md)). The -waiver does not apply to any later release. +owner waived it once more for 1.0.1, which fixes the vulnerabilities 1.0.0 ships +([ADR-0744](adr/0744-the-complement-blocker-is-also-waived-for-1-0-1.md)). The +waiver does not apply to 1.0.2 or any later release. After that, create and push the release tag. It must be exactly `v` followed by the version in `meson.build`, or the workflow refuses it: @@ -211,8 +213,10 @@ artifact. ## Complement -The 1.0.0 release was tagged without a Complement run and carries no Complement -evidence ([ADR-0648](adr/0648-the-complement-blocker-is-waived-for-1-0-0-only.md)). +The 1.0.0 and 1.0.1 releases were tagged without a Complement run and carry no Complement +evidence ([ADR-0648](adr/0648-the-complement-blocker-is-waived-for-1-0-0-only.md), +[ADR-0744](adr/0744-the-complement-blocker-is-also-waived-for-1-0-1.md)). The 1.0.0 GitHub +release was deleted because it ships the vulnerabilities 1.0.1 fixes; its `v1.0.0` tag stays. Every later release follows the steps below. The external Complement suite runs against the candidate's Debian 13 (trixie) package diff --git a/docs/todos/production-milestone.md b/docs/todos/production-milestone.md index eb31904b1..f8f994ea0 100644 --- a/docs/todos/production-milestone.md +++ b/docs/todos/production-milestone.md @@ -16,7 +16,8 @@ and what has been deliberately dropped. ## What blocks 1.0.0 **1.0.0 is released with the Complement blocker waived, not closed.** -The owner waived it for 1.0.0 only ([ADR-0648](../adr/0648-the-complement-blocker-is-waived-for-1-0-0-only.md)). +The owner waived it for 1.0.0 ([ADR-0648](../adr/0648-the-complement-blocker-is-waived-for-1-0-0-only.md)) +and again for 1.0.1 ([ADR-0744](../adr/0744-the-complement-blocker-is-also-waived-for-1-0-1.md)), but for no later release. No green Complement run was ever recorded: the last, run 18 at 0.12.64, had one unexpected failure, a restricted-join power-level latency race. The gate for 1.0.0 was a security audit instead: release-gate audit run 2 at commit 279d4a0b, From b47ff5ce2224f980499a79639124bb770ec9bc80 Mon Sep 17 00:00:00 2001 From: James Chapman Date: Fri, 13 Nov 2026 16:23:51 +0000 Subject: [PATCH 2/5] Name both waived releases in the release process and milestone Co-Authored-By: Claude Opus 5.5 --- docs/release-process.md | 10 +++++----- docs/todos/production-milestone.md | 2 +- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/release-process.md b/docs/release-process.md index f76f16983..1b1501b0d 100644 --- a/docs/release-process.md +++ b/docs/release-process.md @@ -185,11 +185,11 @@ sh scripts/check-release-readiness.sh ``` Then run Complement against the candidate (see "Complement" below). It must be green -before the tag is created. The one exception is 1.0.0: the owner waived this -requirement for that release only, because it was tagged without a green run -and the release-gate security audit was its gate instead -([ADR-0648](adr/0648-the-complement-blocker-is-waived-for-1-0-0-only.md)). The -owner waived it once more for 1.0.1, which fixes the vulnerabilities 1.0.0 ships +before the tag is created. The only exceptions are 1.0.0 and 1.0.1. The owner +waived the requirement for 1.0.0 because it was tagged without a green run and +the release-gate security audit was its gate instead +([ADR-0648](adr/0648-the-complement-blocker-is-waived-for-1-0-0-only.md)), and +for 1.0.1 because it fixes the vulnerabilities 1.0.0 ships ([ADR-0744](adr/0744-the-complement-blocker-is-also-waived-for-1-0-1.md)). The waiver does not apply to 1.0.2 or any later release. diff --git a/docs/todos/production-milestone.md b/docs/todos/production-milestone.md index f8f994ea0..ae3497da5 100644 --- a/docs/todos/production-milestone.md +++ b/docs/todos/production-milestone.md @@ -44,7 +44,7 @@ pinned commit against a package whose digest is in the maintainer-signed `SHA256 green run into a maintainer-signed record, tied to the shipped server binary's SHA-256 and attached to the release (ADR-0235, `docs/release-process.md`, "Complement"). Still open: the first green full run against a candidate, and the skip list it produces. The blocker -is waived for 1.0.0 only (see above); it still applies to later releases. Not yet verified: whether +is waived for 1.0.0 and 1.0.1 only (see above); it applies from 1.0.2 onward. Not yet verified: whether Docker's default seccomp profile admits Landlock, and whether the container's memory-lock limit fits the master key. From b9fe51af2b492612fa2d935e011711fbcdc7d23a Mon Sep 17 00:00:00 2001 From: James Chapman Date: Sun, 15 Nov 2026 22:48:16 +0000 Subject: [PATCH 3/5] Ask for a refused snapshot state event once per recovery, and do not re-ask a silent profile server for a minute (1.0.3) A 1.0.2 production log was about 95% backfill warnings from three retained PDUs: every /state_ids answer from one origin listed the same state event, which the origin refused with 403, so each recovery fetched and logged it once per snapshot target (15 times), and the deferred retry repeated that. - ADR-0760: BackfillBudget::unsupplied_events records state events a /state_ids snapshot listed and the origin did not supply. A later snapshot in the same recovery that lists one ends without a request (DEBUG line). Only the listed-event fetch records or consults it: the per-reference fetch and a snapshot's target fetch ask as before, since the target is fetched again once its state is known (ADR-0069, ADR-0070). - ADR-0761: a remote profile lookup whose destination gave no answer (no address, connection, TLS session or response) records the server in RecentlyUnreachableServers (60 s, at most 4096, under runtime.mutex); lookups for it are answered 502 at once until it expires. Any HTTP answer forgets it. OutboundCallResult::remote_unreachable carries the signal. - Version 1.0.3, CHANGELOG, debug-logging, http-transport, homeserver AGENTS.md A8 rule, ADR index. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 28 +++- ...n-for-a-listed-state-event-at-most-once.md | 91 +++++++++++++ ...-lookup-is-not-asked-again-for-a-minute.md | 79 +++++++++++ docs/adr/index.md | 4 +- docs/debug-logging.md | 16 +++ docs/http-transport.md | 2 +- .../homeserver/HomeserverRuntime.hpp | 4 + .../homeserver/OutboundCallResult.hpp | 4 + .../homeserver/RecentlyUnreachableServers.hpp | 47 +++++++ meson.build | 2 +- packaging/freebsd/+MANIFEST | 2 +- packaging/netbsd/Makefile | 2 +- packaging/opensuse/merovingian.spec | 5 +- packaging/rhel/merovingian.spec | 5 +- packaging/rpm/merovingian.spec | 5 +- scripts/build-deb.sh | 4 +- scripts/build-freebsd-pkg.sh | 4 +- scripts/build-netbsd-pkg.sh | 4 +- scripts/build-openbsd-pkg.sh | 4 +- scripts/build-opensuse-rpm.sh | 4 +- scripts/build-rhel-rpm.sh | 4 +- scripts/build-rpm.sh | 4 +- scripts/build-static-linux.sh | 2 +- src/db_migrate.cpp | 2 +- src/homeserver/AGENTS.md | 5 + src/homeserver/ClientServerRuntime.cpp | 21 ++- src/homeserver/LocalHttpRouter.cpp | 22 +++ src/homeserver/RecentlyUnreachableServers.cpp | 72 ++++++++++ src/homeserver/RoomService.cpp | 29 +++- src/homeserver/meson.build | 1 + src/main.cpp | 2 +- tests/integration/test_client_server_flow.cpp | 109 +++++++++++++++ .../test_pdu_ingestion_backfill_flow.cpp | 67 ++++++++++ tests/meson.build | 1 + ...omeserver_recently_unreachable_servers.cpp | 125 ++++++++++++++++++ 35 files changed, 751 insertions(+), 31 deletions(-) create mode 100644 docs/adr/0760-a-recovery-asks-the-origin-for-a-listed-state-event-at-most-once.md create mode 100644 docs/adr/0761-a-server-that-gave-no-answer-to-a-profile-lookup-is-not-asked-again-for-a-minute.md create mode 100644 include/merovingian/homeserver/RecentlyUnreachableServers.hpp create mode 100644 src/homeserver/RecentlyUnreachableServers.cpp create mode 100644 tests/unit/test_homeserver_recently_unreachable_servers.cpp diff --git a/CHANGELOG.md b/CHANGELOG.md index 0c71b98c6..a2c042d6c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,6 +3,31 @@ Changes from 1.0.0, the first production release, onwards; the format is set out Changes before 1.0.0 are in [CHANGELOG-pre-release.md](CHANGELOG-pre-release.md), which is no longer added to ([ADR-0740](docs/adr/0740-the-changelog-starts-at-1-0-0-and-the-pre-release-entries-are-a-frozen-record.md)). +# Version 1.0.3 + +## Features + +* `docs/release-process.md` and the production milestone record that 1.0.1, like 1.0.0, was tagged without a green + Complement run, and that the 1.0.0 GitHub release is withdrawn (ADR-0744). + +## Security Fixes + +* None. + +## Bug fixes + +* A backfill recovery asks the origin for a listed state event at most once: a later state snapshot listing one the + origin already refused ends without asking, so it is fetched and logged once, not once per gap (ADR-0760). +* A remote profile lookup for a server that just gave no answer is answered 502 at once for the next 60 s, so client + lookups to a dead server no longer each hold an outbound slot for the whole deadline (ADR-0761). + +## Release notes + +1.0.3 cuts repeated federation work and log noise seen on a production server: a refused history event is asked for +once per recovery, and a server that gives no answer to profile lookups is left alone for a minute. No configuration +change is needed. 1.0.1 was also tagged without a green Complement run, and the 1.0.0 GitHub release is withdrawn +because it ships the vulnerabilities 1.0.1 fixes; a 1.0.0 server should upgrade. + # Version 1.0.2 ## Features @@ -192,8 +217,7 @@ high, all availability) and hardens two of the four findings that needed validat `federation.worker.database_uri_file` and `federation.worker.allow_shared_database_credentials` are now refused at start-up, so remove them from a PostgreSQL deployment first; the worker's database role can then be dropped (ADR-0736). The changelog now starts at 1.0.0 and uses the format in `docs/versioning.md` (ADR-0740, ADR-0741). A -tag now ships exactly the `latest` package set (ADR-0742). Like 1.0.0, 1.0.1 was tagged without a green Complement -run (ADR-0744). The 1.0.0 GitHub release is withdrawn because it ships the vulnerabilities fixed here; upgrade to 1.0.1. +tag now ships exactly the `latest` package set (ADR-0742). # Version 1.0.0 diff --git a/docs/adr/0760-a-recovery-asks-the-origin-for-a-listed-state-event-at-most-once.md b/docs/adr/0760-a-recovery-asks-the-origin-for-a-listed-state-event-at-most-once.md new file mode 100644 index 000000000..727790b58 --- /dev/null +++ b/docs/adr/0760-a-recovery-asks-the-origin-for-a-listed-state-event-at-most-once.md @@ -0,0 +1,91 @@ +# A recovery asks the origin for a listed state event at most once + +* Status: accepted +* Deciders: James Chapman +* Date: 2026-11-15 + +Technical Story: production log of pong.ping.me.uk at 1.0.2, 2026-11-15: about 95% of a day's warnings came from +three retained PDUs in one room. + +## Context and Problem Statement + +`backfill_missing_pdu_references` recovers the history a PDU lacks. When `/get_missing_events` and `/event` leave a +gap, it asks `GET /state_ids` at up to `backfill.max_outbound_calls` gap events (`state_snapshot_targets`), and each +snapshot fetches every state event it lists with `GET /event` before it stores any. One listed event the origin will +not supply ends that snapshot. + +In the log, matrix.kahlilia.org answered every `/state_ids` request, and every answer listed the same state event, +which it then refused with 403 on `/event`, most likely because this server had not joined when the event was sent. +Each recovery therefore asked for that one event 15 times, once per snapshot target, and logged two warnings each +time. The deferred PDU retry (30 s doubling to 1 h, 10 attempts) repeated the whole recovery. + +Should a recovery ask the origin again for a listed state event it has already failed to supply? + +## Decision Drivers + +* The origin decides what it serves. A refusal, a 404 or an unusable answer for one event does not change within + one recovery, which is bounded by `backfill.timeout_seconds` (45 s by default). +* Every snapshot needs every state event it lists, so a snapshot listing one that was not supplied cannot be + completed, whatever else it lists. +* The repeat is per snapshot target: the per-reference stage asks for each missing reference once, and each snapshot + asks for its own target once, so neither multiplies with the number of gaps. +* The history may become available later: the origin may fetch it, or this server may gain the right to see it. + The deferred retry exists for that. + +## Considered Options + +* Remember, for the rest of one recovery, each state event a `/state_ids` snapshot listed and the origin did not + supply; a later snapshot that lists one ends at once without asking. +* Remember every event ID any `GET /event` of the recovery did not supply, and send no further request for it. +* Stop trying further snapshot targets after the first snapshot fails. +* Remember unsupplied events across recoveries, for example in the retained PDU's row. + +## Decision Outcome + +Chosen option: the first. `BackfillBudget::unsupplied_events` holds the IDs. `backfill_state_ids_snapshot` records a +listed state event whenever its fetch brought back no usable PDU (refused, missing, another event, or no answer), +and ends a later snapshot that lists a recorded ID without a request, logging at DEBUG, since the failure itself was +logged at WARNING when it happened. `fetch_event_by_id` and `fetch_snapshot_event_by_id` neither record nor consult +the set. The set belongs to one `BackfillBudget`, so it never outlives the recovery, and the next deferred retry asks +again. + +`/state_ids` itself is still asked at every target: each target is a different question, and its answer may list +only events the origin does serve. + +### Positive Consequences + +* A recovery whose snapshots all list one refused event sends one `/event` request for it, not one per target, and + logs its failure once. In the observed case that is 15 requests and 30 warnings down to 1 and 2. +* The calls saved are left in the recovery's budget for targets that can still succeed. +* A snapshot's target is still fetched after its state is known even when the per-reference stage could not get it + (ADR-0069, ADR-0070). + +### Negative Consequences + +* A listed state event whose fetch failed for a transient reason, such as a timeout, is not retried within the same + recovery. The deferred retry still asks again, as it did before. + +## Pros and Cons of the Options + +### Remember every unsupplied event + +* Good, because it would also save the target fetch after a per-reference fetch of the same event failed. +* Bad, because that saves at most one request per gap event, and it takes away the second ask that the snapshot path + relies on: an event the per-reference stage could not get is fetched again as the snapshot's target once its + state is known. The ADR-0069 and ADR-0070 scenarios fail with it. + +### Stop after the first failed snapshot + +* Good, because it saves the `/state_ids` requests too. +* Bad, because the other targets are different gaps, and their snapshots may list only events the origin serves. + +### Remember across recoveries + +* Good, because the retries would also skip the request. +* Bad, because it would refuse history that has become available since, and it needs persistent state that the + deferred retry's own limits already bound. + +## Links + +* Refines [ADR-0069](0069-spec-conformant-state-ids-fallback-via-event-auth.md) and the A8 recovery rules in + `src/homeserver/AGENTS.md` diff --git a/docs/adr/0761-a-server-that-gave-no-answer-to-a-profile-lookup-is-not-asked-again-for-a-minute.md b/docs/adr/0761-a-server-that-gave-no-answer-to-a-profile-lookup-is-not-asked-again-for-a-minute.md new file mode 100644 index 000000000..91bda62ab --- /dev/null +++ b/docs/adr/0761-a-server-that-gave-no-answer-to-a-profile-lookup-is-not-asked-again-for-a-minute.md @@ -0,0 +1,79 @@ +# A server that gave no answer to a profile lookup is not asked again for a minute + +* Status: accepted +* Deciders: James Chapman +* Date: 2026-11-15 + +Technical Story: production log of pong.ping.me.uk at 1.0.2, 2026-11-15: eight `profile.proxy` timeouts to +example.org in two minutes. + +## Context and Problem Statement + +`GET /_matrix/client/v3/profile/{userId}` for a user of another server asks that server's `query/profile`. The call +holds a client outbound slot (ADR-0079, ADR-0751) and runs under the directory deadline. Clients make these lookups +by themselves, for member lists, hover cards and invite dialogs, and retry them. In the log, a client looked up a +user on example.org, which runs no Matrix server: discovery fell back to port 8448, every connection timed out, and +each of eight lookups in two minutes waited out the deadline and logged a warning. + +The server discovery cache does not help: discovery succeeded, and only the connection failed. Should a server that +has just given no answer be asked again on the next lookup? + +## Decision Drivers + +* A lookup is interactive and cheap to repeat. A short delay before a recovered server is asked again costs a + client one missing display name, while a dead server costs a slot and the full deadline per lookup. +* Only the destination's silence should count. An HTTP answer of any status proves the server is there, and a + refusal made locally (federation policy, deadline, missing key) says nothing about it. +* The record must stay bounded, since a client chooses which servers are looked up. +* ADR-0262 and ADR-0749 keep joins and outbound delivery from serving remembered failures. Those retries are paced + by their own backoff or by the user; a profile lookup has neither. + +## Considered Options + +* Remember each server that gave no answer to a profile lookup for 60 seconds, in a bounded record guarded by + `runtime.mutex`, and answer lookups for it 502 at once during that time. +* Serve remembered discovery failures to profile lookups, as key resolution does. +* Shorten the directory deadline for profile lookups. + +## Decision Outcome + +Chosen option: the first. `perform_outbound_call` sets `OutboundCallResult::remote_unreachable` when discovery +found no server or the request got no connection, no verified TLS session or no response in time +(`unresolved_host`, `connection_failed`, `tls_verification_failed`, `timeout`, `network_error`). A redirect, an +oversized body and an error status are answers. `remote_profile_response` records such a server in +`HomeserverRuntime::profile_unreachable_servers` (`RecentlyUnreachableServers`: 60 s, at most 4096 servers, expired +entries and then the oldest dropped when full). While it is recorded, a lookup for any of its users is answered 502 +`M_UNKNOWN` with no outbound call, logged at DEBUG. A server that answers at all is forgotten. + +The record covers profile lookups only. Other client-triggered calls (public room directory, thirdparty) are made +when a user asks for them, not automatically, so they keep asking every time. + +### Positive Consequences + +* A dead server costs one slot and one deadline per minute of lookups, not one per lookup, and logs one warning a + minute. +* Slots stay free for lookups to servers that do answer. + +### Negative Consequences + +* A server that comes back is not asked for up to 60 seconds after its last failure. +* The record is in memory, per process, and is lost on restart. + +## Pros and Cons of the Options + +### Serve remembered discovery failures + +* Good, because the cache already exists. +* Bad, because the observed case failed after discovery succeeded, so it would not have been covered. + +### A shorter deadline for profile lookups + +* Good, because it needs no state. +* Bad, because a slow but working server would then fail every lookup, and a dead one still costs a slot per lookup. + +## Links + +* Bounded by [ADR-0079](0079-client-triggered-outbound-proxying-runs-under-a-bounded-in-flight-budget.md) and + [ADR-0751](0751-the-per-client-outbound-call-cap-is-configurable-and-defaults-to-three.md) +* Contrasts with [ADR-0262](0262-a-join-reuses-server-discovery-but-never-a-remembered-failure.md) and + [ADR-0749](0749-outbound-delivery-never-serves-a-remembered-discovery-failure.md) diff --git a/docs/adr/index.md b/docs/adr/index.md index 959e952ee..f8e74386c 100644 --- a/docs/adr/index.md +++ b/docs/adr/index.md @@ -626,6 +626,7 @@ * [0741 - Each changelog version has Features, Security Fixes, Bug fixes and Release notes](0741-each-changelog-version-has-features-security-fixes-bug-fixes-and-release-notes.md) * [0742 - Fixed build-and-test and package platform lists, and a tag ships what latest ships](0742-build-test-and-package-platform-lists-and-a-tag-ships-what-latest-ships.md) * [0743 - Analyses, then build and test jobs, then packages take runners](0743-analyses-then-build-and-test-then-packages-take-runners.md) +* [0744 - The Complement blocker is also waived for 1.0.1](0744-the-complement-blocker-is-also-waived-for-1-0-1.md) * [0745 - A first join into a restricted room stores the resident's verified co-signature](0745-a-first-restricted-join-stores-the-residents-verified-co-signature.md) * [0746 - A threaded receipt may name any event whose relation chain reaches the thread root](0746-a-threaded-receipt-may-name-any-event-whose-relation-chain-reaches-the-root.md) * [0747 - A remote member may leave through a server whose local members have all left](0747-a-remote-member-may-leave-through-a-server-whose-local-members-have-left.md) @@ -641,4 +642,5 @@ * [0757 - Hardened BSD builds link -pie, and the ELF gate reads elftoolchain readelf](0757-hardened-bsd-builds-link-pie-and-the-elf-gate-reads-elftoolchain-readelf.md) * [0758 - Every worker shard reloads every changed room, and main tracks each shard's copy](0758-every-worker-shard-reloads-every-changed-room-and-main-tracks-each-shards-copy.md) * [0759 - Main builds make_leave and make_knock templates while the worker's reload of the room is pending](0759-main-builds-make-leave-and-make-knock-templates-while-the-workers-reload-is-pending.md) -* [0744 - The Complement blocker is also waived for 1.0.1](0744-the-complement-blocker-is-also-waived-for-1-0-1.md) +* [0760 - A recovery asks the origin for a listed state event at most once](0760-a-recovery-asks-the-origin-for-a-listed-state-event-at-most-once.md) +* [0761 - A server that gave no answer to a profile lookup is not asked again for a minute](0761-a-server-that-gave-no-answer-to-a-profile-lookup-is-not-asked-again-for-a-minute.md) diff --git a/docs/debug-logging.md b/docs/debug-logging.md index ab5e8199c..ed4a87451 100644 --- a/docs/debug-logging.md +++ b/docs/debug-logging.md @@ -38,6 +38,22 @@ when a gateway answered after more than half the total timeout and was counted a deadline and queued a pusher unattempted. `account.deactivated.pusher_removal_failed` (warning, `user_id`) is logged when a pusher of a deactivated account could not be deleted. +## Repeated remote failures logged once (1.0.3) + +Two remote failures are logged at warning when they happen and at debug when this +server declines to repeat them: + +- Backfill: `federation.backfill.snapshot_event_fetch_failed` (module `rooms`) and the + `backfill_state_ids_snapshot: a listed event could not be fetched` warning are + written once per event and recovery. A later state snapshot in the same recovery + that lists the same event ends without a request and logs the debug line + `a listed event was already not supplied in this recovery` (ADR-0760). The + deferred retry runs a new recovery, which asks, and warns, again. +- Remote profile lookups: `profile.proxy` with `http_status=0` is a warning when a + server gives no answer. For the next 60 s, lookups for that server's users are + answered 502 without a call and log the debug line `event=profile.proxy ... + reason=gave no answer recently` (ADR-0761). + ## Room creation failures `room.create.failed` (module `rooms`) is an error, not debug-only. It is diff --git a/docs/http-transport.md b/docs/http-transport.md index 5a19b8e01..9246fb383 100644 --- a/docs/http-transport.md +++ b/docs/http-transport.md @@ -1125,7 +1125,7 @@ The paths under the budget: | `POST /_matrix/client/v3/join/{roomIdOrAlias}` for a remote alias | federation `GET /query/directory`, 10 s directory deadline (F-17, ADR-0350) | | `POST .../join`, `POST .../knock` and `POST .../leave`, when this server must ask a resident | federation `make_*` and `send_*`: one slot for the whole exchange, taken at once, 429 when none is free (ADR-0622) | | `POST /_matrix/client/v3/keys/query` and `/keys/claim`, for a user of another server | federation `POST /user/keys/query` and `/user/keys/claim`: one slot and one 30 s deadline (`remote_keys_deadline_seconds`) for all of the request's remote servers, at most 16 distinct servers (`remote_keys_max_servers`); a server not asked is reported in `failures` and the local answer is still returned, so no slot is not a 429 (F-17, ADR-0350) | -| `GET /_matrix/client/v3/profile/{userId}[/{keyName}]` for a user of another server | federation `GET /query/profile`, 10 s directory deadline | +| `GET /_matrix/client/v3/profile/{userId}[/{keyName}]` for a user of another server | federation `GET /query/profile`, 10 s directory deadline; a server that gave no answer (no address, connection, TLS session or response) is answered 502 `M_UNKNOWN` without a call for 60 s after (`RecentlyUnreachableServers`, ADR-0761) | | `GET /_matrix/client/v1/rooms/{roomId}/hierarchy`, once a child room is not held locally | federation `GET /hierarchy/{roomId}` to the link's `via` servers: one slot and one 10 s deadline for the whole walk, at most 3 servers per room and 16 requests per walk; the slot is taken at the first such request and `429 M_LIMIT_EXCEEDED` answers the whole walk when it is refused | | `GET /_matrix/client/v1/room_summary/{roomIdOrAlias}?via=...`, for a room not held locally (with or without a token, ADR-0556) | federation `GET /hierarchy/{roomId}` to at most 3 of the `via` servers: one slot (never the waiting form; 429 `M_LIMIT_EXCEEDED` at once without one) and one 10 s deadline | | `GET /_matrix/client/v1/rooms/{roomId}/timestamp_to_event`, when the local answer is missing or sits at a gap (ADR-0299) | federation `GET /timestamp_to_event` and `GET /event` to at most 3 servers with joined members: one slot and one 10 s deadline; the request waits for the slot within that deadline (ADR-0306) | diff --git a/include/merovingian/homeserver/HomeserverRuntime.hpp b/include/merovingian/homeserver/HomeserverRuntime.hpp index 5bbe4f1bc..91acbe807 100644 --- a/include/merovingian/homeserver/HomeserverRuntime.hpp +++ b/include/merovingian/homeserver/HomeserverRuntime.hpp @@ -28,6 +28,7 @@ #include "merovingian/homeserver/LocalRoom.hpp" #include "merovingian/homeserver/MediaUploadNotifier.hpp" #include "merovingian/homeserver/PendingFederatedJoin.hpp" +#include "merovingian/homeserver/RecentlyUnreachableServers.hpp" #include "merovingian/homeserver/RemoteMediaFetchCoalescer.hpp" #include "merovingian/homeserver/RoomMembers.hpp" #include "merovingian/homeserver/RuntimeMutex.hpp" @@ -362,6 +363,9 @@ struct HomeserverRuntime final // start-up snapshot on SQLite) and its sandbox are not asked to decide it again. bool federation_policy_decided_upstream{false}; ClientOutboundProxyPolicy client_outbound_proxy_policy{default_client_outbound_proxy_policy()}; + // ADR-0761: servers that recently gave no answer to a client's remote profile lookup, so the lookups that follow + // are answered 502 at once instead of each waiting out the deadline. Guard every access with `mutex`. + RecentlyUnreachableServers profile_unreachable_servers{}; // One fetch from the origin per remote media file at a time (ADR-0121). Its // mutex is a leaf: never waited on with `mutex` held. Held through a // unique_ptr so leads keep a stable address across a runtime move. diff --git a/include/merovingian/homeserver/OutboundCallResult.hpp b/include/merovingian/homeserver/OutboundCallResult.hpp index 236d5e27d..0be799d2a 100644 --- a/include/merovingian/homeserver/OutboundCallResult.hpp +++ b/include/merovingian/homeserver/OutboundCallResult.hpp @@ -20,6 +20,10 @@ struct OutboundCallResult final // The `errcode` of a non-2xx answer's Matrix error body when it is `M_INVITE_BLOCKED` (SS7), else empty. A remote // body is untrusted, so only this one fixed value is carried, never the remote's text. std::string remote_errcode{}; + // ADR-0761: true when the destination itself gave no answer: discovery found no server, or the request got no + // connection, no verified TLS session or no response in time. False on success, on any HTTP answer (including a + // redirect or an oversized body) and on a refusal made here before anything was sent (policy, deadline, key). + bool remote_unreachable{false}; }; } // namespace merovingian::homeserver diff --git a/include/merovingian/homeserver/RecentlyUnreachableServers.hpp b/include/merovingian/homeserver/RecentlyUnreachableServers.hpp new file mode 100644 index 000000000..e3edfdede --- /dev/null +++ b/include/merovingian/homeserver/RecentlyUnreachableServers.hpp @@ -0,0 +1,47 @@ +// SPDX-FileCopyrightText: 2026 James Chapman +// SPDX-License-Identifier: GPL-3.0-or-later +#pragma once + +#include +#include +#include +#include +#include + +namespace merovingian::homeserver +{ + +// ADR-0761: servers that gave no answer to a client's remote profile lookup (no address, no connection, no TLS +// session or no response before the deadline), remembered for `ttl` so that the lookups which follow fail at once +// instead of each holding a client outbound slot for the whole deadline. A server that answers at all, even with an +// error, is forgotten. Holds at most `capacity` servers. Not thread-safe: the runtime guards it with runtime.mutex. +class RecentlyUnreachableServers final +{ +public: + using Clock = std::chrono::steady_clock; + + static constexpr auto default_ttl = std::chrono::seconds{60}; + static constexpr auto default_capacity = std::size_t{4096U}; + + RecentlyUnreachableServers() = default; + // `capacity` 0 remembers nothing. + RecentlyUnreachableServers(Clock::duration ttl, std::size_t capacity) noexcept; + + // Whether `server` gave no answer less than `ttl` before `now`. + [[nodiscard]] auto contains(std::string_view server, Clock::time_point now) const -> bool; + // Records that `server` gave no answer at `now`. When the set is full, expired entries are dropped first and then + // the server remembered longest, so a burst of distinct servers cannot grow it past `capacity`. + auto remember(std::string_view server, Clock::time_point now) -> void; + // Forgets `server`, which has answered. + auto forget(std::string_view server) -> void; + [[nodiscard]] auto size() const noexcept -> std::size_t; + +private: + auto make_room(Clock::time_point now) -> void; + + Clock::duration m_ttl{default_ttl}; + std::size_t m_capacity{default_capacity}; + std::unordered_map m_failed_at{}; +}; + +} // namespace merovingian::homeserver diff --git a/meson.build b/meson.build index 0f334b178..652b5b1fd 100644 --- a/meson.build +++ b/meson.build @@ -2,7 +2,7 @@ project( 'merovingian', 'cpp', 'c', - version: '1.0.2', + version: '1.0.3', license: 'GPL-3.0-or-later', meson_version: '>=1.1.0', default_options: [ diff --git a/packaging/freebsd/+MANIFEST b/packaging/freebsd/+MANIFEST index a1eabee9e..3d0de47ba 100644 --- a/packaging/freebsd/+MANIFEST +++ b/packaging/freebsd/+MANIFEST @@ -1,5 +1,5 @@ name: "merovingian" -version: "1.0.2" +version: "1.0.3" origin: net/merovingian comment: Secure Matrix Protocol homeserver desc: "Merovingian is a Matrix Protocol homeserver." diff --git a/packaging/netbsd/Makefile b/packaging/netbsd/Makefile index 313c53462..dd18b8ef4 100644 --- a/packaging/netbsd/Makefile +++ b/packaging/netbsd/Makefile @@ -1,4 +1,4 @@ -DISTNAME= merovingian-1.0.2 +DISTNAME= merovingian-1.0.3 CATEGORIES= net MASTER_SITES= ${MASTER_SITE_GITHUB:=OMG-Software/} GITHUB_PROJECT= Merovingian diff --git a/packaging/opensuse/merovingian.spec b/packaging/opensuse/merovingian.spec index b4ace1a06..109a27b42 100644 --- a/packaging/opensuse/merovingian.spec +++ b/packaging/opensuse/merovingian.spec @@ -1,5 +1,5 @@ Name: merovingian -Version: 1.0.2 +Version: 1.0.3 Release: 1%{?dist} Summary: Secure Matrix Protocol homeserver @@ -115,6 +115,9 @@ fi %{_sysconfdir}/merovingian/merovingian.conf.example %changelog +* Sun Nov 15 2026 James Chapman - 1.0.3-1 +- Merovingian 1.0.3 + * Sat Nov 14 2026 James Chapman - 1.0.2-1 - Merovingian 1.0.2 diff --git a/packaging/rhel/merovingian.spec b/packaging/rhel/merovingian.spec index 2c1ca4cb0..cd5ea6173 100644 --- a/packaging/rhel/merovingian.spec +++ b/packaging/rhel/merovingian.spec @@ -1,5 +1,5 @@ Name: merovingian -Version: 1.0.2 +Version: 1.0.3 Release: 1%{?dist} Summary: Secure Matrix Protocol homeserver @@ -100,6 +100,9 @@ fi %{_sysconfdir}/merovingian/merovingian.conf.example %changelog +* Sun Nov 15 2026 James Chapman - 1.0.3-1 +- Merovingian 1.0.3 + * Sat Nov 14 2026 James Chapman - 1.0.2-1 - Merovingian 1.0.2 diff --git a/packaging/rpm/merovingian.spec b/packaging/rpm/merovingian.spec index e86b3a071..effe5b678 100644 --- a/packaging/rpm/merovingian.spec +++ b/packaging/rpm/merovingian.spec @@ -1,5 +1,5 @@ Name: merovingian -Version: 1.0.2 +Version: 1.0.3 Release: 1%{?dist} Summary: Secure Matrix Protocol homeserver @@ -104,6 +104,9 @@ fi %{_sysconfdir}/merovingian/merovingian.conf.example %changelog +* Sun Nov 15 2026 James Chapman - 1.0.3-1 +- Merovingian 1.0.3 + * Sat Nov 14 2026 James Chapman - 1.0.2-1 - Merovingian 1.0.2 diff --git a/scripts/build-deb.sh b/scripts/build-deb.sh index 04f9ee573..bfc0bc5b5 100755 --- a/scripts/build-deb.sh +++ b/scripts/build-deb.sh @@ -1,9 +1,9 @@ #!/bin/sh # SPDX-License-Identifier: GPL-3.0-or-later -# Build for merovingian 1.0.2. +# Build for merovingian 1.0.3. set -e -VERSION="1.0.2" +VERSION="1.0.3" PKG_NAME="merovingian" STAGING="staging-deb" diff --git a/scripts/build-freebsd-pkg.sh b/scripts/build-freebsd-pkg.sh index 240e68bbe..4fce1078a 100755 --- a/scripts/build-freebsd-pkg.sh +++ b/scripts/build-freebsd-pkg.sh @@ -1,9 +1,9 @@ #!/bin/sh # SPDX-License-Identifier: GPL-3.0-or-later -# Build for merovingian 1.0.2. +# Build for merovingian 1.0.3. set -e -VERSION="1.0.2" +VERSION="1.0.3" STAGING="staging-fbsd" # Clean any state (staged files, build dir) from cached FreeBSD VM runs. diff --git a/scripts/build-netbsd-pkg.sh b/scripts/build-netbsd-pkg.sh index 701ebff62..37f24cedf 100644 --- a/scripts/build-netbsd-pkg.sh +++ b/scripts/build-netbsd-pkg.sh @@ -1,6 +1,6 @@ #!/bin/sh # SPDX-License-Identifier: GPL-3.0-or-later -# Build for merovingian 1.0.2. +# Build for merovingian 1.0.3. # # The checked-in packaging/netbsd/Makefile is the pkgsrc recipe kept for # downstream porters. This script generates a framework-free package so CI @@ -11,7 +11,7 @@ # a staged tree using tar, which produces a package that pkg_add can install. set -e -VERSION="1.0.2" +VERSION="1.0.3" STAGE="staging-netbsd" PREFIX=/usr/pkg diff --git a/scripts/build-openbsd-pkg.sh b/scripts/build-openbsd-pkg.sh index d76bf87ef..a8f791812 100644 --- a/scripts/build-openbsd-pkg.sh +++ b/scripts/build-openbsd-pkg.sh @@ -1,6 +1,6 @@ #!/bin/sh # SPDX-License-Identifier: GPL-3.0-or-later -# Build for merovingian 1.0.2. +# Build for merovingian 1.0.3. # # Standalone pkg_create(1), no ports tree. The checked-in packaging/openbsd/PLIST # is the ports-framework packing list kept for downstream porters; this script @@ -8,7 +8,7 @@ # produce and validate an installable package on every run. set -e -VERSION="1.0.2" +VERSION="1.0.3" STAGE="staging-openbsd" PREFIX=/usr/local diff --git a/scripts/build-opensuse-rpm.sh b/scripts/build-opensuse-rpm.sh index fd29b83a6..c3f3dac99 100644 --- a/scripts/build-opensuse-rpm.sh +++ b/scripts/build-opensuse-rpm.sh @@ -1,6 +1,6 @@ #!/bin/sh # SPDX-License-Identifier: GPL-3.0-or-later -# Build an RPM package for merovingian 1.0.2 on OpenSUSE Tumbleweed or Leap. +# Build an RPM package for merovingian 1.0.3 on OpenSUSE Tumbleweed or Leap. # # Uses packaging/opensuse/merovingian.spec which adapts BuildRequires to # OpenSUSE package names (libopenssl-devel, postgresql-devel, libpng16-devel, @@ -12,7 +12,7 @@ # (.leap16, ADR-0650). set -e -VERSION="1.0.2" +VERSION="1.0.3" DIST="${MEROVINGIAN_RPM_DIST:-.tumbleweed}" case "${DIST}" in .[a-z]*) ;; diff --git a/scripts/build-rhel-rpm.sh b/scripts/build-rhel-rpm.sh index 977c66110..27ce4c891 100644 --- a/scripts/build-rhel-rpm.sh +++ b/scripts/build-rhel-rpm.sh @@ -1,13 +1,13 @@ #!/bin/sh # SPDX-License-Identifier: GPL-3.0-or-later -# Build an RPM package for merovingian 1.0.2 targeting RHEL 10 / AlmaLinux 10. +# Build an RPM package for merovingian 1.0.3 targeting RHEL 10 / AlmaLinux 10. # # Differs from build-rpm.sh only in the spec file used: packaging/rhel/merovingian.spec # omits catch-devel from BuildRequires (Catch2 is not reliably available in EPEL 10 # and is not needed when -Dbuild_tests=false). set -e -VERSION="1.0.2" +VERSION="1.0.3" mkdir -p "${HOME}/rpmbuild/BUILD" mkdir -p "${HOME}/rpmbuild/BUILDROOT" diff --git a/scripts/build-rpm.sh b/scripts/build-rpm.sh index e23336ea5..c6a63b479 100755 --- a/scripts/build-rpm.sh +++ b/scripts/build-rpm.sh @@ -1,9 +1,9 @@ #!/bin/sh # SPDX-License-Identifier: GPL-3.0-or-later -# Build an RPM package for merovingian 1.0.2 using rpmbuild. +# Build an RPM package for merovingian 1.0.3 using rpmbuild. set -e -VERSION="1.0.2" +VERSION="1.0.3" # 1. Create rpmbuild directory tree mkdir -p "${HOME}/rpmbuild/BUILD" diff --git a/scripts/build-static-linux.sh b/scripts/build-static-linux.sh index 2f9620722..934084172 100755 --- a/scripts/build-static-linux.sh +++ b/scripts/build-static-linux.sh @@ -3,7 +3,7 @@ # Build a portable Linux fallback tarball with musl-linked static PIE binaries. set -eu -VERSION="${MEROVINGIAN_VERSION:-1.0.2}" +VERSION="${MEROVINGIAN_VERSION:-1.0.3}" BUILD_DIR="${BUILD_DIR:-build-static-linux}" STAGING="staging-static-linux" PACKAGE_ROOT="merovingian-${VERSION}-linux-static-x86_64" diff --git a/src/db_migrate.cpp b/src/db_migrate.cpp index a55a54ff3..776d6002c 100644 --- a/src/db_migrate.cpp +++ b/src/db_migrate.cpp @@ -23,7 +23,7 @@ namespace { -constexpr auto version = std::string_view{"1.0.2"}; +constexpr auto version = std::string_view{"1.0.3"}; auto print_help() -> void { diff --git a/src/homeserver/AGENTS.md b/src/homeserver/AGENTS.md index e1c7c7d84..e8fdc349f 100644 --- a/src/homeserver/AGENTS.md +++ b/src/homeserver/AGENTS.md @@ -612,6 +612,11 @@ only endpoints the spec requires and the origin may be asked for: supply, never at a prev_event that was already fetched; held PDUs are placed oldest first afterwards. - A snapshot fetches every missing named event before it stores any, so an auth chain the origin cannot supply leaves nothing stored. +- A recovery asks the origin for a listed state event at most once (ADR-0760). A state event a `/state_ids` snapshot + listed and the origin did not supply is kept in `BackfillBudget::unsupplied_events`, and a later snapshot that lists + it ends without a request. Only `backfill_state_ids_snapshot`'s listed-event fetch records or consults the set: the + per-reference fetch and a snapshot's target fetch ask as before, since the target is fetched again once its state is + known (ADR-0069). The set lives only as long as the recovery's `BackfillBudget`; the deferred retry asks again. - `verify_and_store_backfilled_event` is called with `runtime.mutex` released. It takes the room stripe and then `runtime.mutex` itself once its last signature check (which may fetch a key) is done, and holds them through the authorization reads and the write. Never read the store in it before that point. diff --git a/src/homeserver/ClientServerRuntime.cpp b/src/homeserver/ClientServerRuntime.cpp index 19fcb4ae7..27aa96f9f 100644 --- a/src/homeserver/ClientServerRuntime.cpp +++ b/src/homeserver/ClientServerRuntime.cpp @@ -107,6 +107,7 @@ #include "merovingian/homeserver/PresenceService.hpp" #include "merovingian/homeserver/PublicRooms.hpp" #include "merovingian/homeserver/PushRuleUnreadClassifier.hpp" +#include "merovingian/homeserver/RecentlyUnreachableServers.hpp" #include "merovingian/homeserver/RegistrationValidationSession.hpp" #include "merovingian/homeserver/RemoteDeviceListScope.hpp" #include "merovingian/homeserver/RemoteInviteResult.hpp" @@ -12381,6 +12382,15 @@ namespace { return dispatch_err(req, rt, 404U, "M_NOT_FOUND", "profile field not found"); } + auto const destination = std::string{server_name_from_user_id(target_user)}; + // ADR-0761: a server that gave no answer moments ago is not asked again until the record expires. Clients + // look profiles up by themselves (member lists, invite dialogs), so each would otherwise wait out the deadline. + if (rt.homeserver.profile_unreachable_servers.contains(destination, RecentlyUnreachableServers::Clock::now())) + { + LOG_DEBUG("event=profile.proxy destination=" + destination + + " reason=gave no answer recently; not asked again yet"); + return dispatch_err(req, rt, 502U, "M_UNKNOWN", "The remote server did not answer a recent request"); + } auto const proxy_deadline = effective_client_outbound_deadline( rt.homeserver, rt.homeserver.client_outbound_proxy_policy.directory_deadline_seconds); wire_federation_callbacks(rt.homeserver); @@ -12394,8 +12404,7 @@ namespace { target += "&field=" + std::string{field}; } - auto const tx = federation::make_outbound_transaction(std::string{server_name_from_user_id(target_user)}, "GET", - target, our_server, {}); + auto const tx = federation::make_outbound_transaction(destination, "GET", target, our_server, {}); auto const result = [&] { // Re-acquire before returning: dispatch_resp/dispatch_err read reloadable runtime state. auto const released = merovingian::homeserver::RuntimeLockRelease{guard}; @@ -12403,6 +12412,14 @@ namespace key_id, std::move(secret), "profile.proxy", proxy_deadline); }(); + if (result.remote_unreachable) + { + rt.homeserver.profile_unreachable_servers.remember(destination, RecentlyUnreachableServers::Clock::now()); + } + else if (result.ok || result.remote_status != 0U) + { + rt.homeserver.profile_unreachable_servers.forget(destination); + } if (result.remote_status == 404U) { return dispatch_err(req, rt, 404U, "M_NOT_FOUND", "profile not found"); diff --git a/src/homeserver/LocalHttpRouter.cpp b/src/homeserver/LocalHttpRouter.cpp index cc43500fb..f650e053e 100644 --- a/src/homeserver/LocalHttpRouter.cpp +++ b/src/homeserver/LocalHttpRouter.cpp @@ -3651,6 +3651,17 @@ namespace std::size_t max_calls{}; std::chrono::steady_clock::time_point deadline{}; bool key_resolution_locally_denied{}; + // ADR-0760: state events a /state_ids snapshot listed that the origin, asked with GET /event during this + // recovery, did not supply (refused, missing, unusable or no answer). A later snapshot in the same recovery + // that lists one of them cannot be completed, so it ends without asking again. Only listed state events are + // recorded: the per-reference fetch and a snapshot's own target are asked for separately, as before. Never + // outlives the recovery; the next deferred retry asks again. + std::unordered_set unsupplied_events{}; + + [[nodiscard]] auto origin_did_not_supply(std::string_view event_id) const -> bool + { + return unsupplied_events.contains(std::string{event_id}); + } [[nodiscard]] auto within_deadline() const noexcept -> bool { @@ -4792,10 +4803,21 @@ namespace { return false; } + // ADR-0760: the origin already failed to supply this event during this recovery and its answer + // was logged then, so this snapshot cannot be completed and the event is not asked for again. + if (budget.origin_did_not_supply(id)) + { + LOG_DEBUG("backfill_state_ids_snapshot: a listed event was already not supplied in this " + "recovery; room_id=" + + std::string{room_id} + " target_event_id=" + std::string{target_event_id} + + " event_id=" + id); + return false; + } auto supplied = fetch_snapshot_event_by_id(runtime, room_id, origin, id, policy, snapshot_calls, budget); if (!supplied.has_value()) { + budget.unsupplied_events.emplace(id); LOG_WARNING("backfill_state_ids_snapshot: a listed event could not be fetched with /event; " "room_id=" + std::string{room_id} + " target_event_id=" + std::string{target_event_id} + diff --git a/src/homeserver/RecentlyUnreachableServers.cpp b/src/homeserver/RecentlyUnreachableServers.cpp new file mode 100644 index 000000000..e1484de21 --- /dev/null +++ b/src/homeserver/RecentlyUnreachableServers.cpp @@ -0,0 +1,72 @@ +// SPDX-FileCopyrightText: 2026 James Chapman +// SPDX-License-Identifier: GPL-3.0-or-later + +#include "merovingian/homeserver/RecentlyUnreachableServers.hpp" + +#include +#include +#include + +namespace merovingian::homeserver +{ + +RecentlyUnreachableServers::RecentlyUnreachableServers(Clock::duration ttl, std::size_t capacity) noexcept + : m_ttl{ttl} + , m_capacity{capacity} +{ +} + +auto RecentlyUnreachableServers::contains(std::string_view server, Clock::time_point now) const -> bool +{ + auto const found = m_failed_at.find(std::string{server}); + return found != m_failed_at.end() && now - found->second < m_ttl; +} + +auto RecentlyUnreachableServers::remember(std::string_view server, Clock::time_point now) -> void +{ + if (m_capacity == 0U) + { + return; + } + auto key = std::string{server}; + if (auto const found = m_failed_at.find(key); found != m_failed_at.end()) + { + found->second = now; + return; + } + make_room(now); + m_failed_at.emplace(std::move(key), now); +} + +auto RecentlyUnreachableServers::forget(std::string_view server) -> void +{ + m_failed_at.erase(std::string{server}); +} + +auto RecentlyUnreachableServers::size() const noexcept -> std::size_t +{ + return m_failed_at.size(); +} + +// Leaves room for one more entry. The scan is linear in `capacity`; it runs only on a lookup that has already spent +// a client outbound slot and failed, so it is bounded by that admission. +auto RecentlyUnreachableServers::make_room(Clock::time_point now) -> void +{ + if (m_failed_at.size() < m_capacity) + { + return; + } + std::erase_if(m_failed_at, [&](auto const& entry) { + return now - entry.second >= m_ttl; + }); + if (m_failed_at.size() < m_capacity) + { + return; + } + auto const oldest = std::ranges::min_element(m_failed_at, {}, [](auto const& entry) { + return entry.second; + }); + m_failed_at.erase(oldest); +} + +} // namespace merovingian::homeserver diff --git a/src/homeserver/RoomService.cpp b/src/homeserver/RoomService.cpp index 324cbbac4..e849bb757 100644 --- a/src/homeserver/RoomService.cpp +++ b/src/homeserver/RoomService.cpp @@ -2828,6 +2828,30 @@ namespace return federation::pdu_format_violation(canonicaljson::Value{completed_event}, policy); } + // ADR-0761: the transport failures in which the destination gave no answer. A redirect or an oversized body is + // an answer, and a request refused before any I/O (bad URL, method or header) says nothing about the destination. + [[nodiscard]] auto destination_gave_no_answer(http::OutboundError error) noexcept -> bool + { + switch (error) + { + case http::OutboundError::unresolved_host: + case http::OutboundError::tls_verification_failed: + case http::OutboundError::connection_failed: + case http::OutboundError::timeout: + case http::OutboundError::network_error: + return true; + case http::OutboundError::none: + case http::OutboundError::invalid_url: + case http::OutboundError::invalid_method: + case http::OutboundError::https_required: + case http::OutboundError::redirect_rejected: + case http::OutboundError::response_too_large: + case http::OutboundError::invalid_header: + return false; + } + return false; + } + } // namespace [[nodiscard]] auto perform_outbound_call(HomeserverRuntime& runtime, OutboundAnswerTrust trust, @@ -2912,7 +2936,7 @@ namespace {"destination", transaction.destination, false}, {"reason", "server discovery failed", false} }); - return {false, "server discovery failed"}; + return {false, "server discovery failed", 0U, {}, true}; } resolved_host = resolution.resolved_host; tls_server_name = resolution.tls_server_name; @@ -3019,7 +3043,8 @@ namespace outcome.error_detail.empty() ? "remote server returned " + std::to_string(outcome.response.status) : outcome.error_detail, outcome.ok ? outcome.response.status : std::uint16_t{0U}, - outcome.ok ? remote_invite_blocked_errcode(outcome.response.body) : std::string{}}; + outcome.ok ? remote_invite_blocked_errcode(outcome.response.body) : std::string{}, + !outcome.ok && destination_gave_no_answer(outcome.error)}; } return {true, outcome.response.body, outcome.response.status}; } diff --git a/src/homeserver/meson.build b/src/homeserver/meson.build index 13adb22a9..336f973ac 100644 --- a/src/homeserver/meson.build +++ b/src/homeserver/meson.build @@ -50,6 +50,7 @@ homeserver_sources = files( 'StoredRoomVersion.cpp', 'StreamTokenEpoch.cpp', 'RemoteMediaFetchScope.cpp', + 'RecentlyUnreachableServers.cpp', 'RequestLockScope.cpp', 'RuntimeLockRelease.cpp', 'RestrictedJoin.cpp', diff --git a/src/main.cpp b/src/main.cpp index 54ee2ee72..9d4d735a9 100644 --- a/src/main.cpp +++ b/src/main.cpp @@ -81,7 +81,7 @@ namespace { -constexpr auto version = std::string_view{"1.0.2"}; +constexpr auto version = std::string_view{"1.0.3"}; struct BootstrapConfigResult final { diff --git a/tests/integration/test_client_server_flow.cpp b/tests/integration/test_client_server_flow.cpp index 600dfcb8a..919bb5045 100644 --- a/tests/integration/test_client_server_flow.cpp +++ b/tests/integration/test_client_server_flow.cpp @@ -14,6 +14,7 @@ #include "merovingian/config/LogModulesConfig.hpp" #include "merovingian/config/SecurityConfig.hpp" #include "merovingian/config/ServerConfig.hpp" +#include "merovingian/core/SocketHandle.hpp" #include "merovingian/database/ExaminedRows.hpp" #include "merovingian/database/PersistentMembership.hpp" #include "merovingian/database/PersistentStateEvent.hpp" @@ -43,6 +44,7 @@ #include #include +#include #include #include #include @@ -1129,6 +1131,113 @@ SCENARIO("GET /profile/{userId} for a user of another server asks that server ov } } +// ADR-0761: clients look remote profiles up by themselves, from member lists and invite dialogs, so a server that +// gives no answer would otherwise cost one outbound slot and the whole deadline per lookup. Once it has given no +// answer it is not asked again for a minute; a server that answers, even 404, is asked every time. +SCENARIO("GET /profile/{userId} does not ask again a server that just gave no answer", + "[homeserver][client-server][profile][fed-endpoints][adr0761]") +{ + GIVEN("a remote server that accepts each connection and closes it without a TLS session or an answer") + { + auto const config = registration_enabled_config(); + auto started = merovingian::homeserver::start_client_server(config); + REQUIRE(started.started); + auto& rt = started.runtime; + + auto certificate = merovingian::tests::tls_mock::write_test_tls_certificate("localhost"); + auto acceptor = merovingian::net::TcpAcceptor{}; + REQUIRE(acceptor.bind("127.0.0.1", 0U).ok); + rt.homeserver.test_forced_outbound_resolution["silent.example.net"] = + merovingian::homeserver::TestOnlyForcedOutboundResolution{ + "localhost", acceptor.bound_port(), {"127.0.0.1"}, certificate.certificate_pem}; + + auto connections = std::atomic{0U}; + auto stop = std::atomic{false}; + auto server_thread = std::thread{[&]() { + while (!stop.load()) + { + auto const fd = merovingian::tests::tls_mock::accept_loopback(acceptor, 50); + if (fd >= 0) + { + auto const closed_at_once = merovingian::core::SocketHandle{fd}; + connections.fetch_add(1U); + } + } + }}; + auto const join_server = merovingian::tests::tls_mock::ScopedThreadJoin{server_thread}; + struct StopOnExit final + { + std::atomic& flag; + ~StopOnExit() noexcept + { + flag.store(true); + } + }; + auto const stop_on_exit = StopOnExit{stop}; + + WHEN("a client reads profiles of that server's users twice in a row") + { + auto const first = merovingian::homeserver::handle_client_server_request( + rt, {"GET", "/_matrix/client/v3/profile/%40bob%3Asilent.example.net", {}, {}}); + auto const second = merovingian::homeserver::handle_client_server_request( + rt, {"GET", "/_matrix/client/v3/profile/%40carol%3Asilent.example.net/displayname", {}, {}}); + + THEN("both are answered 502, and only the first reached the server") + { + REQUIRE(first.response.status == 502U); + REQUIRE(second.response.status == 502U); + REQUIRE(second.response.body.find("M_UNKNOWN") != std::string::npos); + REQUIRE(connections.load() == 1U); + } + } + } + + GIVEN("a remote server that answers each profile query with 404") + { + auto const config = registration_enabled_config(); + auto started = merovingian::homeserver::start_client_server(config); + REQUIRE(started.started); + auto& rt = started.runtime; + + auto certificate = merovingian::tests::tls_mock::write_test_tls_certificate("localhost"); + auto tls = merovingian::homeserver::make_tls_server_context(certificate.certificate_file, + certificate.private_key_file); + REQUIRE(tls.ok()); + auto tls_context = std::move(*tls.context); + auto acceptor = merovingian::net::TcpAcceptor{}; + REQUIRE(acceptor.bind("127.0.0.1", 0U).ok); + rt.homeserver.test_forced_outbound_resolution["remote.example.net"] = + merovingian::homeserver::TestOnlyForcedOutboundResolution{ + "localhost", acceptor.bound_port(), {"127.0.0.1"}, certificate.certificate_pem}; + auto const not_found = merovingian::tests::tls_mock::json_http_response( + "404 Not Found", R"({"errcode":"M_NOT_FOUND","error":"no such user"})"); + auto const responses = std::vector>{ + {"user_id=%40nobody%3Aremote.example.net", not_found}, + {"user_id=%40nobody%3Aremote.example.net", not_found}, + }; + auto captured = std::vector{}; + auto server_thread = std::thread{[&]() { + merovingian::tests::tls_mock::run_path_dispatch_tls_server(acceptor, tls_context, responses, &captured); + }}; + auto const join_server = merovingian::tests::tls_mock::ScopedThreadJoin{server_thread}; + + WHEN("a client reads the same missing profile twice") + { + auto const first = merovingian::homeserver::handle_client_server_request( + rt, {"GET", "/_matrix/client/v3/profile/%40nobody%3Aremote.example.net", {}, {}}); + auto const second = merovingian::homeserver::handle_client_server_request( + rt, {"GET", "/_matrix/client/v3/profile/%40nobody%3Aremote.example.net", {}, {}}); + + THEN("both are 404 and both were asked of the server") + { + REQUIRE(first.response.status == 404U); + REQUIRE(second.response.status == 404U); + REQUIRE(captured.size() == 2U); + } + } + } +} + // Spec: Matrix Client-Server API v1.19, GET /profile/{userId}; Server-Server API v1.19, query/profile // ("Additional profile fields"). A remote answer is untrusted: only m.tz as a string and custom keys that meet // the key grammar are relayed, never null, and nothing else. diff --git a/tests/integration/test_pdu_ingestion_backfill_flow.cpp b/tests/integration/test_pdu_ingestion_backfill_flow.cpp index 1112d4270..0e63fd524 100644 --- a/tests/integration/test_pdu_ingestion_backfill_flow.cpp +++ b/tests/integration/test_pdu_ingestion_backfill_flow.cpp @@ -4078,6 +4078,73 @@ SCENARIO("A snapshot whose auth chain the origin will not supply stores nothing } } +[[nodiscard]] auto event_fetches_of(std::vector const& requests, std::string const& event_id) + -> std::size_t +{ + return count_requests_starting_with(requests, "GET /_matrix/federation/v1/event/" + + core::percent_encode_path_component(event_id) + " "); +} + +// Spec: server-server-api.md GET /state_ids and GET /event/{eventId}. A snapshot is usable only when every state event +// it lists can be fetched, and the origin decides what it serves. ADR-0760: once the origin has not supplied a listed +// state event, asking it again in the same recovery only repeats the refusal, so every later snapshot that lists the +// event ends without the request. A later recovery (the deferred retry) asks again. +SCENARIO("A recovery asks the origin for a listed state event it did not supply at most once", + "[pdu_ingestion][backfill][adr0760]") +{ + GIVEN("a PDU with two unknown prev_events, whose /state_ids answers both name a state event the origin refuses") + { + auto fixture = RecoveryFixture{"!adr0760-refused:local.example.org"}; + auto const& room = fixture.room_id; + auto const first_gap = make_remote_message_pdu(room, {fixture.bob_member_id()}, fixture.auth_ids(), 4, 10); + auto const second_gap = make_remote_message_pdu(room, {fixture.bob_member_id()}, fixture.auth_ids(), 4, 11); + auto const refused = + make_remote_event_pdu(room, "m.room.member", std::string{remote_member}, remote_member, member_content("R"), + {fixture.bob_member_id()}, fixture.auth_ids(), 4, 12); + auto const pdu = + make_remote_message_pdu(room, {first_gap.event_id, second_gap.event_id}, fixture.auth_ids(), 5, 20); + + auto pdu_ids = fixture.genesis_state_ids(); + pdu_ids.push_back(refused.event_id); + auto script = PeerScript{}; + script.state_ids[first_gap.event_id] = make_state_ids_response(pdu_ids, fixture.auth_ids()); + script.state_ids[second_gap.event_id] = make_state_ids_response(pdu_ids, fixture.auth_ids()); + auto const peer = fixture.serve(script); + + WHEN("the PDU is ingested") + { + auto const result = fixture.ingest(pdu); + auto const requests = peer->requests(); + + THEN("the state is asked at both gaps, but the refused event is asked for once") + { + CHECK(state_asked_at(requests, first_gap.event_id)); + CHECK(state_asked_at(requests, second_gap.event_id)); + CHECK(event_fetches_of(requests, refused.event_id) == 1U); + } + + THEN("each prev_event the origin did not supply was asked for once, and the PDU is not placed") + { + CHECK(event_fetches_of(requests, first_gap.event_id) == 1U); + CHECK(event_fetches_of(requests, second_gap.event_id) == 1U); + CHECK(result.status != PduIngestionStatus::accepted); + CHECK_FALSE(stored_status(fixture.runtime(), refused.event_id).has_value()); + } + + AND_WHEN("the PDU is ingested again, as the deferred retry does") + { + std::ignore = fixture.ingest(pdu); + auto const later = peer->requests(); + + THEN("the new recovery asks the origin for the refused event again, once") + { + CHECK(event_fetches_of(later, refused.event_id) == 2U); + } + } + } + } +} + // Spec: server-server-api.md "Rejection": "Subsequent events from other servers // that reference rejected events should be allowed if they still pass the auth // rules"; and room version auth rule 2.3 (rooms/v10.md): an event whose diff --git a/tests/meson.build b/tests/meson.build index d5485ce80..cbc55647d 100644 --- a/tests/meson.build +++ b/tests/meson.build @@ -545,6 +545,7 @@ shell = find_program('sh') 'unit/test_database_schema.cpp', 'unit/test_homeserver_auth_service.cpp', 'unit/test_homeserver_media_service.cpp', + 'unit/test_homeserver_recently_unreachable_servers.cpp', 'unit/test_streamed_body.cpp', 'unit/test_client_outbound_proxy.cpp', 'unit/test_federation_server_name.cpp', diff --git a/tests/unit/test_homeserver_recently_unreachable_servers.cpp b/tests/unit/test_homeserver_recently_unreachable_servers.cpp new file mode 100644 index 000000000..87d24653d --- /dev/null +++ b/tests/unit/test_homeserver_recently_unreachable_servers.cpp @@ -0,0 +1,125 @@ +// SPDX-FileCopyrightText: 2026 James Chapman +// SPDX-License-Identifier: GPL-3.0-or-later +// +// ADR-0761: the bounded, expiring record of servers that gave no answer to a remote profile lookup. +// Tags: [homeserver][adr0761]. + +#include "merovingian/homeserver/RecentlyUnreachableServers.hpp" + +#include + +#include +#include +#include + +using merovingian::homeserver::RecentlyUnreachableServers; + +namespace +{ + +constexpr auto ttl = std::chrono::seconds{60}; +auto const start = RecentlyUnreachableServers::Clock::time_point{} + std::chrono::hours{1}; + +} // namespace + +SCENARIO("A server that gave no answer is remembered until its record expires", "[homeserver][adr0761]") +{ + GIVEN("a record with a 60 second lifetime") + { + auto servers = RecentlyUnreachableServers{ttl, 8U}; + + WHEN("a server is remembered") + { + servers.remember("down.example.org", start); + + THEN("it is remembered just before the lifetime ends and not at its end") + { + REQUIRE(servers.contains("down.example.org", start)); + REQUIRE(servers.contains("down.example.org", start + ttl - std::chrono::milliseconds{1})); + REQUIRE_FALSE(servers.contains("down.example.org", start + ttl)); + } + + THEN("another server is not remembered") + { + REQUIRE_FALSE(servers.contains("up.example.org", start)); + } + } + + WHEN("a server is remembered and fails again later") + { + servers.remember("down.example.org", start); + servers.remember("down.example.org", start + std::chrono::seconds{50}); + + THEN("the later failure starts its lifetime again, and it is still one entry") + { + REQUIRE(servers.contains("down.example.org", start + std::chrono::seconds{100})); + REQUIRE(servers.size() == 1U); + } + } + + WHEN("a remembered server answers and is forgotten") + { + servers.remember("down.example.org", start); + servers.forget("down.example.org"); + + THEN("it is no longer remembered") + { + REQUIRE_FALSE(servers.contains("down.example.org", start)); + REQUIRE(servers.size() == 0U); + } + } + } +} + +SCENARIO("The record of servers that gave no answer never grows past its capacity", "[homeserver][adr0761]") +{ + GIVEN("a record that holds two servers, already full") + { + auto servers = RecentlyUnreachableServers{ttl, 2U}; + servers.remember("first.example.org", start); + servers.remember("second.example.org", start + std::chrono::seconds{10}); + + WHEN("a third server fails while both are still live") + { + servers.remember("third.example.org", start + std::chrono::seconds{20}); + + THEN("the server remembered longest is dropped and the record still holds two") + { + auto const now = start + std::chrono::seconds{20}; + REQUIRE(servers.size() == 2U); + REQUIRE_FALSE(servers.contains("first.example.org", now)); + REQUIRE(servers.contains("second.example.org", now)); + REQUIRE(servers.contains("third.example.org", now)); + } + } + + WHEN("a third server fails after the first record has expired") + { + auto const now = start + ttl + std::chrono::seconds{5}; + servers.remember("third.example.org", now); + + THEN("the expired record makes room and the live one is kept") + { + REQUIRE(servers.size() == 2U); + REQUIRE(servers.contains("second.example.org", now)); + REQUIRE(servers.contains("third.example.org", now)); + } + } + } + + GIVEN("a record with no capacity") + { + auto servers = RecentlyUnreachableServers{ttl, 0U}; + + WHEN("a server fails") + { + servers.remember("down.example.org", start); + + THEN("nothing is remembered") + { + REQUIRE(servers.size() == 0U); + REQUIRE_FALSE(servers.contains("down.example.org", start)); + } + } + } +} From 5635c4c56692b5ce2eddd08044d0b83f861b81b4 Mon Sep 17 00:00:00 2001 From: James Chapman Date: Sun, 15 Nov 2026 23:06:28 +0000 Subject: [PATCH 4/5] Clear the profile record on a redirect, and end a snapshot before refetching its events Review fixes for PR #593. - ADR-0761: OutboundCallResult::remote_answered is set for any HTTP response, a refused redirect and an oversized body included, and remote_profile_response forgets the server on it. remote_status is 0 for those two, so a lookup that overlapped a failed one left the server recorded although it had just answered. - ADR-0760: backfill_state_ids_snapshot checks its listed events for a recorded refusal (not held or stored since) before fetching any, since phase 1 stores nothing and would refetch the events listed ahead of it. - Tests: an expired record entry is removed by a redirect answer; a later snapshot fetches none of its events when it lists a refused one. Co-Authored-By: Claude Opus 5.5 --- ...n-for-a-listed-state-event-at-most-once.md | 6 ++- ...-lookup-is-not-asked-again-for-a-minute.md | 5 +- docs/debug-logging.md | 2 +- .../homeserver/OutboundCallResult.hpp | 4 ++ src/homeserver/AGENTS.md | 2 +- src/homeserver/ClientServerRuntime.cpp | 2 +- src/homeserver/LocalHttpRouter.cpp | 38 +++++++++++---- src/homeserver/RoomService.cpp | 12 ++++- tests/integration/test_client_server_flow.cpp | 48 +++++++++++++++++++ .../test_pdu_ingestion_backfill_flow.cpp | 47 ++++++++++++++++++ 10 files changed, 148 insertions(+), 18 deletions(-) diff --git a/docs/adr/0760-a-recovery-asks-the-origin-for-a-listed-state-event-at-most-once.md b/docs/adr/0760-a-recovery-asks-the-origin-for-a-listed-state-event-at-most-once.md index 727790b58..b7c99caca 100644 --- a/docs/adr/0760-a-recovery-asks-the-origin-for-a-listed-state-event-at-most-once.md +++ b/docs/adr/0760-a-recovery-asks-the-origin-for-a-listed-state-event-at-most-once.md @@ -44,8 +44,10 @@ Should a recovery ask the origin again for a listed state event it has already f Chosen option: the first. `BackfillBudget::unsupplied_events` holds the IDs. `backfill_state_ids_snapshot` records a listed state event whenever its fetch brought back no usable PDU (refused, missing, another event, or no answer), -and ends a later snapshot that lists a recorded ID without a request, logging at DEBUG, since the failure itself was -logged at WARNING when it happened. `fetch_event_by_id` and `fetch_snapshot_event_by_id` neither record nor consult +and ends a later snapshot that lists a recorded ID, neither held nor stored since, before it fetches any of its listed +events, logging at DEBUG, since the failure itself was logged at WARNING when it happened. The check comes first +because phase 1 stores nothing until every event is in hand: a snapshot that fetched the events listed before the +refused one would fetch them again only to discard them. `fetch_event_by_id` and `fetch_snapshot_event_by_id` neither record nor consult the set. The set belongs to one `BackfillBudget`, so it never outlives the recovery, and the next deferred retry asks again. diff --git a/docs/adr/0761-a-server-that-gave-no-answer-to-a-profile-lookup-is-not-asked-again-for-a-minute.md b/docs/adr/0761-a-server-that-gave-no-answer-to-a-profile-lookup-is-not-asked-again-for-a-minute.md index 91bda62ab..b32c935ca 100644 --- a/docs/adr/0761-a-server-that-gave-no-answer-to-a-profile-lookup-is-not-asked-again-for-a-minute.md +++ b/docs/adr/0761-a-server-that-gave-no-answer-to-a-profile-lookup-is-not-asked-again-for-a-minute.md @@ -43,7 +43,10 @@ found no server or the request got no connection, no verified TLS session or no oversized body and an error status are answers. `remote_profile_response` records such a server in `HomeserverRuntime::profile_unreachable_servers` (`RecentlyUnreachableServers`: 60 s, at most 4096 servers, expired entries and then the oldest dropped when full). While it is recorded, a lookup for any of its users is answered 502 -`M_UNKNOWN` with no outbound call, logged at DEBUG. A server that answers at all is forgotten. +`M_UNKNOWN` with no outbound call, logged at DEBUG. A server that answers at all is forgotten: `perform_outbound_call` +sets `OutboundCallResult::remote_answered` for any HTTP response, a redirect it refuses to follow and an oversized +body included. `remote_status` is 0 for those two, so it is not the test; a lookup overlapping one that failed would +otherwise leave the server recorded although it had just answered. The record covers profile lookups only. Other client-triggered calls (public room directory, thirdparty) are made when a user asks for them, not automatically, so they keep asking every time. diff --git a/docs/debug-logging.md b/docs/debug-logging.md index ed4a87451..28a6352e0 100644 --- a/docs/debug-logging.md +++ b/docs/debug-logging.md @@ -46,7 +46,7 @@ server declines to repeat them: - Backfill: `federation.backfill.snapshot_event_fetch_failed` (module `rooms`) and the `backfill_state_ids_snapshot: a listed event could not be fetched` warning are written once per event and recovery. A later state snapshot in the same recovery - that lists the same event ends without a request and logs the debug line + that lists the same event ends before fetching any of its events and logs the debug line `a listed event was already not supplied in this recovery` (ADR-0760). The deferred retry runs a new recovery, which asks, and warns, again. - Remote profile lookups: `profile.proxy` with `http_status=0` is a warning when a diff --git a/include/merovingian/homeserver/OutboundCallResult.hpp b/include/merovingian/homeserver/OutboundCallResult.hpp index 0be799d2a..9d9d461e0 100644 --- a/include/merovingian/homeserver/OutboundCallResult.hpp +++ b/include/merovingian/homeserver/OutboundCallResult.hpp @@ -24,6 +24,10 @@ struct OutboundCallResult final // connection, no verified TLS session or no response in time. False on success, on any HTTP answer (including a // redirect or an oversized body) and on a refusal made here before anything was sent (policy, deadline, key). bool remote_unreachable{false}; + // ADR-0761: true when the destination sent an HTTP response, whatever this server then made of it: a success, any + // other status, a redirect it refused to follow or a body too large to accept. `remote_status` is 0 for the last + // two, so it cannot stand in for this. + bool remote_answered{false}; }; } // namespace merovingian::homeserver diff --git a/src/homeserver/AGENTS.md b/src/homeserver/AGENTS.md index e8fdc349f..15cafeaf6 100644 --- a/src/homeserver/AGENTS.md +++ b/src/homeserver/AGENTS.md @@ -614,7 +614,7 @@ only endpoints the spec requires and the origin may be asked for: nothing stored. - A recovery asks the origin for a listed state event at most once (ADR-0760). A state event a `/state_ids` snapshot listed and the origin did not supply is kept in `BackfillBudget::unsupplied_events`, and a later snapshot that lists - it ends without a request. Only `backfill_state_ids_snapshot`'s listed-event fetch records or consults the set: the + it (neither held nor stored since) ends before it fetches any of its events. Only `backfill_state_ids_snapshot`'s listed-event fetch records or consults the set: the per-reference fetch and a snapshot's target fetch ask as before, since the target is fetched again once its state is known (ADR-0069). The set lives only as long as the recovery's `BackfillBudget`; the deferred retry asks again. - `verify_and_store_backfilled_event` is called with `runtime.mutex` released. It takes the room stripe and then diff --git a/src/homeserver/ClientServerRuntime.cpp b/src/homeserver/ClientServerRuntime.cpp index 27aa96f9f..2f42e00e5 100644 --- a/src/homeserver/ClientServerRuntime.cpp +++ b/src/homeserver/ClientServerRuntime.cpp @@ -12416,7 +12416,7 @@ namespace { rt.homeserver.profile_unreachable_servers.remember(destination, RecentlyUnreachableServers::Clock::now()); } - else if (result.ok || result.remote_status != 0U) + else if (result.remote_answered) { rt.homeserver.profile_unreachable_servers.forget(destination); } diff --git a/src/homeserver/LocalHttpRouter.cpp b/src/homeserver/LocalHttpRouter.cpp index f650e053e..ed54d6073 100644 --- a/src/homeserver/LocalHttpRouter.cpp +++ b/src/homeserver/LocalHttpRouter.cpp @@ -4764,6 +4764,34 @@ namespace // event the origin cannot supply (a gap in the auth chain, a state event // it will not serve) ends the attempt with nothing stored. The fetches // are bounded by max_snapshot_outbound_calls and the shared budget. + // ADR-0760: a listed state event the origin already failed to supply in this recovery, and that is neither + // held nor stored since, means this snapshot cannot be completed. End it before fetching any of the others, + // which phase 1 would fetch only to discard. The failure itself was logged when it happened. + if (!budget.unsupplied_events.empty()) + { + auto const stripe = std::hash{}(std::string{room_id}) % room_mutex_stripe_count; + for (auto const& id : needed_ids) + { + if (!budget.origin_did_not_supply(id) || held.find(id) != held.end()) + { + continue; + } + { + auto stripe_guard = std::unique_lock{runtime.room_stripe_mutexes[stripe]}; + auto global_guard = RuntimeLock{runtime.mutex}; + if (runtime.database.persistent_store.events.contains(id)) + { + continue; + } + } + LOG_DEBUG("backfill_state_ids_snapshot: a listed event was already not supplied in this recovery; " + "room_id=" + + std::string{room_id} + " target_event_id=" + std::string{target_event_id} + + " event_id=" + id); + return false; + } + } + struct SnapshotFetch final { std::string id{}; @@ -4803,16 +4831,6 @@ namespace { return false; } - // ADR-0760: the origin already failed to supply this event during this recovery and its answer - // was logged then, so this snapshot cannot be completed and the event is not asked for again. - if (budget.origin_did_not_supply(id)) - { - LOG_DEBUG("backfill_state_ids_snapshot: a listed event was already not supplied in this " - "recovery; room_id=" + - std::string{room_id} + " target_event_id=" + std::string{target_event_id} + - " event_id=" + id); - return false; - } auto supplied = fetch_snapshot_event_by_id(runtime, room_id, origin, id, policy, snapshot_calls, budget); if (!supplied.has_value()) diff --git a/src/homeserver/RoomService.cpp b/src/homeserver/RoomService.cpp index e849bb757..e61c9cb38 100644 --- a/src/homeserver/RoomService.cpp +++ b/src/homeserver/RoomService.cpp @@ -2852,6 +2852,13 @@ namespace return false; } + // ADR-0761: the transport failures that still mean the destination sent an HTTP response: a redirect this server + // will not follow, and a response too large to accept. + [[nodiscard]] auto destination_sent_a_response(http::OutboundError error) noexcept -> bool + { + return error == http::OutboundError::redirect_rejected || error == http::OutboundError::response_too_large; + } + } // namespace [[nodiscard]] auto perform_outbound_call(HomeserverRuntime& runtime, OutboundAnswerTrust trust, @@ -3044,9 +3051,10 @@ namespace : outcome.error_detail, outcome.ok ? outcome.response.status : std::uint16_t{0U}, outcome.ok ? remote_invite_blocked_errcode(outcome.response.body) : std::string{}, - !outcome.ok && destination_gave_no_answer(outcome.error)}; + !outcome.ok && destination_gave_no_answer(outcome.error), + outcome.ok || destination_sent_a_response(outcome.error)}; } - return {true, outcome.response.body, outcome.response.status}; + return {true, outcome.response.body, outcome.response.status, {}, false, true}; } // How long a published signing key advertises itself as valid. Federation peers diff --git a/tests/integration/test_client_server_flow.cpp b/tests/integration/test_client_server_flow.cpp index 919bb5045..8d2e06e6e 100644 --- a/tests/integration/test_client_server_flow.cpp +++ b/tests/integration/test_client_server_flow.cpp @@ -26,6 +26,7 @@ #include "merovingian/homeserver/ClientServerStartResult.hpp" #include "merovingian/homeserver/LocalHttpRequest.hpp" #include "merovingian/homeserver/OutboundDeadline.hpp" +#include "merovingian/homeserver/RecentlyUnreachableServers.hpp" #include "merovingian/homeserver/RemoteTimestampEvent.hpp" #include "merovingian/homeserver/RuntimeStartResult.hpp" #include "merovingian/homeserver/AuthService.hpp" @@ -1236,6 +1237,53 @@ SCENARIO("GET /profile/{userId} does not ask again a server that just gave no an } } } + + GIVEN("a server still in the record, its entry expired, that now answers with a redirect") + { + auto const config = registration_enabled_config(); + auto started = merovingian::homeserver::start_client_server(config); + REQUIRE(started.started); + auto& rt = started.runtime; + + auto certificate = merovingian::tests::tls_mock::write_test_tls_certificate("localhost"); + auto tls = merovingian::homeserver::make_tls_server_context(certificate.certificate_file, + certificate.private_key_file); + REQUIRE(tls.ok()); + auto tls_context = std::move(*tls.context); + auto acceptor = merovingian::net::TcpAcceptor{}; + REQUIRE(acceptor.bind("127.0.0.1", 0U).ok); + rt.homeserver.test_forced_outbound_resolution["moved.example.net"] = + merovingian::homeserver::TestOnlyForcedOutboundResolution{ + "localhost", acceptor.bound_port(), {"127.0.0.1"}, certificate.certificate_pem}; + // An entry that has expired but not been dropped, as an overlapping lookup's failure leaves while this one + // is in flight: the lookup goes out, and only the answer can remove the entry. + rt.homeserver.profile_unreachable_servers.remember( + "moved.example.net", + merovingian::homeserver::RecentlyUnreachableServers::Clock::now() - std::chrono::minutes{2}); + REQUIRE(rt.homeserver.profile_unreachable_servers.size() == 1U); + auto const responses = std::vector>{ + {"user_id=%40dave%3Amoved.example.net", + merovingian::tests::tls_mock::json_http_response("302 Found", R"({})")}, + }; + auto captured = std::vector{}; + auto server_thread = std::thread{[&]() { + merovingian::tests::tls_mock::run_path_dispatch_tls_server(acceptor, tls_context, responses, &captured); + }}; + auto const join_server = merovingian::tests::tls_mock::ScopedThreadJoin{server_thread}; + + WHEN("a client reads a profile of that server's user") + { + auto const response = merovingian::homeserver::handle_client_server_request( + rt, {"GET", "/_matrix/client/v3/profile/%40dave%3Amoved.example.net", {}, {}}); + + THEN("the redirect is not followed, but the server answered, so it is forgotten") + { + REQUIRE(response.response.status != 200U); + REQUIRE(captured.size() == 1U); + REQUIRE(rt.homeserver.profile_unreachable_servers.size() == 0U); + } + } + } } // Spec: Matrix Client-Server API v1.19, GET /profile/{userId}; Server-Server API v1.19, query/profile diff --git a/tests/integration/test_pdu_ingestion_backfill_flow.cpp b/tests/integration/test_pdu_ingestion_backfill_flow.cpp index 0e63fd524..f2ae6c7d6 100644 --- a/tests/integration/test_pdu_ingestion_backfill_flow.cpp +++ b/tests/integration/test_pdu_ingestion_backfill_flow.cpp @@ -4145,6 +4145,53 @@ SCENARIO("A recovery asks the origin for a listed state event it did not supply } } +// ADR-0760: phase 1 stores nothing until every listed event is in hand, so a later snapshot that would fetch the +// events listed before a refused one again, only to discard them, ends before fetching any of them. +SCENARIO("A later snapshot listing a state event the origin refused fetches none of its other events", + "[pdu_ingestion][backfill][adr0760]") +{ + GIVEN("two gaps whose /state_ids answers list a state event the origin serves before one it refuses") + { + auto fixture = RecoveryFixture{"!adr0760-refused-after:local.example.org"}; + auto const& room = fixture.room_id; + auto const first_gap = make_remote_message_pdu(room, {fixture.bob_member_id()}, fixture.auth_ids(), 4, 10); + auto const second_gap = make_remote_message_pdu(room, {fixture.bob_member_id()}, fixture.auth_ids(), 4, 11); + auto const served = make_remote_event_pdu(room, "m.room.member", std::string{"@alice:remote.example.org"}, + "@alice:remote.example.org", member_content("A"), + {fixture.bob_member_id()}, fixture.auth_ids(), 4, 12); + auto const refused = + make_remote_event_pdu(room, "m.room.member", std::string{remote_member}, remote_member, member_content("R"), + {fixture.bob_member_id()}, fixture.auth_ids(), 4, 13); + auto const pdu = + make_remote_message_pdu(room, {first_gap.event_id, second_gap.event_id}, fixture.auth_ids(), 5, 20); + + auto pdu_ids = fixture.genesis_state_ids(); + pdu_ids.push_back(served.event_id); + pdu_ids.push_back(refused.event_id); + auto script = PeerScript{}; + script.events[served.event_id] = served.json; + script.state_ids[first_gap.event_id] = make_state_ids_response(pdu_ids, fixture.auth_ids()); + script.state_ids[second_gap.event_id] = make_state_ids_response(pdu_ids, fixture.auth_ids()); + auto const peer = fixture.serve(script); + + WHEN("the PDU is ingested") + { + auto const result = fixture.ingest(pdu); + auto const requests = peer->requests(); + + THEN("the state is asked at both gaps, but each listed event is fetched only by the first snapshot") + { + CHECK(state_asked_at(requests, first_gap.event_id)); + CHECK(state_asked_at(requests, second_gap.event_id)); + CHECK(event_fetches_of(requests, served.event_id) == 1U); + CHECK(event_fetches_of(requests, refused.event_id) == 1U); + CHECK(result.status != PduIngestionStatus::accepted); + CHECK_FALSE(stored_status(fixture.runtime(), served.event_id).has_value()); + } + } + } +} + // Spec: server-server-api.md "Rejection": "Subsequent events from other servers // that reference rejected events should be allowed if they still pass the auth // rules"; and room version auth rule 2.3 (rooms/v10.md): an event whose From dd546b377e1ac42275b334945799da7b8d1831ec Mon Sep 17 00:00:00 2001 From: James Chapman Date: Sun, 15 Nov 2026 23:34:53 +0000 Subject: [PATCH 5/5] Guard the oldest-entry erase in RecentlyUnreachableServers for GCC 16 Fedora 45 (GCC 16, -O3, -Werror) reports erasing min_element's result as a potential null dereference: it cannot see that the set is not empty there (size >= capacity > 0). Erase only an element that was found. Co-Authored-By: Claude Opus 5.5 --- src/homeserver/RecentlyUnreachableServers.cpp | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/src/homeserver/RecentlyUnreachableServers.cpp b/src/homeserver/RecentlyUnreachableServers.cpp index e1484de21..2c9c0e9f6 100644 --- a/src/homeserver/RecentlyUnreachableServers.cpp +++ b/src/homeserver/RecentlyUnreachableServers.cpp @@ -66,7 +66,12 @@ auto RecentlyUnreachableServers::make_room(Clock::time_point now) -> void auto const oldest = std::ranges::min_element(m_failed_at, {}, [](auto const& entry) { return entry.second; }); - m_failed_at.erase(oldest); + // The set is not empty here (size >= capacity > 0), but GCC 16 at -O3 cannot see that and reports erasing end() + // as a null dereference. + if (oldest != m_failed_at.end()) + { + m_failed_at.erase(oldest); + } } } // namespace merovingian::homeserver