diff --git a/CHANGELOG.md b/CHANGELOG.md index b7732b7db..a2c042d6c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,6 +3,31 @@ Changes from 1.0.0, the first production release, onwards; the format is set out Changes before 1.0.0 are in [CHANGELOG-pre-release.md](CHANGELOG-pre-release.md), which is no longer added to ([ADR-0740](docs/adr/0740-the-changelog-starts-at-1-0-0-and-the-pre-release-entries-are-a-frozen-record.md)). +# Version 1.0.3 + +## Features + +* `docs/release-process.md` and the production milestone record that 1.0.1, like 1.0.0, was tagged without a green + Complement run, and that the 1.0.0 GitHub release is withdrawn (ADR-0744). + +## Security Fixes + +* None. + +## Bug fixes + +* A backfill recovery asks the origin for a listed state event at most once: a later state snapshot listing one the + origin already refused ends without asking, so it is fetched and logged once, not once per gap (ADR-0760). +* A remote profile lookup for a server that just gave no answer is answered 502 at once for the next 60 s, so client + lookups to a dead server no longer each hold an outbound slot for the whole deadline (ADR-0761). + +## Release notes + +1.0.3 cuts repeated federation work and log noise seen on a production server: a refused history event is asked for +once per recovery, and a server that gives no answer to profile lookups is left alone for a minute. No configuration +change is needed. 1.0.1 was also tagged without a green Complement run, and the 1.0.0 GitHub release is withdrawn +because it ships the vulnerabilities 1.0.1 fixes; a 1.0.0 server should upgrade. + # Version 1.0.2 ## Features diff --git a/docs/adr/0744-the-complement-blocker-is-also-waived-for-1-0-1.md b/docs/adr/0744-the-complement-blocker-is-also-waived-for-1-0-1.md new file mode 100644 index 000000000..4b6693570 --- /dev/null +++ b/docs/adr/0744-the-complement-blocker-is-also-waived-for-1-0-1.md @@ -0,0 +1,55 @@ +# The Complement blocker is also waived for 1.0.1 + +* Status: accepted +* Date: 2026-11-13 + +## Context and Problem Statement + +[ADR-0648](0648-the-complement-blocker-is-waived-for-1-0-0-only.md) waived the green Complement run for 1.0.0 only, and +`docs/release-process.md` said the waiver applies to no later release. + +1.0.1 fixes the 49 findings of the run-3 security audit of the 1.0.0 release candidate, 9 of them high. The 1.0.0 release +ships those vulnerabilities, so the owner decided on 2026-11-13 to tag 1.0.1 at once and remove the 1.0.0 release. No +Complement run exists for 1.0.1. Can 1.0.1 be tagged without one? + +## Considered Options + +* Waive the blocker for 1.0.1 as well; it stays open for every later release +* Hold the 1.0.1 tag until a green Complement run exists + +## Decision Outcome + +Chosen option: "Waive the blocker for 1.0.1 as well". + +What is waived: the green Complement run before the 1.0.1 tag. 1.0.1 carries no Complement evidence and no signed +Complement record is attached to it. + +Why: the only published release, 1.0.0, has known vulnerabilities and 1.0.1 fixes them. Holding the fix for a Complement +run would leave 1.0.0 as the only release to install. The run-3 audit and this project's own suites gate 1.0.1, as the +run-2 audit gated 1.0.0. + +What still applies: + +* The blocker stays open in `docs/todos/production-milestone.md`. +* 1.0.2 and every later release follow `docs/release-process.md` as written, including a green Complement run before the + tag. This waiver is no precedent for them. +* The 1.0.0 GitHub release is deleted, because it ships the vulnerabilities 1.0.1 fixes. The `v1.0.0` tag stays: a tag is + never moved or removed once something was published under it, and ADR-0651 cites it. + +### Negative Consequences + +* Federation conformance for 1.0.1 rests on this project's own tests and the audit. No external check supports it. + +## Pros and Cons of the Options + +### Hold the 1.0.1 tag until a green Complement run exists + +* Good, because it follows the release process as written. +* Bad, because the vulnerable 1.0.0 stays the only release until a run is green, and no green run has ever been recorded. + +## Links + +* [ADR-0648](0648-the-complement-blocker-is-waived-for-1-0-0-only.md), + [ADR-0651](0651-the-v1-0-0-tag-moves-to-the-release-workflow-fix-without-a-version-bump.md), + [ADR-0235](0235-complement-evidence-is-a-maintainer-signed-record-tied-to-the-shipped-binary.md) +* [docs/release-process.md](../release-process.md), [docs/todos/production-milestone.md](../todos/production-milestone.md) diff --git a/docs/adr/0760-a-recovery-asks-the-origin-for-a-listed-state-event-at-most-once.md b/docs/adr/0760-a-recovery-asks-the-origin-for-a-listed-state-event-at-most-once.md new file mode 100644 index 000000000..b7c99caca --- /dev/null +++ b/docs/adr/0760-a-recovery-asks-the-origin-for-a-listed-state-event-at-most-once.md @@ -0,0 +1,93 @@ +# A recovery asks the origin for a listed state event at most once + +* Status: accepted +* Deciders: James Chapman +* Date: 2026-11-15 + +Technical Story: production log of pong.ping.me.uk at 1.0.2, 2026-11-15: about 95% of a day's warnings came from +three retained PDUs in one room. + +## Context and Problem Statement + +`backfill_missing_pdu_references` recovers the history a PDU lacks. When `/get_missing_events` and `/event` leave a +gap, it asks `GET /state_ids` at up to `backfill.max_outbound_calls` gap events (`state_snapshot_targets`), and each +snapshot fetches every state event it lists with `GET /event` before it stores any. One listed event the origin will +not supply ends that snapshot. + +In the log, matrix.kahlilia.org answered every `/state_ids` request, and every answer listed the same state event, +which it then refused with 403 on `/event`, most likely because this server had not joined when the event was sent. +Each recovery therefore asked for that one event 15 times, once per snapshot target, and logged two warnings each +time. The deferred PDU retry (30 s doubling to 1 h, 10 attempts) repeated the whole recovery. + +Should a recovery ask the origin again for a listed state event it has already failed to supply? + +## Decision Drivers + +* The origin decides what it serves. A refusal, a 404 or an unusable answer for one event does not change within + one recovery, which is bounded by `backfill.timeout_seconds` (45 s by default). +* Every snapshot needs every state event it lists, so a snapshot listing one that was not supplied cannot be + completed, whatever else it lists. +* The repeat is per snapshot target: the per-reference stage asks for each missing reference once, and each snapshot + asks for its own target once, so neither multiplies with the number of gaps. +* The history may become available later: the origin may fetch it, or this server may gain the right to see it. + The deferred retry exists for that. + +## Considered Options + +* Remember, for the rest of one recovery, each state event a `/state_ids` snapshot listed and the origin did not + supply; a later snapshot that lists one ends at once without asking. +* Remember every event ID any `GET /event` of the recovery did not supply, and send no further request for it. +* Stop trying further snapshot targets after the first snapshot fails. +* Remember unsupplied events across recoveries, for example in the retained PDU's row. + +## Decision Outcome + +Chosen option: the first. `BackfillBudget::unsupplied_events` holds the IDs. `backfill_state_ids_snapshot` records a +listed state event whenever its fetch brought back no usable PDU (refused, missing, another event, or no answer), +and ends a later snapshot that lists a recorded ID, neither held nor stored since, before it fetches any of its listed +events, logging at DEBUG, since the failure itself was logged at WARNING when it happened. The check comes first +because phase 1 stores nothing until every event is in hand: a snapshot that fetched the events listed before the +refused one would fetch them again only to discard them. `fetch_event_by_id` and `fetch_snapshot_event_by_id` neither record nor consult +the set. The set belongs to one `BackfillBudget`, so it never outlives the recovery, and the next deferred retry asks +again. + +`/state_ids` itself is still asked at every target: each target is a different question, and its answer may list +only events the origin does serve. + +### Positive Consequences + +* A recovery whose snapshots all list one refused event sends one `/event` request for it, not one per target, and + logs its failure once. In the observed case that is 15 requests and 30 warnings down to 1 and 2. +* The calls saved are left in the recovery's budget for targets that can still succeed. +* A snapshot's target is still fetched after its state is known even when the per-reference stage could not get it + (ADR-0069, ADR-0070). + +### Negative Consequences + +* A listed state event whose fetch failed for a transient reason, such as a timeout, is not retried within the same + recovery. The deferred retry still asks again, as it did before. + +## Pros and Cons of the Options + +### Remember every unsupplied event + +* Good, because it would also save the target fetch after a per-reference fetch of the same event failed. +* Bad, because that saves at most one request per gap event, and it takes away the second ask that the snapshot path + relies on: an event the per-reference stage could not get is fetched again as the snapshot's target once its + state is known. The ADR-0069 and ADR-0070 scenarios fail with it. + +### Stop after the first failed snapshot + +* Good, because it saves the `/state_ids` requests too. +* Bad, because the other targets are different gaps, and their snapshots may list only events the origin serves. + +### Remember across recoveries + +* Good, because the retries would also skip the request. +* Bad, because it would refuse history that has become available since, and it needs persistent state that the + deferred retry's own limits already bound. + +## Links + +* Refines [ADR-0069](0069-spec-conformant-state-ids-fallback-via-event-auth.md) and the A8 recovery rules in + `src/homeserver/AGENTS.md` diff --git a/docs/adr/0761-a-server-that-gave-no-answer-to-a-profile-lookup-is-not-asked-again-for-a-minute.md b/docs/adr/0761-a-server-that-gave-no-answer-to-a-profile-lookup-is-not-asked-again-for-a-minute.md new file mode 100644 index 000000000..b32c935ca --- /dev/null +++ b/docs/adr/0761-a-server-that-gave-no-answer-to-a-profile-lookup-is-not-asked-again-for-a-minute.md @@ -0,0 +1,82 @@ +# A server that gave no answer to a profile lookup is not asked again for a minute + +* Status: accepted +* Deciders: James Chapman +* Date: 2026-11-15 + +Technical Story: production log of pong.ping.me.uk at 1.0.2, 2026-11-15: eight `profile.proxy` timeouts to +example.org in two minutes. + +## Context and Problem Statement + +`GET /_matrix/client/v3/profile/{userId}` for a user of another server asks that server's `query/profile`. The call +holds a client outbound slot (ADR-0079, ADR-0751) and runs under the directory deadline. Clients make these lookups +by themselves, for member lists, hover cards and invite dialogs, and retry them. In the log, a client looked up a +user on example.org, which runs no Matrix server: discovery fell back to port 8448, every connection timed out, and +each of eight lookups in two minutes waited out the deadline and logged a warning. + +The server discovery cache does not help: discovery succeeded, and only the connection failed. Should a server that +has just given no answer be asked again on the next lookup? + +## Decision Drivers + +* A lookup is interactive and cheap to repeat. A short delay before a recovered server is asked again costs a + client one missing display name, while a dead server costs a slot and the full deadline per lookup. +* Only the destination's silence should count. An HTTP answer of any status proves the server is there, and a + refusal made locally (federation policy, deadline, missing key) says nothing about it. +* The record must stay bounded, since a client chooses which servers are looked up. +* ADR-0262 and ADR-0749 keep joins and outbound delivery from serving remembered failures. Those retries are paced + by their own backoff or by the user; a profile lookup has neither. + +## Considered Options + +* Remember each server that gave no answer to a profile lookup for 60 seconds, in a bounded record guarded by + `runtime.mutex`, and answer lookups for it 502 at once during that time. +* Serve remembered discovery failures to profile lookups, as key resolution does. +* Shorten the directory deadline for profile lookups. + +## Decision Outcome + +Chosen option: the first. `perform_outbound_call` sets `OutboundCallResult::remote_unreachable` when discovery +found no server or the request got no connection, no verified TLS session or no response in time +(`unresolved_host`, `connection_failed`, `tls_verification_failed`, `timeout`, `network_error`). A redirect, an +oversized body and an error status are answers. `remote_profile_response` records such a server in +`HomeserverRuntime::profile_unreachable_servers` (`RecentlyUnreachableServers`: 60 s, at most 4096 servers, expired +entries and then the oldest dropped when full). While it is recorded, a lookup for any of its users is answered 502 +`M_UNKNOWN` with no outbound call, logged at DEBUG. A server that answers at all is forgotten: `perform_outbound_call` +sets `OutboundCallResult::remote_answered` for any HTTP response, a redirect it refuses to follow and an oversized +body included. `remote_status` is 0 for those two, so it is not the test; a lookup overlapping one that failed would +otherwise leave the server recorded although it had just answered. + +The record covers profile lookups only. Other client-triggered calls (public room directory, thirdparty) are made +when a user asks for them, not automatically, so they keep asking every time. + +### Positive Consequences + +* A dead server costs one slot and one deadline per minute of lookups, not one per lookup, and logs one warning a + minute. +* Slots stay free for lookups to servers that do answer. + +### Negative Consequences + +* A server that comes back is not asked for up to 60 seconds after its last failure. +* The record is in memory, per process, and is lost on restart. + +## Pros and Cons of the Options + +### Serve remembered discovery failures + +* Good, because the cache already exists. +* Bad, because the observed case failed after discovery succeeded, so it would not have been covered. + +### A shorter deadline for profile lookups + +* Good, because it needs no state. +* Bad, because a slow but working server would then fail every lookup, and a dead one still costs a slot per lookup. + +## Links + +* Bounded by [ADR-0079](0079-client-triggered-outbound-proxying-runs-under-a-bounded-in-flight-budget.md) and + [ADR-0751](0751-the-per-client-outbound-call-cap-is-configurable-and-defaults-to-three.md) +* Contrasts with [ADR-0262](0262-a-join-reuses-server-discovery-but-never-a-remembered-failure.md) and + [ADR-0749](0749-outbound-delivery-never-serves-a-remembered-discovery-failure.md) diff --git a/docs/adr/index.md b/docs/adr/index.md index 5791492bc..f8e74386c 100644 --- a/docs/adr/index.md +++ b/docs/adr/index.md @@ -560,7 +560,7 @@ * [0635 - An inbound federation origin is asked about once at a time, and its decision is kept for a short time](0635-an-inbound-federation-origin-is-asked-about-once-at-a-time-and-its-decision-is-kept-for-a-short-time.md) * [0636 - A blob row is deleted with its last reference, and a removed record holds no reference](0636-a-blob-row-is-deleted-with-its-last-reference-and-a-removed-record-holds-none.md) * [0647 - Versions from 1.0.0 are MAJOR.MINOR.BUILD](0647-versions-from-1-0-0-are-major-minor-build.md) -* [0648 - The Complement blocker is waived for 1.0.0 only](0648-the-complement-blocker-is-waived-for-1-0-0-only.md) +* [0648 - The Complement blocker is waived for 1.0.0 only](0648-the-complement-blocker-is-waived-for-1-0-0-only.md) — Extended by ADR-0744: 1.0.1 is waived too * [0731 - A spawned worker gets no terminal, and no sandbox profile allows TIOCSTI or TIOCLINUX](0731-a-spawned-worker-gets-no-terminal-and-no-sandbox-profile-allows-tiocsti.md) * [0732 - Every published attestation check pins the signing workflow and the ref](0732-every-published-attestation-check-pins-the-signing-workflow-and-the-ref.md) — Partly superseded by ADR-0742: every published file is attested by packages.yml * [0719 - One client key holds a sixteenth of the parked-connection budgets, and a readable connection waits a bounded time](0719-one-client-key-holds-a-sixteenth-of-the-parked-connection-budgets-and-a-readable-connection-waits-a-bounded-time.md) @@ -626,6 +626,7 @@ * [0741 - Each changelog version has Features, Security Fixes, Bug fixes and Release notes](0741-each-changelog-version-has-features-security-fixes-bug-fixes-and-release-notes.md) * [0742 - Fixed build-and-test and package platform lists, and a tag ships what latest ships](0742-build-test-and-package-platform-lists-and-a-tag-ships-what-latest-ships.md) * [0743 - Analyses, then build and test jobs, then packages take runners](0743-analyses-then-build-and-test-then-packages-take-runners.md) +* [0744 - The Complement blocker is also waived for 1.0.1](0744-the-complement-blocker-is-also-waived-for-1-0-1.md) * [0745 - A first join into a restricted room stores the resident's verified co-signature](0745-a-first-restricted-join-stores-the-residents-verified-co-signature.md) * [0746 - A threaded receipt may name any event whose relation chain reaches the thread root](0746-a-threaded-receipt-may-name-any-event-whose-relation-chain-reaches-the-root.md) * [0747 - A remote member may leave through a server whose local members have all left](0747-a-remote-member-may-leave-through-a-server-whose-local-members-have-left.md) @@ -641,3 +642,5 @@ * [0757 - Hardened BSD builds link -pie, and the ELF gate reads elftoolchain readelf](0757-hardened-bsd-builds-link-pie-and-the-elf-gate-reads-elftoolchain-readelf.md) * [0758 - Every worker shard reloads every changed room, and main tracks each shard's copy](0758-every-worker-shard-reloads-every-changed-room-and-main-tracks-each-shards-copy.md) * [0759 - Main builds make_leave and make_knock templates while the worker's reload of the room is pending](0759-main-builds-make-leave-and-make-knock-templates-while-the-workers-reload-is-pending.md) +* [0760 - A recovery asks the origin for a listed state event at most once](0760-a-recovery-asks-the-origin-for-a-listed-state-event-at-most-once.md) +* [0761 - A server that gave no answer to a profile lookup is not asked again for a minute](0761-a-server-that-gave-no-answer-to-a-profile-lookup-is-not-asked-again-for-a-minute.md) diff --git a/docs/debug-logging.md b/docs/debug-logging.md index ab5e8199c..28a6352e0 100644 --- a/docs/debug-logging.md +++ b/docs/debug-logging.md @@ -38,6 +38,22 @@ when a gateway answered after more than half the total timeout and was counted a deadline and queued a pusher unattempted. `account.deactivated.pusher_removal_failed` (warning, `user_id`) is logged when a pusher of a deactivated account could not be deleted. +## Repeated remote failures logged once (1.0.3) + +Two remote failures are logged at warning when they happen and at debug when this +server declines to repeat them: + +- Backfill: `federation.backfill.snapshot_event_fetch_failed` (module `rooms`) and the + `backfill_state_ids_snapshot: a listed event could not be fetched` warning are + written once per event and recovery. A later state snapshot in the same recovery + that lists the same event ends before fetching any of its events and logs the debug line + `a listed event was already not supplied in this recovery` (ADR-0760). The + deferred retry runs a new recovery, which asks, and warns, again. +- Remote profile lookups: `profile.proxy` with `http_status=0` is a warning when a + server gives no answer. For the next 60 s, lookups for that server's users are + answered 502 without a call and log the debug line `event=profile.proxy ... + reason=gave no answer recently` (ADR-0761). + ## Room creation failures `room.create.failed` (module `rooms`) is an error, not debug-only. It is diff --git a/docs/http-transport.md b/docs/http-transport.md index 5a19b8e01..9246fb383 100644 --- a/docs/http-transport.md +++ b/docs/http-transport.md @@ -1125,7 +1125,7 @@ The paths under the budget: | `POST /_matrix/client/v3/join/{roomIdOrAlias}` for a remote alias | federation `GET /query/directory`, 10 s directory deadline (F-17, ADR-0350) | | `POST .../join`, `POST .../knock` and `POST .../leave`, when this server must ask a resident | federation `make_*` and `send_*`: one slot for the whole exchange, taken at once, 429 when none is free (ADR-0622) | | `POST /_matrix/client/v3/keys/query` and `/keys/claim`, for a user of another server | federation `POST /user/keys/query` and `/user/keys/claim`: one slot and one 30 s deadline (`remote_keys_deadline_seconds`) for all of the request's remote servers, at most 16 distinct servers (`remote_keys_max_servers`); a server not asked is reported in `failures` and the local answer is still returned, so no slot is not a 429 (F-17, ADR-0350) | -| `GET /_matrix/client/v3/profile/{userId}[/{keyName}]` for a user of another server | federation `GET /query/profile`, 10 s directory deadline | +| `GET /_matrix/client/v3/profile/{userId}[/{keyName}]` for a user of another server | federation `GET /query/profile`, 10 s directory deadline; a server that gave no answer (no address, connection, TLS session or response) is answered 502 `M_UNKNOWN` without a call for 60 s after (`RecentlyUnreachableServers`, ADR-0761) | | `GET /_matrix/client/v1/rooms/{roomId}/hierarchy`, once a child room is not held locally | federation `GET /hierarchy/{roomId}` to the link's `via` servers: one slot and one 10 s deadline for the whole walk, at most 3 servers per room and 16 requests per walk; the slot is taken at the first such request and `429 M_LIMIT_EXCEEDED` answers the whole walk when it is refused | | `GET /_matrix/client/v1/room_summary/{roomIdOrAlias}?via=...`, for a room not held locally (with or without a token, ADR-0556) | federation `GET /hierarchy/{roomId}` to at most 3 of the `via` servers: one slot (never the waiting form; 429 `M_LIMIT_EXCEEDED` at once without one) and one 10 s deadline | | `GET /_matrix/client/v1/rooms/{roomId}/timestamp_to_event`, when the local answer is missing or sits at a gap (ADR-0299) | federation `GET /timestamp_to_event` and `GET /event` to at most 3 servers with joined members: one slot and one 10 s deadline; the request waits for the slot within that deadline (ADR-0306) | diff --git a/docs/release-process.md b/docs/release-process.md index ab9befeba..1b1501b0d 100644 --- a/docs/release-process.md +++ b/docs/release-process.md @@ -185,11 +185,13 @@ sh scripts/check-release-readiness.sh ``` Then run Complement against the candidate (see "Complement" below). It must be green -before the tag is created. The one exception is 1.0.0: the owner waived this -requirement for that release only, because it was tagged without a green run -and the release-gate security audit was its gate instead -([ADR-0648](adr/0648-the-complement-blocker-is-waived-for-1-0-0-only.md)). The -waiver does not apply to any later release. +before the tag is created. The only exceptions are 1.0.0 and 1.0.1. The owner +waived the requirement for 1.0.0 because it was tagged without a green run and +the release-gate security audit was its gate instead +([ADR-0648](adr/0648-the-complement-blocker-is-waived-for-1-0-0-only.md)), and +for 1.0.1 because it fixes the vulnerabilities 1.0.0 ships +([ADR-0744](adr/0744-the-complement-blocker-is-also-waived-for-1-0-1.md)). The +waiver does not apply to 1.0.2 or any later release. After that, create and push the release tag. It must be exactly `v` followed by the version in `meson.build`, or the workflow refuses it: @@ -211,8 +213,10 @@ artifact. ## Complement -The 1.0.0 release was tagged without a Complement run and carries no Complement -evidence ([ADR-0648](adr/0648-the-complement-blocker-is-waived-for-1-0-0-only.md)). +The 1.0.0 and 1.0.1 releases were tagged without a Complement run and carry no Complement +evidence ([ADR-0648](adr/0648-the-complement-blocker-is-waived-for-1-0-0-only.md), +[ADR-0744](adr/0744-the-complement-blocker-is-also-waived-for-1-0-1.md)). The 1.0.0 GitHub +release was deleted because it ships the vulnerabilities 1.0.1 fixes; its `v1.0.0` tag stays. Every later release follows the steps below. The external Complement suite runs against the candidate's Debian 13 (trixie) package diff --git a/docs/todos/production-milestone.md b/docs/todos/production-milestone.md index eb31904b1..ae3497da5 100644 --- a/docs/todos/production-milestone.md +++ b/docs/todos/production-milestone.md @@ -16,7 +16,8 @@ and what has been deliberately dropped. ## What blocks 1.0.0 **1.0.0 is released with the Complement blocker waived, not closed.** -The owner waived it for 1.0.0 only ([ADR-0648](../adr/0648-the-complement-blocker-is-waived-for-1-0-0-only.md)). +The owner waived it for 1.0.0 ([ADR-0648](../adr/0648-the-complement-blocker-is-waived-for-1-0-0-only.md)) +and again for 1.0.1 ([ADR-0744](../adr/0744-the-complement-blocker-is-also-waived-for-1-0-1.md)), but for no later release. No green Complement run was ever recorded: the last, run 18 at 0.12.64, had one unexpected failure, a restricted-join power-level latency race. The gate for 1.0.0 was a security audit instead: release-gate audit run 2 at commit 279d4a0b, @@ -43,7 +44,7 @@ pinned commit against a package whose digest is in the maintainer-signed `SHA256 green run into a maintainer-signed record, tied to the shipped server binary's SHA-256 and attached to the release (ADR-0235, `docs/release-process.md`, "Complement"). Still open: the first green full run against a candidate, and the skip list it produces. The blocker -is waived for 1.0.0 only (see above); it still applies to later releases. Not yet verified: whether +is waived for 1.0.0 and 1.0.1 only (see above); it applies from 1.0.2 onward. Not yet verified: whether Docker's default seccomp profile admits Landlock, and whether the container's memory-lock limit fits the master key. diff --git a/include/merovingian/homeserver/HomeserverRuntime.hpp b/include/merovingian/homeserver/HomeserverRuntime.hpp index 5bbe4f1bc..91acbe807 100644 --- a/include/merovingian/homeserver/HomeserverRuntime.hpp +++ b/include/merovingian/homeserver/HomeserverRuntime.hpp @@ -28,6 +28,7 @@ #include "merovingian/homeserver/LocalRoom.hpp" #include "merovingian/homeserver/MediaUploadNotifier.hpp" #include "merovingian/homeserver/PendingFederatedJoin.hpp" +#include "merovingian/homeserver/RecentlyUnreachableServers.hpp" #include "merovingian/homeserver/RemoteMediaFetchCoalescer.hpp" #include "merovingian/homeserver/RoomMembers.hpp" #include "merovingian/homeserver/RuntimeMutex.hpp" @@ -362,6 +363,9 @@ struct HomeserverRuntime final // start-up snapshot on SQLite) and its sandbox are not asked to decide it again. bool federation_policy_decided_upstream{false}; ClientOutboundProxyPolicy client_outbound_proxy_policy{default_client_outbound_proxy_policy()}; + // ADR-0761: servers that recently gave no answer to a client's remote profile lookup, so the lookups that follow + // are answered 502 at once instead of each waiting out the deadline. Guard every access with `mutex`. + RecentlyUnreachableServers profile_unreachable_servers{}; // One fetch from the origin per remote media file at a time (ADR-0121). Its // mutex is a leaf: never waited on with `mutex` held. Held through a // unique_ptr so leads keep a stable address across a runtime move. diff --git a/include/merovingian/homeserver/OutboundCallResult.hpp b/include/merovingian/homeserver/OutboundCallResult.hpp index 236d5e27d..9d9d461e0 100644 --- a/include/merovingian/homeserver/OutboundCallResult.hpp +++ b/include/merovingian/homeserver/OutboundCallResult.hpp @@ -20,6 +20,14 @@ struct OutboundCallResult final // The `errcode` of a non-2xx answer's Matrix error body when it is `M_INVITE_BLOCKED` (SS7), else empty. A remote // body is untrusted, so only this one fixed value is carried, never the remote's text. std::string remote_errcode{}; + // ADR-0761: true when the destination itself gave no answer: discovery found no server, or the request got no + // connection, no verified TLS session or no response in time. False on success, on any HTTP answer (including a + // redirect or an oversized body) and on a refusal made here before anything was sent (policy, deadline, key). + bool remote_unreachable{false}; + // ADR-0761: true when the destination sent an HTTP response, whatever this server then made of it: a success, any + // other status, a redirect it refused to follow or a body too large to accept. `remote_status` is 0 for the last + // two, so it cannot stand in for this. + bool remote_answered{false}; }; } // namespace merovingian::homeserver diff --git a/include/merovingian/homeserver/RecentlyUnreachableServers.hpp b/include/merovingian/homeserver/RecentlyUnreachableServers.hpp new file mode 100644 index 000000000..e3edfdede --- /dev/null +++ b/include/merovingian/homeserver/RecentlyUnreachableServers.hpp @@ -0,0 +1,47 @@ +// SPDX-FileCopyrightText: 2026 James Chapman +// SPDX-License-Identifier: GPL-3.0-or-later +#pragma once + +#include +#include +#include +#include +#include + +namespace merovingian::homeserver +{ + +// ADR-0761: servers that gave no answer to a client's remote profile lookup (no address, no connection, no TLS +// session or no response before the deadline), remembered for `ttl` so that the lookups which follow fail at once +// instead of each holding a client outbound slot for the whole deadline. A server that answers at all, even with an +// error, is forgotten. Holds at most `capacity` servers. Not thread-safe: the runtime guards it with runtime.mutex. +class RecentlyUnreachableServers final +{ +public: + using Clock = std::chrono::steady_clock; + + static constexpr auto default_ttl = std::chrono::seconds{60}; + static constexpr auto default_capacity = std::size_t{4096U}; + + RecentlyUnreachableServers() = default; + // `capacity` 0 remembers nothing. + RecentlyUnreachableServers(Clock::duration ttl, std::size_t capacity) noexcept; + + // Whether `server` gave no answer less than `ttl` before `now`. + [[nodiscard]] auto contains(std::string_view server, Clock::time_point now) const -> bool; + // Records that `server` gave no answer at `now`. When the set is full, expired entries are dropped first and then + // the server remembered longest, so a burst of distinct servers cannot grow it past `capacity`. + auto remember(std::string_view server, Clock::time_point now) -> void; + // Forgets `server`, which has answered. + auto forget(std::string_view server) -> void; + [[nodiscard]] auto size() const noexcept -> std::size_t; + +private: + auto make_room(Clock::time_point now) -> void; + + Clock::duration m_ttl{default_ttl}; + std::size_t m_capacity{default_capacity}; + std::unordered_map m_failed_at{}; +}; + +} // namespace merovingian::homeserver diff --git a/meson.build b/meson.build index 0f334b178..652b5b1fd 100644 --- a/meson.build +++ b/meson.build @@ -2,7 +2,7 @@ project( 'merovingian', 'cpp', 'c', - version: '1.0.2', + version: '1.0.3', license: 'GPL-3.0-or-later', meson_version: '>=1.1.0', default_options: [ diff --git a/packaging/freebsd/+MANIFEST b/packaging/freebsd/+MANIFEST index a1eabee9e..3d0de47ba 100644 --- a/packaging/freebsd/+MANIFEST +++ b/packaging/freebsd/+MANIFEST @@ -1,5 +1,5 @@ name: "merovingian" -version: "1.0.2" +version: "1.0.3" origin: net/merovingian comment: Secure Matrix Protocol homeserver desc: "Merovingian is a Matrix Protocol homeserver." diff --git a/packaging/netbsd/Makefile b/packaging/netbsd/Makefile index 313c53462..dd18b8ef4 100644 --- a/packaging/netbsd/Makefile +++ b/packaging/netbsd/Makefile @@ -1,4 +1,4 @@ -DISTNAME= merovingian-1.0.2 +DISTNAME= merovingian-1.0.3 CATEGORIES= net MASTER_SITES= ${MASTER_SITE_GITHUB:=OMG-Software/} GITHUB_PROJECT= Merovingian diff --git a/packaging/opensuse/merovingian.spec b/packaging/opensuse/merovingian.spec index b4ace1a06..109a27b42 100644 --- a/packaging/opensuse/merovingian.spec +++ b/packaging/opensuse/merovingian.spec @@ -1,5 +1,5 @@ Name: merovingian -Version: 1.0.2 +Version: 1.0.3 Release: 1%{?dist} Summary: Secure Matrix Protocol homeserver @@ -115,6 +115,9 @@ fi %{_sysconfdir}/merovingian/merovingian.conf.example %changelog +* Sun Nov 15 2026 James Chapman - 1.0.3-1 +- Merovingian 1.0.3 + * Sat Nov 14 2026 James Chapman - 1.0.2-1 - Merovingian 1.0.2 diff --git a/packaging/rhel/merovingian.spec b/packaging/rhel/merovingian.spec index 2c1ca4cb0..cd5ea6173 100644 --- a/packaging/rhel/merovingian.spec +++ b/packaging/rhel/merovingian.spec @@ -1,5 +1,5 @@ Name: merovingian -Version: 1.0.2 +Version: 1.0.3 Release: 1%{?dist} Summary: Secure Matrix Protocol homeserver @@ -100,6 +100,9 @@ fi %{_sysconfdir}/merovingian/merovingian.conf.example %changelog +* Sun Nov 15 2026 James Chapman - 1.0.3-1 +- Merovingian 1.0.3 + * Sat Nov 14 2026 James Chapman - 1.0.2-1 - Merovingian 1.0.2 diff --git a/packaging/rpm/merovingian.spec b/packaging/rpm/merovingian.spec index e86b3a071..effe5b678 100644 --- a/packaging/rpm/merovingian.spec +++ b/packaging/rpm/merovingian.spec @@ -1,5 +1,5 @@ Name: merovingian -Version: 1.0.2 +Version: 1.0.3 Release: 1%{?dist} Summary: Secure Matrix Protocol homeserver @@ -104,6 +104,9 @@ fi %{_sysconfdir}/merovingian/merovingian.conf.example %changelog +* Sun Nov 15 2026 James Chapman - 1.0.3-1 +- Merovingian 1.0.3 + * Sat Nov 14 2026 James Chapman - 1.0.2-1 - Merovingian 1.0.2 diff --git a/scripts/build-deb.sh b/scripts/build-deb.sh index 04f9ee573..bfc0bc5b5 100755 --- a/scripts/build-deb.sh +++ b/scripts/build-deb.sh @@ -1,9 +1,9 @@ #!/bin/sh # SPDX-License-Identifier: GPL-3.0-or-later -# Build for merovingian 1.0.2. +# Build for merovingian 1.0.3. set -e -VERSION="1.0.2" +VERSION="1.0.3" PKG_NAME="merovingian" STAGING="staging-deb" diff --git a/scripts/build-freebsd-pkg.sh b/scripts/build-freebsd-pkg.sh index 240e68bbe..4fce1078a 100755 --- a/scripts/build-freebsd-pkg.sh +++ b/scripts/build-freebsd-pkg.sh @@ -1,9 +1,9 @@ #!/bin/sh # SPDX-License-Identifier: GPL-3.0-or-later -# Build for merovingian 1.0.2. +# Build for merovingian 1.0.3. set -e -VERSION="1.0.2" +VERSION="1.0.3" STAGING="staging-fbsd" # Clean any state (staged files, build dir) from cached FreeBSD VM runs. diff --git a/scripts/build-netbsd-pkg.sh b/scripts/build-netbsd-pkg.sh index 701ebff62..37f24cedf 100644 --- a/scripts/build-netbsd-pkg.sh +++ b/scripts/build-netbsd-pkg.sh @@ -1,6 +1,6 @@ #!/bin/sh # SPDX-License-Identifier: GPL-3.0-or-later -# Build for merovingian 1.0.2. +# Build for merovingian 1.0.3. # # The checked-in packaging/netbsd/Makefile is the pkgsrc recipe kept for # downstream porters. This script generates a framework-free package so CI @@ -11,7 +11,7 @@ # a staged tree using tar, which produces a package that pkg_add can install. set -e -VERSION="1.0.2" +VERSION="1.0.3" STAGE="staging-netbsd" PREFIX=/usr/pkg diff --git a/scripts/build-openbsd-pkg.sh b/scripts/build-openbsd-pkg.sh index d76bf87ef..a8f791812 100644 --- a/scripts/build-openbsd-pkg.sh +++ b/scripts/build-openbsd-pkg.sh @@ -1,6 +1,6 @@ #!/bin/sh # SPDX-License-Identifier: GPL-3.0-or-later -# Build for merovingian 1.0.2. +# Build for merovingian 1.0.3. # # Standalone pkg_create(1), no ports tree. The checked-in packaging/openbsd/PLIST # is the ports-framework packing list kept for downstream porters; this script @@ -8,7 +8,7 @@ # produce and validate an installable package on every run. set -e -VERSION="1.0.2" +VERSION="1.0.3" STAGE="staging-openbsd" PREFIX=/usr/local diff --git a/scripts/build-opensuse-rpm.sh b/scripts/build-opensuse-rpm.sh index fd29b83a6..c3f3dac99 100644 --- a/scripts/build-opensuse-rpm.sh +++ b/scripts/build-opensuse-rpm.sh @@ -1,6 +1,6 @@ #!/bin/sh # SPDX-License-Identifier: GPL-3.0-or-later -# Build an RPM package for merovingian 1.0.2 on OpenSUSE Tumbleweed or Leap. +# Build an RPM package for merovingian 1.0.3 on OpenSUSE Tumbleweed or Leap. # # Uses packaging/opensuse/merovingian.spec which adapts BuildRequires to # OpenSUSE package names (libopenssl-devel, postgresql-devel, libpng16-devel, @@ -12,7 +12,7 @@ # (.leap16, ADR-0650). set -e -VERSION="1.0.2" +VERSION="1.0.3" DIST="${MEROVINGIAN_RPM_DIST:-.tumbleweed}" case "${DIST}" in .[a-z]*) ;; diff --git a/scripts/build-rhel-rpm.sh b/scripts/build-rhel-rpm.sh index 977c66110..27ce4c891 100644 --- a/scripts/build-rhel-rpm.sh +++ b/scripts/build-rhel-rpm.sh @@ -1,13 +1,13 @@ #!/bin/sh # SPDX-License-Identifier: GPL-3.0-or-later -# Build an RPM package for merovingian 1.0.2 targeting RHEL 10 / AlmaLinux 10. +# Build an RPM package for merovingian 1.0.3 targeting RHEL 10 / AlmaLinux 10. # # Differs from build-rpm.sh only in the spec file used: packaging/rhel/merovingian.spec # omits catch-devel from BuildRequires (Catch2 is not reliably available in EPEL 10 # and is not needed when -Dbuild_tests=false). set -e -VERSION="1.0.2" +VERSION="1.0.3" mkdir -p "${HOME}/rpmbuild/BUILD" mkdir -p "${HOME}/rpmbuild/BUILDROOT" diff --git a/scripts/build-rpm.sh b/scripts/build-rpm.sh index e23336ea5..c6a63b479 100755 --- a/scripts/build-rpm.sh +++ b/scripts/build-rpm.sh @@ -1,9 +1,9 @@ #!/bin/sh # SPDX-License-Identifier: GPL-3.0-or-later -# Build an RPM package for merovingian 1.0.2 using rpmbuild. +# Build an RPM package for merovingian 1.0.3 using rpmbuild. set -e -VERSION="1.0.2" +VERSION="1.0.3" # 1. Create rpmbuild directory tree mkdir -p "${HOME}/rpmbuild/BUILD" diff --git a/scripts/build-static-linux.sh b/scripts/build-static-linux.sh index 2f9620722..934084172 100755 --- a/scripts/build-static-linux.sh +++ b/scripts/build-static-linux.sh @@ -3,7 +3,7 @@ # Build a portable Linux fallback tarball with musl-linked static PIE binaries. set -eu -VERSION="${MEROVINGIAN_VERSION:-1.0.2}" +VERSION="${MEROVINGIAN_VERSION:-1.0.3}" BUILD_DIR="${BUILD_DIR:-build-static-linux}" STAGING="staging-static-linux" PACKAGE_ROOT="merovingian-${VERSION}-linux-static-x86_64" diff --git a/src/db_migrate.cpp b/src/db_migrate.cpp index a55a54ff3..776d6002c 100644 --- a/src/db_migrate.cpp +++ b/src/db_migrate.cpp @@ -23,7 +23,7 @@ namespace { -constexpr auto version = std::string_view{"1.0.2"}; +constexpr auto version = std::string_view{"1.0.3"}; auto print_help() -> void { diff --git a/src/homeserver/AGENTS.md b/src/homeserver/AGENTS.md index e1c7c7d84..15cafeaf6 100644 --- a/src/homeserver/AGENTS.md +++ b/src/homeserver/AGENTS.md @@ -612,6 +612,11 @@ only endpoints the spec requires and the origin may be asked for: supply, never at a prev_event that was already fetched; held PDUs are placed oldest first afterwards. - A snapshot fetches every missing named event before it stores any, so an auth chain the origin cannot supply leaves nothing stored. +- A recovery asks the origin for a listed state event at most once (ADR-0760). A state event a `/state_ids` snapshot + listed and the origin did not supply is kept in `BackfillBudget::unsupplied_events`, and a later snapshot that lists + it (neither held nor stored since) ends before it fetches any of its events. Only `backfill_state_ids_snapshot`'s listed-event fetch records or consults the set: the + per-reference fetch and a snapshot's target fetch ask as before, since the target is fetched again once its state is + known (ADR-0069). The set lives only as long as the recovery's `BackfillBudget`; the deferred retry asks again. - `verify_and_store_backfilled_event` is called with `runtime.mutex` released. It takes the room stripe and then `runtime.mutex` itself once its last signature check (which may fetch a key) is done, and holds them through the authorization reads and the write. Never read the store in it before that point. diff --git a/src/homeserver/ClientServerRuntime.cpp b/src/homeserver/ClientServerRuntime.cpp index 19fcb4ae7..2f42e00e5 100644 --- a/src/homeserver/ClientServerRuntime.cpp +++ b/src/homeserver/ClientServerRuntime.cpp @@ -107,6 +107,7 @@ #include "merovingian/homeserver/PresenceService.hpp" #include "merovingian/homeserver/PublicRooms.hpp" #include "merovingian/homeserver/PushRuleUnreadClassifier.hpp" +#include "merovingian/homeserver/RecentlyUnreachableServers.hpp" #include "merovingian/homeserver/RegistrationValidationSession.hpp" #include "merovingian/homeserver/RemoteDeviceListScope.hpp" #include "merovingian/homeserver/RemoteInviteResult.hpp" @@ -12381,6 +12382,15 @@ namespace { return dispatch_err(req, rt, 404U, "M_NOT_FOUND", "profile field not found"); } + auto const destination = std::string{server_name_from_user_id(target_user)}; + // ADR-0761: a server that gave no answer moments ago is not asked again until the record expires. Clients + // look profiles up by themselves (member lists, invite dialogs), so each would otherwise wait out the deadline. + if (rt.homeserver.profile_unreachable_servers.contains(destination, RecentlyUnreachableServers::Clock::now())) + { + LOG_DEBUG("event=profile.proxy destination=" + destination + + " reason=gave no answer recently; not asked again yet"); + return dispatch_err(req, rt, 502U, "M_UNKNOWN", "The remote server did not answer a recent request"); + } auto const proxy_deadline = effective_client_outbound_deadline( rt.homeserver, rt.homeserver.client_outbound_proxy_policy.directory_deadline_seconds); wire_federation_callbacks(rt.homeserver); @@ -12394,8 +12404,7 @@ namespace { target += "&field=" + std::string{field}; } - auto const tx = federation::make_outbound_transaction(std::string{server_name_from_user_id(target_user)}, "GET", - target, our_server, {}); + auto const tx = federation::make_outbound_transaction(destination, "GET", target, our_server, {}); auto const result = [&] { // Re-acquire before returning: dispatch_resp/dispatch_err read reloadable runtime state. auto const released = merovingian::homeserver::RuntimeLockRelease{guard}; @@ -12403,6 +12412,14 @@ namespace key_id, std::move(secret), "profile.proxy", proxy_deadline); }(); + if (result.remote_unreachable) + { + rt.homeserver.profile_unreachable_servers.remember(destination, RecentlyUnreachableServers::Clock::now()); + } + else if (result.remote_answered) + { + rt.homeserver.profile_unreachable_servers.forget(destination); + } if (result.remote_status == 404U) { return dispatch_err(req, rt, 404U, "M_NOT_FOUND", "profile not found"); diff --git a/src/homeserver/LocalHttpRouter.cpp b/src/homeserver/LocalHttpRouter.cpp index cc43500fb..ed54d6073 100644 --- a/src/homeserver/LocalHttpRouter.cpp +++ b/src/homeserver/LocalHttpRouter.cpp @@ -3651,6 +3651,17 @@ namespace std::size_t max_calls{}; std::chrono::steady_clock::time_point deadline{}; bool key_resolution_locally_denied{}; + // ADR-0760: state events a /state_ids snapshot listed that the origin, asked with GET /event during this + // recovery, did not supply (refused, missing, unusable or no answer). A later snapshot in the same recovery + // that lists one of them cannot be completed, so it ends without asking again. Only listed state events are + // recorded: the per-reference fetch and a snapshot's own target are asked for separately, as before. Never + // outlives the recovery; the next deferred retry asks again. + std::unordered_set unsupplied_events{}; + + [[nodiscard]] auto origin_did_not_supply(std::string_view event_id) const -> bool + { + return unsupplied_events.contains(std::string{event_id}); + } [[nodiscard]] auto within_deadline() const noexcept -> bool { @@ -4753,6 +4764,34 @@ namespace // event the origin cannot supply (a gap in the auth chain, a state event // it will not serve) ends the attempt with nothing stored. The fetches // are bounded by max_snapshot_outbound_calls and the shared budget. + // ADR-0760: a listed state event the origin already failed to supply in this recovery, and that is neither + // held nor stored since, means this snapshot cannot be completed. End it before fetching any of the others, + // which phase 1 would fetch only to discard. The failure itself was logged when it happened. + if (!budget.unsupplied_events.empty()) + { + auto const stripe = std::hash{}(std::string{room_id}) % room_mutex_stripe_count; + for (auto const& id : needed_ids) + { + if (!budget.origin_did_not_supply(id) || held.find(id) != held.end()) + { + continue; + } + { + auto stripe_guard = std::unique_lock{runtime.room_stripe_mutexes[stripe]}; + auto global_guard = RuntimeLock{runtime.mutex}; + if (runtime.database.persistent_store.events.contains(id)) + { + continue; + } + } + LOG_DEBUG("backfill_state_ids_snapshot: a listed event was already not supplied in this recovery; " + "room_id=" + + std::string{room_id} + " target_event_id=" + std::string{target_event_id} + + " event_id=" + id); + return false; + } + } + struct SnapshotFetch final { std::string id{}; @@ -4796,6 +4835,7 @@ namespace fetch_snapshot_event_by_id(runtime, room_id, origin, id, policy, snapshot_calls, budget); if (!supplied.has_value()) { + budget.unsupplied_events.emplace(id); LOG_WARNING("backfill_state_ids_snapshot: a listed event could not be fetched with /event; " "room_id=" + std::string{room_id} + " target_event_id=" + std::string{target_event_id} + diff --git a/src/homeserver/RecentlyUnreachableServers.cpp b/src/homeserver/RecentlyUnreachableServers.cpp new file mode 100644 index 000000000..2c9c0e9f6 --- /dev/null +++ b/src/homeserver/RecentlyUnreachableServers.cpp @@ -0,0 +1,77 @@ +// SPDX-FileCopyrightText: 2026 James Chapman +// SPDX-License-Identifier: GPL-3.0-or-later + +#include "merovingian/homeserver/RecentlyUnreachableServers.hpp" + +#include +#include +#include + +namespace merovingian::homeserver +{ + +RecentlyUnreachableServers::RecentlyUnreachableServers(Clock::duration ttl, std::size_t capacity) noexcept + : m_ttl{ttl} + , m_capacity{capacity} +{ +} + +auto RecentlyUnreachableServers::contains(std::string_view server, Clock::time_point now) const -> bool +{ + auto const found = m_failed_at.find(std::string{server}); + return found != m_failed_at.end() && now - found->second < m_ttl; +} + +auto RecentlyUnreachableServers::remember(std::string_view server, Clock::time_point now) -> void +{ + if (m_capacity == 0U) + { + return; + } + auto key = std::string{server}; + if (auto const found = m_failed_at.find(key); found != m_failed_at.end()) + { + found->second = now; + return; + } + make_room(now); + m_failed_at.emplace(std::move(key), now); +} + +auto RecentlyUnreachableServers::forget(std::string_view server) -> void +{ + m_failed_at.erase(std::string{server}); +} + +auto RecentlyUnreachableServers::size() const noexcept -> std::size_t +{ + return m_failed_at.size(); +} + +// Leaves room for one more entry. The scan is linear in `capacity`; it runs only on a lookup that has already spent +// a client outbound slot and failed, so it is bounded by that admission. +auto RecentlyUnreachableServers::make_room(Clock::time_point now) -> void +{ + if (m_failed_at.size() < m_capacity) + { + return; + } + std::erase_if(m_failed_at, [&](auto const& entry) { + return now - entry.second >= m_ttl; + }); + if (m_failed_at.size() < m_capacity) + { + return; + } + auto const oldest = std::ranges::min_element(m_failed_at, {}, [](auto const& entry) { + return entry.second; + }); + // The set is not empty here (size >= capacity > 0), but GCC 16 at -O3 cannot see that and reports erasing end() + // as a null dereference. + if (oldest != m_failed_at.end()) + { + m_failed_at.erase(oldest); + } +} + +} // namespace merovingian::homeserver diff --git a/src/homeserver/RoomService.cpp b/src/homeserver/RoomService.cpp index 324cbbac4..e61c9cb38 100644 --- a/src/homeserver/RoomService.cpp +++ b/src/homeserver/RoomService.cpp @@ -2828,6 +2828,37 @@ namespace return federation::pdu_format_violation(canonicaljson::Value{completed_event}, policy); } + // ADR-0761: the transport failures in which the destination gave no answer. A redirect or an oversized body is + // an answer, and a request refused before any I/O (bad URL, method or header) says nothing about the destination. + [[nodiscard]] auto destination_gave_no_answer(http::OutboundError error) noexcept -> bool + { + switch (error) + { + case http::OutboundError::unresolved_host: + case http::OutboundError::tls_verification_failed: + case http::OutboundError::connection_failed: + case http::OutboundError::timeout: + case http::OutboundError::network_error: + return true; + case http::OutboundError::none: + case http::OutboundError::invalid_url: + case http::OutboundError::invalid_method: + case http::OutboundError::https_required: + case http::OutboundError::redirect_rejected: + case http::OutboundError::response_too_large: + case http::OutboundError::invalid_header: + return false; + } + return false; + } + + // ADR-0761: the transport failures that still mean the destination sent an HTTP response: a redirect this server + // will not follow, and a response too large to accept. + [[nodiscard]] auto destination_sent_a_response(http::OutboundError error) noexcept -> bool + { + return error == http::OutboundError::redirect_rejected || error == http::OutboundError::response_too_large; + } + } // namespace [[nodiscard]] auto perform_outbound_call(HomeserverRuntime& runtime, OutboundAnswerTrust trust, @@ -2912,7 +2943,7 @@ namespace {"destination", transaction.destination, false}, {"reason", "server discovery failed", false} }); - return {false, "server discovery failed"}; + return {false, "server discovery failed", 0U, {}, true}; } resolved_host = resolution.resolved_host; tls_server_name = resolution.tls_server_name; @@ -3019,9 +3050,11 @@ namespace outcome.error_detail.empty() ? "remote server returned " + std::to_string(outcome.response.status) : outcome.error_detail, outcome.ok ? outcome.response.status : std::uint16_t{0U}, - outcome.ok ? remote_invite_blocked_errcode(outcome.response.body) : std::string{}}; + outcome.ok ? remote_invite_blocked_errcode(outcome.response.body) : std::string{}, + !outcome.ok && destination_gave_no_answer(outcome.error), + outcome.ok || destination_sent_a_response(outcome.error)}; } - return {true, outcome.response.body, outcome.response.status}; + return {true, outcome.response.body, outcome.response.status, {}, false, true}; } // How long a published signing key advertises itself as valid. Federation peers diff --git a/src/homeserver/meson.build b/src/homeserver/meson.build index 13adb22a9..336f973ac 100644 --- a/src/homeserver/meson.build +++ b/src/homeserver/meson.build @@ -50,6 +50,7 @@ homeserver_sources = files( 'StoredRoomVersion.cpp', 'StreamTokenEpoch.cpp', 'RemoteMediaFetchScope.cpp', + 'RecentlyUnreachableServers.cpp', 'RequestLockScope.cpp', 'RuntimeLockRelease.cpp', 'RestrictedJoin.cpp', diff --git a/src/main.cpp b/src/main.cpp index 54ee2ee72..9d4d735a9 100644 --- a/src/main.cpp +++ b/src/main.cpp @@ -81,7 +81,7 @@ namespace { -constexpr auto version = std::string_view{"1.0.2"}; +constexpr auto version = std::string_view{"1.0.3"}; struct BootstrapConfigResult final { diff --git a/tests/integration/test_client_server_flow.cpp b/tests/integration/test_client_server_flow.cpp index 600dfcb8a..8d2e06e6e 100644 --- a/tests/integration/test_client_server_flow.cpp +++ b/tests/integration/test_client_server_flow.cpp @@ -14,6 +14,7 @@ #include "merovingian/config/LogModulesConfig.hpp" #include "merovingian/config/SecurityConfig.hpp" #include "merovingian/config/ServerConfig.hpp" +#include "merovingian/core/SocketHandle.hpp" #include "merovingian/database/ExaminedRows.hpp" #include "merovingian/database/PersistentMembership.hpp" #include "merovingian/database/PersistentStateEvent.hpp" @@ -25,6 +26,7 @@ #include "merovingian/homeserver/ClientServerStartResult.hpp" #include "merovingian/homeserver/LocalHttpRequest.hpp" #include "merovingian/homeserver/OutboundDeadline.hpp" +#include "merovingian/homeserver/RecentlyUnreachableServers.hpp" #include "merovingian/homeserver/RemoteTimestampEvent.hpp" #include "merovingian/homeserver/RuntimeStartResult.hpp" #include "merovingian/homeserver/AuthService.hpp" @@ -43,6 +45,7 @@ #include #include +#include #include #include #include @@ -1129,6 +1132,160 @@ SCENARIO("GET /profile/{userId} for a user of another server asks that server ov } } +// ADR-0761: clients look remote profiles up by themselves, from member lists and invite dialogs, so a server that +// gives no answer would otherwise cost one outbound slot and the whole deadline per lookup. Once it has given no +// answer it is not asked again for a minute; a server that answers, even 404, is asked every time. +SCENARIO("GET /profile/{userId} does not ask again a server that just gave no answer", + "[homeserver][client-server][profile][fed-endpoints][adr0761]") +{ + GIVEN("a remote server that accepts each connection and closes it without a TLS session or an answer") + { + auto const config = registration_enabled_config(); + auto started = merovingian::homeserver::start_client_server(config); + REQUIRE(started.started); + auto& rt = started.runtime; + + auto certificate = merovingian::tests::tls_mock::write_test_tls_certificate("localhost"); + auto acceptor = merovingian::net::TcpAcceptor{}; + REQUIRE(acceptor.bind("127.0.0.1", 0U).ok); + rt.homeserver.test_forced_outbound_resolution["silent.example.net"] = + merovingian::homeserver::TestOnlyForcedOutboundResolution{ + "localhost", acceptor.bound_port(), {"127.0.0.1"}, certificate.certificate_pem}; + + auto connections = std::atomic{0U}; + auto stop = std::atomic{false}; + auto server_thread = std::thread{[&]() { + while (!stop.load()) + { + auto const fd = merovingian::tests::tls_mock::accept_loopback(acceptor, 50); + if (fd >= 0) + { + auto const closed_at_once = merovingian::core::SocketHandle{fd}; + connections.fetch_add(1U); + } + } + }}; + auto const join_server = merovingian::tests::tls_mock::ScopedThreadJoin{server_thread}; + struct StopOnExit final + { + std::atomic& flag; + ~StopOnExit() noexcept + { + flag.store(true); + } + }; + auto const stop_on_exit = StopOnExit{stop}; + + WHEN("a client reads profiles of that server's users twice in a row") + { + auto const first = merovingian::homeserver::handle_client_server_request( + rt, {"GET", "/_matrix/client/v3/profile/%40bob%3Asilent.example.net", {}, {}}); + auto const second = merovingian::homeserver::handle_client_server_request( + rt, {"GET", "/_matrix/client/v3/profile/%40carol%3Asilent.example.net/displayname", {}, {}}); + + THEN("both are answered 502, and only the first reached the server") + { + REQUIRE(first.response.status == 502U); + REQUIRE(second.response.status == 502U); + REQUIRE(second.response.body.find("M_UNKNOWN") != std::string::npos); + REQUIRE(connections.load() == 1U); + } + } + } + + GIVEN("a remote server that answers each profile query with 404") + { + auto const config = registration_enabled_config(); + auto started = merovingian::homeserver::start_client_server(config); + REQUIRE(started.started); + auto& rt = started.runtime; + + auto certificate = merovingian::tests::tls_mock::write_test_tls_certificate("localhost"); + auto tls = merovingian::homeserver::make_tls_server_context(certificate.certificate_file, + certificate.private_key_file); + REQUIRE(tls.ok()); + auto tls_context = std::move(*tls.context); + auto acceptor = merovingian::net::TcpAcceptor{}; + REQUIRE(acceptor.bind("127.0.0.1", 0U).ok); + rt.homeserver.test_forced_outbound_resolution["remote.example.net"] = + merovingian::homeserver::TestOnlyForcedOutboundResolution{ + "localhost", acceptor.bound_port(), {"127.0.0.1"}, certificate.certificate_pem}; + auto const not_found = merovingian::tests::tls_mock::json_http_response( + "404 Not Found", R"({"errcode":"M_NOT_FOUND","error":"no such user"})"); + auto const responses = std::vector>{ + {"user_id=%40nobody%3Aremote.example.net", not_found}, + {"user_id=%40nobody%3Aremote.example.net", not_found}, + }; + auto captured = std::vector{}; + auto server_thread = std::thread{[&]() { + merovingian::tests::tls_mock::run_path_dispatch_tls_server(acceptor, tls_context, responses, &captured); + }}; + auto const join_server = merovingian::tests::tls_mock::ScopedThreadJoin{server_thread}; + + WHEN("a client reads the same missing profile twice") + { + auto const first = merovingian::homeserver::handle_client_server_request( + rt, {"GET", "/_matrix/client/v3/profile/%40nobody%3Aremote.example.net", {}, {}}); + auto const second = merovingian::homeserver::handle_client_server_request( + rt, {"GET", "/_matrix/client/v3/profile/%40nobody%3Aremote.example.net", {}, {}}); + + THEN("both are 404 and both were asked of the server") + { + REQUIRE(first.response.status == 404U); + REQUIRE(second.response.status == 404U); + REQUIRE(captured.size() == 2U); + } + } + } + + GIVEN("a server still in the record, its entry expired, that now answers with a redirect") + { + auto const config = registration_enabled_config(); + auto started = merovingian::homeserver::start_client_server(config); + REQUIRE(started.started); + auto& rt = started.runtime; + + auto certificate = merovingian::tests::tls_mock::write_test_tls_certificate("localhost"); + auto tls = merovingian::homeserver::make_tls_server_context(certificate.certificate_file, + certificate.private_key_file); + REQUIRE(tls.ok()); + auto tls_context = std::move(*tls.context); + auto acceptor = merovingian::net::TcpAcceptor{}; + REQUIRE(acceptor.bind("127.0.0.1", 0U).ok); + rt.homeserver.test_forced_outbound_resolution["moved.example.net"] = + merovingian::homeserver::TestOnlyForcedOutboundResolution{ + "localhost", acceptor.bound_port(), {"127.0.0.1"}, certificate.certificate_pem}; + // An entry that has expired but not been dropped, as an overlapping lookup's failure leaves while this one + // is in flight: the lookup goes out, and only the answer can remove the entry. + rt.homeserver.profile_unreachable_servers.remember( + "moved.example.net", + merovingian::homeserver::RecentlyUnreachableServers::Clock::now() - std::chrono::minutes{2}); + REQUIRE(rt.homeserver.profile_unreachable_servers.size() == 1U); + auto const responses = std::vector>{ + {"user_id=%40dave%3Amoved.example.net", + merovingian::tests::tls_mock::json_http_response("302 Found", R"({})")}, + }; + auto captured = std::vector{}; + auto server_thread = std::thread{[&]() { + merovingian::tests::tls_mock::run_path_dispatch_tls_server(acceptor, tls_context, responses, &captured); + }}; + auto const join_server = merovingian::tests::tls_mock::ScopedThreadJoin{server_thread}; + + WHEN("a client reads a profile of that server's user") + { + auto const response = merovingian::homeserver::handle_client_server_request( + rt, {"GET", "/_matrix/client/v3/profile/%40dave%3Amoved.example.net", {}, {}}); + + THEN("the redirect is not followed, but the server answered, so it is forgotten") + { + REQUIRE(response.response.status != 200U); + REQUIRE(captured.size() == 1U); + REQUIRE(rt.homeserver.profile_unreachable_servers.size() == 0U); + } + } + } +} + // Spec: Matrix Client-Server API v1.19, GET /profile/{userId}; Server-Server API v1.19, query/profile // ("Additional profile fields"). A remote answer is untrusted: only m.tz as a string and custom keys that meet // the key grammar are relayed, never null, and nothing else. diff --git a/tests/integration/test_pdu_ingestion_backfill_flow.cpp b/tests/integration/test_pdu_ingestion_backfill_flow.cpp index 1112d4270..f2ae6c7d6 100644 --- a/tests/integration/test_pdu_ingestion_backfill_flow.cpp +++ b/tests/integration/test_pdu_ingestion_backfill_flow.cpp @@ -4078,6 +4078,120 @@ SCENARIO("A snapshot whose auth chain the origin will not supply stores nothing } } +[[nodiscard]] auto event_fetches_of(std::vector const& requests, std::string const& event_id) + -> std::size_t +{ + return count_requests_starting_with(requests, "GET /_matrix/federation/v1/event/" + + core::percent_encode_path_component(event_id) + " "); +} + +// Spec: server-server-api.md GET /state_ids and GET /event/{eventId}. A snapshot is usable only when every state event +// it lists can be fetched, and the origin decides what it serves. ADR-0760: once the origin has not supplied a listed +// state event, asking it again in the same recovery only repeats the refusal, so every later snapshot that lists the +// event ends without the request. A later recovery (the deferred retry) asks again. +SCENARIO("A recovery asks the origin for a listed state event it did not supply at most once", + "[pdu_ingestion][backfill][adr0760]") +{ + GIVEN("a PDU with two unknown prev_events, whose /state_ids answers both name a state event the origin refuses") + { + auto fixture = RecoveryFixture{"!adr0760-refused:local.example.org"}; + auto const& room = fixture.room_id; + auto const first_gap = make_remote_message_pdu(room, {fixture.bob_member_id()}, fixture.auth_ids(), 4, 10); + auto const second_gap = make_remote_message_pdu(room, {fixture.bob_member_id()}, fixture.auth_ids(), 4, 11); + auto const refused = + make_remote_event_pdu(room, "m.room.member", std::string{remote_member}, remote_member, member_content("R"), + {fixture.bob_member_id()}, fixture.auth_ids(), 4, 12); + auto const pdu = + make_remote_message_pdu(room, {first_gap.event_id, second_gap.event_id}, fixture.auth_ids(), 5, 20); + + auto pdu_ids = fixture.genesis_state_ids(); + pdu_ids.push_back(refused.event_id); + auto script = PeerScript{}; + script.state_ids[first_gap.event_id] = make_state_ids_response(pdu_ids, fixture.auth_ids()); + script.state_ids[second_gap.event_id] = make_state_ids_response(pdu_ids, fixture.auth_ids()); + auto const peer = fixture.serve(script); + + WHEN("the PDU is ingested") + { + auto const result = fixture.ingest(pdu); + auto const requests = peer->requests(); + + THEN("the state is asked at both gaps, but the refused event is asked for once") + { + CHECK(state_asked_at(requests, first_gap.event_id)); + CHECK(state_asked_at(requests, second_gap.event_id)); + CHECK(event_fetches_of(requests, refused.event_id) == 1U); + } + + THEN("each prev_event the origin did not supply was asked for once, and the PDU is not placed") + { + CHECK(event_fetches_of(requests, first_gap.event_id) == 1U); + CHECK(event_fetches_of(requests, second_gap.event_id) == 1U); + CHECK(result.status != PduIngestionStatus::accepted); + CHECK_FALSE(stored_status(fixture.runtime(), refused.event_id).has_value()); + } + + AND_WHEN("the PDU is ingested again, as the deferred retry does") + { + std::ignore = fixture.ingest(pdu); + auto const later = peer->requests(); + + THEN("the new recovery asks the origin for the refused event again, once") + { + CHECK(event_fetches_of(later, refused.event_id) == 2U); + } + } + } + } +} + +// ADR-0760: phase 1 stores nothing until every listed event is in hand, so a later snapshot that would fetch the +// events listed before a refused one again, only to discard them, ends before fetching any of them. +SCENARIO("A later snapshot listing a state event the origin refused fetches none of its other events", + "[pdu_ingestion][backfill][adr0760]") +{ + GIVEN("two gaps whose /state_ids answers list a state event the origin serves before one it refuses") + { + auto fixture = RecoveryFixture{"!adr0760-refused-after:local.example.org"}; + auto const& room = fixture.room_id; + auto const first_gap = make_remote_message_pdu(room, {fixture.bob_member_id()}, fixture.auth_ids(), 4, 10); + auto const second_gap = make_remote_message_pdu(room, {fixture.bob_member_id()}, fixture.auth_ids(), 4, 11); + auto const served = make_remote_event_pdu(room, "m.room.member", std::string{"@alice:remote.example.org"}, + "@alice:remote.example.org", member_content("A"), + {fixture.bob_member_id()}, fixture.auth_ids(), 4, 12); + auto const refused = + make_remote_event_pdu(room, "m.room.member", std::string{remote_member}, remote_member, member_content("R"), + {fixture.bob_member_id()}, fixture.auth_ids(), 4, 13); + auto const pdu = + make_remote_message_pdu(room, {first_gap.event_id, second_gap.event_id}, fixture.auth_ids(), 5, 20); + + auto pdu_ids = fixture.genesis_state_ids(); + pdu_ids.push_back(served.event_id); + pdu_ids.push_back(refused.event_id); + auto script = PeerScript{}; + script.events[served.event_id] = served.json; + script.state_ids[first_gap.event_id] = make_state_ids_response(pdu_ids, fixture.auth_ids()); + script.state_ids[second_gap.event_id] = make_state_ids_response(pdu_ids, fixture.auth_ids()); + auto const peer = fixture.serve(script); + + WHEN("the PDU is ingested") + { + auto const result = fixture.ingest(pdu); + auto const requests = peer->requests(); + + THEN("the state is asked at both gaps, but each listed event is fetched only by the first snapshot") + { + CHECK(state_asked_at(requests, first_gap.event_id)); + CHECK(state_asked_at(requests, second_gap.event_id)); + CHECK(event_fetches_of(requests, served.event_id) == 1U); + CHECK(event_fetches_of(requests, refused.event_id) == 1U); + CHECK(result.status != PduIngestionStatus::accepted); + CHECK_FALSE(stored_status(fixture.runtime(), served.event_id).has_value()); + } + } + } +} + // Spec: server-server-api.md "Rejection": "Subsequent events from other servers // that reference rejected events should be allowed if they still pass the auth // rules"; and room version auth rule 2.3 (rooms/v10.md): an event whose diff --git a/tests/meson.build b/tests/meson.build index d5485ce80..cbc55647d 100644 --- a/tests/meson.build +++ b/tests/meson.build @@ -545,6 +545,7 @@ shell = find_program('sh') 'unit/test_database_schema.cpp', 'unit/test_homeserver_auth_service.cpp', 'unit/test_homeserver_media_service.cpp', + 'unit/test_homeserver_recently_unreachable_servers.cpp', 'unit/test_streamed_body.cpp', 'unit/test_client_outbound_proxy.cpp', 'unit/test_federation_server_name.cpp', diff --git a/tests/unit/test_homeserver_recently_unreachable_servers.cpp b/tests/unit/test_homeserver_recently_unreachable_servers.cpp new file mode 100644 index 000000000..87d24653d --- /dev/null +++ b/tests/unit/test_homeserver_recently_unreachable_servers.cpp @@ -0,0 +1,125 @@ +// SPDX-FileCopyrightText: 2026 James Chapman +// SPDX-License-Identifier: GPL-3.0-or-later +// +// ADR-0761: the bounded, expiring record of servers that gave no answer to a remote profile lookup. +// Tags: [homeserver][adr0761]. + +#include "merovingian/homeserver/RecentlyUnreachableServers.hpp" + +#include + +#include +#include +#include + +using merovingian::homeserver::RecentlyUnreachableServers; + +namespace +{ + +constexpr auto ttl = std::chrono::seconds{60}; +auto const start = RecentlyUnreachableServers::Clock::time_point{} + std::chrono::hours{1}; + +} // namespace + +SCENARIO("A server that gave no answer is remembered until its record expires", "[homeserver][adr0761]") +{ + GIVEN("a record with a 60 second lifetime") + { + auto servers = RecentlyUnreachableServers{ttl, 8U}; + + WHEN("a server is remembered") + { + servers.remember("down.example.org", start); + + THEN("it is remembered just before the lifetime ends and not at its end") + { + REQUIRE(servers.contains("down.example.org", start)); + REQUIRE(servers.contains("down.example.org", start + ttl - std::chrono::milliseconds{1})); + REQUIRE_FALSE(servers.contains("down.example.org", start + ttl)); + } + + THEN("another server is not remembered") + { + REQUIRE_FALSE(servers.contains("up.example.org", start)); + } + } + + WHEN("a server is remembered and fails again later") + { + servers.remember("down.example.org", start); + servers.remember("down.example.org", start + std::chrono::seconds{50}); + + THEN("the later failure starts its lifetime again, and it is still one entry") + { + REQUIRE(servers.contains("down.example.org", start + std::chrono::seconds{100})); + REQUIRE(servers.size() == 1U); + } + } + + WHEN("a remembered server answers and is forgotten") + { + servers.remember("down.example.org", start); + servers.forget("down.example.org"); + + THEN("it is no longer remembered") + { + REQUIRE_FALSE(servers.contains("down.example.org", start)); + REQUIRE(servers.size() == 0U); + } + } + } +} + +SCENARIO("The record of servers that gave no answer never grows past its capacity", "[homeserver][adr0761]") +{ + GIVEN("a record that holds two servers, already full") + { + auto servers = RecentlyUnreachableServers{ttl, 2U}; + servers.remember("first.example.org", start); + servers.remember("second.example.org", start + std::chrono::seconds{10}); + + WHEN("a third server fails while both are still live") + { + servers.remember("third.example.org", start + std::chrono::seconds{20}); + + THEN("the server remembered longest is dropped and the record still holds two") + { + auto const now = start + std::chrono::seconds{20}; + REQUIRE(servers.size() == 2U); + REQUIRE_FALSE(servers.contains("first.example.org", now)); + REQUIRE(servers.contains("second.example.org", now)); + REQUIRE(servers.contains("third.example.org", now)); + } + } + + WHEN("a third server fails after the first record has expired") + { + auto const now = start + ttl + std::chrono::seconds{5}; + servers.remember("third.example.org", now); + + THEN("the expired record makes room and the live one is kept") + { + REQUIRE(servers.size() == 2U); + REQUIRE(servers.contains("second.example.org", now)); + REQUIRE(servers.contains("third.example.org", now)); + } + } + } + + GIVEN("a record with no capacity") + { + auto servers = RecentlyUnreachableServers{ttl, 0U}; + + WHEN("a server fails") + { + servers.remember("down.example.org", start); + + THEN("nothing is remembered") + { + REQUIRE(servers.size() == 0U); + REQUIRE_FALSE(servers.contains("down.example.org", start)); + } + } + } +}