Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 25 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,31 @@ Changes from 1.0.0, the first production release, onwards; the format is set out
Changes before 1.0.0 are in [CHANGELOG-pre-release.md](CHANGELOG-pre-release.md), which is no longer added to
([ADR-0740](docs/adr/0740-the-changelog-starts-at-1-0-0-and-the-pre-release-entries-are-a-frozen-record.md)).

# Version 1.0.3

## Features

* `docs/release-process.md` and the production milestone record that 1.0.1, like 1.0.0, was tagged without a green
Complement run, and that the 1.0.0 GitHub release is withdrawn (ADR-0744).

## Security Fixes

* None.

## Bug fixes

* A backfill recovery asks the origin for a listed state event at most once: a later state snapshot listing one the
origin already refused ends without asking, so it is fetched and logged once, not once per gap (ADR-0760).
* A remote profile lookup for a server that just gave no answer is answered 502 at once for the next 60 s, so client
lookups to a dead server no longer each hold an outbound slot for the whole deadline (ADR-0761).

## Release notes

1.0.3 cuts repeated federation work and log noise seen on a production server: a refused history event is asked for
once per recovery, and a server that gives no answer to profile lookups is left alone for a minute. No configuration
change is needed. 1.0.1 was also tagged without a green Complement run, and the 1.0.0 GitHub release is withdrawn
because it ships the vulnerabilities 1.0.1 fixes; a 1.0.0 server should upgrade.

# Version 1.0.2

## Features
Expand Down
55 changes: 55 additions & 0 deletions docs/adr/0744-the-complement-blocker-is-also-waived-for-1-0-1.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
# The Complement blocker is also waived for 1.0.1

* Status: accepted
* Date: 2026-11-13

## Context and Problem Statement

[ADR-0648](0648-the-complement-blocker-is-waived-for-1-0-0-only.md) waived the green Complement run for 1.0.0 only, and
`docs/release-process.md` said the waiver applies to no later release.

1.0.1 fixes the 49 findings of the run-3 security audit of the 1.0.0 release candidate, 9 of them high. The 1.0.0 release
ships those vulnerabilities, so the owner decided on 2026-11-13 to tag 1.0.1 at once and remove the 1.0.0 release. No
Complement run exists for 1.0.1. Can 1.0.1 be tagged without one?

## Considered Options

* Waive the blocker for 1.0.1 as well; it stays open for every later release
* Hold the 1.0.1 tag until a green Complement run exists

## Decision Outcome

Chosen option: "Waive the blocker for 1.0.1 as well".

What is waived: the green Complement run before the 1.0.1 tag. 1.0.1 carries no Complement evidence and no signed
Complement record is attached to it.

Why: the only published release, 1.0.0, has known vulnerabilities and 1.0.1 fixes them. Holding the fix for a Complement
run would leave 1.0.0 as the only release to install. The run-3 audit and this project's own suites gate 1.0.1, as the
run-2 audit gated 1.0.0.

What still applies:

* The blocker stays open in `docs/todos/production-milestone.md`.
* 1.0.2 and every later release follow `docs/release-process.md` as written, including a green Complement run before the
tag. This waiver is no precedent for them.
* The 1.0.0 GitHub release is deleted, because it ships the vulnerabilities 1.0.1 fixes. The `v1.0.0` tag stays: a tag is
never moved or removed once something was published under it, and ADR-0651 cites it.

### Negative Consequences

* Federation conformance for 1.0.1 rests on this project's own tests and the audit. No external check supports it.

## Pros and Cons of the Options

### Hold the 1.0.1 tag until a green Complement run exists

* Good, because it follows the release process as written.
* Bad, because the vulnerable 1.0.0 stays the only release until a run is green, and no green run has ever been recorded.

## Links

* [ADR-0648](0648-the-complement-blocker-is-waived-for-1-0-0-only.md),
[ADR-0651](0651-the-v1-0-0-tag-moves-to-the-release-workflow-fix-without-a-version-bump.md),
[ADR-0235](0235-complement-evidence-is-a-maintainer-signed-record-tied-to-the-shipped-binary.md)
* [docs/release-process.md](../release-process.md), [docs/todos/production-milestone.md](../todos/production-milestone.md)
Original file line number Diff line number Diff line change
@@ -0,0 +1,93 @@
# A recovery asks the origin for a listed state event at most once

* Status: accepted
* Deciders: James Chapman
* Date: 2026-11-15

Technical Story: production log of pong.ping.me.uk at 1.0.2, 2026-11-15: about 95% of a day's warnings came from
three retained PDUs in one room.

## Context and Problem Statement

`backfill_missing_pdu_references` recovers the history a PDU lacks. When `/get_missing_events` and `/event` leave a
gap, it asks `GET /state_ids` at up to `backfill.max_outbound_calls` gap events (`state_snapshot_targets`), and each
snapshot fetches every state event it lists with `GET /event` before it stores any. One listed event the origin will
not supply ends that snapshot.

In the log, matrix.kahlilia.org answered every `/state_ids` request, and every answer listed the same state event,
which it then refused with 403 on `/event`, most likely because this server had not joined when the event was sent.
Each recovery therefore asked for that one event 15 times, once per snapshot target, and logged two warnings each
time. The deferred PDU retry (30 s doubling to 1 h, 10 attempts) repeated the whole recovery.

Should a recovery ask the origin again for a listed state event it has already failed to supply?

## Decision Drivers

* The origin decides what it serves. A refusal, a 404 or an unusable answer for one event does not change within
one recovery, which is bounded by `backfill.timeout_seconds` (45 s by default).
* Every snapshot needs every state event it lists, so a snapshot listing one that was not supplied cannot be
completed, whatever else it lists.
* The repeat is per snapshot target: the per-reference stage asks for each missing reference once, and each snapshot
asks for its own target once, so neither multiplies with the number of gaps.
* The history may become available later: the origin may fetch it, or this server may gain the right to see it.
The deferred retry exists for that.

## Considered Options

* Remember, for the rest of one recovery, each state event a `/state_ids` snapshot listed and the origin did not
supply; a later snapshot that lists one ends at once without asking.
* Remember every event ID any `GET /event` of the recovery did not supply, and send no further request for it.
* Stop trying further snapshot targets after the first snapshot fails.
* Remember unsupplied events across recoveries, for example in the retained PDU's row.

## Decision Outcome

Chosen option: the first. `BackfillBudget::unsupplied_events` holds the IDs. `backfill_state_ids_snapshot` records a
listed state event whenever its fetch brought back no usable PDU (refused, missing, another event, or no answer),
and ends a later snapshot that lists a recorded ID, neither held nor stored since, before it fetches any of its listed
events, logging at DEBUG, since the failure itself was logged at WARNING when it happened. The check comes first
because phase 1 stores nothing until every event is in hand: a snapshot that fetched the events listed before the
refused one would fetch them again only to discard them. `fetch_event_by_id` and `fetch_snapshot_event_by_id` neither record nor consult
the set. The set belongs to one `BackfillBudget`, so it never outlives the recovery, and the next deferred retry asks
again.

`/state_ids` itself is still asked at every target: each target is a different question, and its answer may list
only events the origin does serve.

### Positive Consequences

* A recovery whose snapshots all list one refused event sends one `/event` request for it, not one per target, and
logs its failure once. In the observed case that is 15 requests and 30 warnings down to 1 and 2.
* The calls saved are left in the recovery's budget for targets that can still succeed.
* A snapshot's target is still fetched after its state is known even when the per-reference stage could not get it
(ADR-0069, ADR-0070).

### Negative Consequences

* A listed state event whose fetch failed for a transient reason, such as a timeout, is not retried within the same
recovery. The deferred retry still asks again, as it did before.

## Pros and Cons of the Options

### Remember every unsupplied event

* Good, because it would also save the target fetch after a per-reference fetch of the same event failed.
* Bad, because that saves at most one request per gap event, and it takes away the second ask that the snapshot path
relies on: an event the per-reference stage could not get is fetched again as the snapshot's target once its
state is known. The ADR-0069 and ADR-0070 scenarios fail with it.

### Stop after the first failed snapshot

* Good, because it saves the `/state_ids` requests too.
* Bad, because the other targets are different gaps, and their snapshots may list only events the origin serves.

### Remember across recoveries

* Good, because the retries would also skip the request.
* Bad, because it would refuse history that has become available since, and it needs persistent state that the
deferred retry's own limits already bound.

## Links

* Refines [ADR-0069](0069-spec-conformant-state-ids-fallback-via-event-auth.md) and the A8 recovery rules in
`src/homeserver/AGENTS.md`
Original file line number Diff line number Diff line change
@@ -0,0 +1,82 @@
# A server that gave no answer to a profile lookup is not asked again for a minute

* Status: accepted
* Deciders: James Chapman
* Date: 2026-11-15

Technical Story: production log of pong.ping.me.uk at 1.0.2, 2026-11-15: eight `profile.proxy` timeouts to
example.org in two minutes.

## Context and Problem Statement

`GET /_matrix/client/v3/profile/{userId}` for a user of another server asks that server's `query/profile`. The call
holds a client outbound slot (ADR-0079, ADR-0751) and runs under the directory deadline. Clients make these lookups
by themselves, for member lists, hover cards and invite dialogs, and retry them. In the log, a client looked up a
user on example.org, which runs no Matrix server: discovery fell back to port 8448, every connection timed out, and
each of eight lookups in two minutes waited out the deadline and logged a warning.

The server discovery cache does not help: discovery succeeded, and only the connection failed. Should a server that
has just given no answer be asked again on the next lookup?

## Decision Drivers

* A lookup is interactive and cheap to repeat. A short delay before a recovered server is asked again costs a
client one missing display name, while a dead server costs a slot and the full deadline per lookup.
* Only the destination's silence should count. An HTTP answer of any status proves the server is there, and a
refusal made locally (federation policy, deadline, missing key) says nothing about it.
* The record must stay bounded, since a client chooses which servers are looked up.
* ADR-0262 and ADR-0749 keep joins and outbound delivery from serving remembered failures. Those retries are paced
by their own backoff or by the user; a profile lookup has neither.

## Considered Options

* Remember each server that gave no answer to a profile lookup for 60 seconds, in a bounded record guarded by
`runtime.mutex`, and answer lookups for it 502 at once during that time.
* Serve remembered discovery failures to profile lookups, as key resolution does.
* Shorten the directory deadline for profile lookups.

## Decision Outcome

Chosen option: the first. `perform_outbound_call` sets `OutboundCallResult::remote_unreachable` when discovery
found no server or the request got no connection, no verified TLS session or no response in time
(`unresolved_host`, `connection_failed`, `tls_verification_failed`, `timeout`, `network_error`). A redirect, an
oversized body and an error status are answers. `remote_profile_response` records such a server in
`HomeserverRuntime::profile_unreachable_servers` (`RecentlyUnreachableServers`: 60 s, at most 4096 servers, expired
entries and then the oldest dropped when full). While it is recorded, a lookup for any of its users is answered 502
`M_UNKNOWN` with no outbound call, logged at DEBUG. A server that answers at all is forgotten: `perform_outbound_call`
sets `OutboundCallResult::remote_answered` for any HTTP response, a redirect it refuses to follow and an oversized
body included. `remote_status` is 0 for those two, so it is not the test; a lookup overlapping one that failed would
otherwise leave the server recorded although it had just answered.

The record covers profile lookups only. Other client-triggered calls (public room directory, thirdparty) are made
when a user asks for them, not automatically, so they keep asking every time.

### Positive Consequences

* A dead server costs one slot and one deadline per minute of lookups, not one per lookup, and logs one warning a
minute.
* Slots stay free for lookups to servers that do answer.

### Negative Consequences

* A server that comes back is not asked for up to 60 seconds after its last failure.
* The record is in memory, per process, and is lost on restart.

## Pros and Cons of the Options

### Serve remembered discovery failures

* Good, because the cache already exists.
* Bad, because the observed case failed after discovery succeeded, so it would not have been covered.

### A shorter deadline for profile lookups

* Good, because it needs no state.
* Bad, because a slow but working server would then fail every lookup, and a dead one still costs a slot per lookup.

## Links

* Bounded by [ADR-0079](0079-client-triggered-outbound-proxying-runs-under-a-bounded-in-flight-budget.md) and
[ADR-0751](0751-the-per-client-outbound-call-cap-is-configurable-and-defaults-to-three.md)
* Contrasts with [ADR-0262](0262-a-join-reuses-server-discovery-but-never-a-remembered-failure.md) and
[ADR-0749](0749-outbound-delivery-never-serves-a-remembered-discovery-failure.md)
5 changes: 4 additions & 1 deletion docs/adr/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -560,7 +560,7 @@
* [0635 - An inbound federation origin is asked about once at a time, and its decision is kept for a short time](0635-an-inbound-federation-origin-is-asked-about-once-at-a-time-and-its-decision-is-kept-for-a-short-time.md)
* [0636 - A blob row is deleted with its last reference, and a removed record holds no reference](0636-a-blob-row-is-deleted-with-its-last-reference-and-a-removed-record-holds-none.md)
* [0647 - Versions from 1.0.0 are MAJOR.MINOR.BUILD](0647-versions-from-1-0-0-are-major-minor-build.md)
* [0648 - The Complement blocker is waived for 1.0.0 only](0648-the-complement-blocker-is-waived-for-1-0-0-only.md)
* [0648 - The Complement blocker is waived for 1.0.0 only](0648-the-complement-blocker-is-waived-for-1-0-0-only.md) — Extended by ADR-0744: 1.0.1 is waived too
* [0731 - A spawned worker gets no terminal, and no sandbox profile allows TIOCSTI or TIOCLINUX](0731-a-spawned-worker-gets-no-terminal-and-no-sandbox-profile-allows-tiocsti.md)
* [0732 - Every published attestation check pins the signing workflow and the ref](0732-every-published-attestation-check-pins-the-signing-workflow-and-the-ref.md) — Partly superseded by ADR-0742: every published file is attested by packages.yml
* [0719 - One client key holds a sixteenth of the parked-connection budgets, and a readable connection waits a bounded time](0719-one-client-key-holds-a-sixteenth-of-the-parked-connection-budgets-and-a-readable-connection-waits-a-bounded-time.md)
Expand Down Expand Up @@ -626,6 +626,7 @@
* [0741 - Each changelog version has Features, Security Fixes, Bug fixes and Release notes](0741-each-changelog-version-has-features-security-fixes-bug-fixes-and-release-notes.md)
* [0742 - Fixed build-and-test and package platform lists, and a tag ships what latest ships](0742-build-test-and-package-platform-lists-and-a-tag-ships-what-latest-ships.md)
* [0743 - Analyses, then build and test jobs, then packages take runners](0743-analyses-then-build-and-test-then-packages-take-runners.md)
* [0744 - The Complement blocker is also waived for 1.0.1](0744-the-complement-blocker-is-also-waived-for-1-0-1.md)
* [0745 - A first join into a restricted room stores the resident's verified co-signature](0745-a-first-restricted-join-stores-the-residents-verified-co-signature.md)
* [0746 - A threaded receipt may name any event whose relation chain reaches the thread root](0746-a-threaded-receipt-may-name-any-event-whose-relation-chain-reaches-the-root.md)
* [0747 - A remote member may leave through a server whose local members have all left](0747-a-remote-member-may-leave-through-a-server-whose-local-members-have-left.md)
Expand All @@ -641,3 +642,5 @@
* [0757 - Hardened BSD builds link -pie, and the ELF gate reads elftoolchain readelf](0757-hardened-bsd-builds-link-pie-and-the-elf-gate-reads-elftoolchain-readelf.md)
* [0758 - Every worker shard reloads every changed room, and main tracks each shard's copy](0758-every-worker-shard-reloads-every-changed-room-and-main-tracks-each-shards-copy.md)
* [0759 - Main builds make_leave and make_knock templates while the worker's reload of the room is pending](0759-main-builds-make-leave-and-make-knock-templates-while-the-workers-reload-is-pending.md)
* [0760 - A recovery asks the origin for a listed state event at most once](0760-a-recovery-asks-the-origin-for-a-listed-state-event-at-most-once.md)
* [0761 - A server that gave no answer to a profile lookup is not asked again for a minute](0761-a-server-that-gave-no-answer-to-a-profile-lookup-is-not-asked-again-for-a-minute.md)
16 changes: 16 additions & 0 deletions docs/debug-logging.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,22 @@ when a gateway answered after more than half the total timeout and was counted a
deadline and queued a pusher unattempted. `account.deactivated.pusher_removal_failed` (warning,
`user_id`) is logged when a pusher of a deactivated account could not be deleted.

## Repeated remote failures logged once (1.0.3)

Two remote failures are logged at warning when they happen and at debug when this
server declines to repeat them:

- Backfill: `federation.backfill.snapshot_event_fetch_failed` (module `rooms`) and the
`backfill_state_ids_snapshot: a listed event could not be fetched` warning are
written once per event and recovery. A later state snapshot in the same recovery
that lists the same event ends before fetching any of its events and logs the debug line
`a listed event was already not supplied in this recovery` (ADR-0760). The
deferred retry runs a new recovery, which asks, and warns, again.
- Remote profile lookups: `profile.proxy` with `http_status=0` is a warning when a
server gives no answer. For the next 60 s, lookups for that server's users are
answered 502 without a call and log the debug line `event=profile.proxy ...
reason=gave no answer recently` (ADR-0761).

## Room creation failures

`room.create.failed` (module `rooms`) is an error, not debug-only. It is
Expand Down
Loading
Loading