[release-7.3] Fix stale peers for client facing roles (storage server, commit proxy, grv proxy)

[spraza]

Problem

When a server process a client talks to is killed (storage server, commit/grv proxy, etc.), the client can keep stale references to the dead process (via RequestStream). For example:

• Storage servers: DatabaseContext::locationCache holds the dead SS's StorageServerInterface until either a read hits that shard and fails (baseline error-driven invalidateCache), or clientInfo/shard reassignment replaces it. A DatabaseContext that doesn't actively read the affected shards (for example a server-hosted/internal client, or an idle one) can hold the dead SS's interface for a long time.

• Proxies: when the recruited proxy count exceeds MAX_*_PROXY_CONNECTIONS, MonitorLeader::shrinkProxyList caches a sampled subset (lastCommitProxies/lastGrvProxies). If a proxy is killed and the recruited count then drops back under the cap, the cached subset keeps pinning the killed proxy's RequestStreams.

Each pinned RequestStream holds a FlowTransport peer reference, which keeps connectionKeeper alive retrying the dead address. That produces connection-timeout (CTO) churn and wasted reconnect attempts. At scale this can show up as a problem with various symptoms.

Solution

The core fixes make the client drain its connections to dead peers. Each fix is an opt-in knob (default off in production, randomized in simulation, forced on in the new test):

1. locationCachePeerEvictor (NativeAPI, knob LOCATION_CACHE_PEER_EVICTOR_ENABLED): a per-DatabaseContext actor that periodically (LOCATION_CACHE_PEER_EVICTOR_DELAY) samples FlowTransport's persistent per-address connect-failed counter, and for any address whose count went up beyond LOCATION_CACHE_PEER_EVICTOR_FAILED_THRESHOLD since the previous sweep, calls invalidateCacheByAddress to evict every cached location that references it. Dropping the cache's Reference<LocationInfo> lets the dead SS's streams release.

   • To get that persistent count, I added a persistent per-address connect-failed counter (FlowTransport, knob PERSISTENT_CONNECT_FAILED_COUNT_TTL): connectionKeeper records a per-destination cumulative connect-failure count plus last-failure time that survives Peer destruction. That's the stable signal the evictor reads. It's TTL-pruned to stay bounded.

2. shrinkProxyList under-cap cache-clear (MonitorLeader, knob SHRINK_PROXY_LIST_CLEAR_CACHE_BELOW_THRESHOLD): clears the cached sampled subset on iterations where the recruited count is below the cap, so a killed proxy's pinned RequestStreams release.

3. Eager proxy rebuild (knob DBCONTEXT_EAGER_PROXY_UPDATE): rebuilds cx->commitProxies/grvProxies right away on a clientInfo proxy-list change instead of waiting for the next transaction's lazy lookup.

Note: this PR is about the problem and the fix, but quite a lot of time was spent on figuring out where and what were the problems leading to high connection timeouts. These changes assisted with that:

• InterfaceTracker (knob STALE_PEER_OBSERVABILITY, default off): at various layers of the RPC stack, diagnostic per-(address, role) interface create/delete accounting plus per-ref backtraces. This helps get close to where the leak was coming from. Every method is a no-op when the knob is off.

• StalePeerTest workload + toml: kills a configurable client-facing role, waits for recovery, then asserts no leaked interface copies (Delta == 0) remain on the client. A big part of this PR is a result of constant iteration on this failing test, and leveraging observability above to figure out why the test was failing. The goal was to make the test pass, and meanwhile, ensure general db functionality is not broken (general 100K signal).

Testing

Simulation: ran StalePeerTest 10K times, all passed:
20260622-234822-praza-stalepeer-v41m4-on7.3.77-c37d2130797b9 compressed=True data_size=34905210 duration=433655 ended=10000 fail_fast=100 max_runs=10000 pass=10000 priority=100 remaining=0 runtime=0:08:02 sanity=False started=10000 stopped=20260622-235624 submitted=20260622-234822 timeout=5400 username=praza-stalepeer-v41m4-on7.3.77-c37d2130797b99acc9da0313be445d751fee5d62

Simulation: ran the general 100K, all passed:
20260622-234826-praza-stalepeer-v41m4-on7.3.77-c37d2130797b9 compressed=True data_size=34872287 duration=3302230 ended=100000 fail_fast=10 max_runs=100000 pass=100000 priority=100 remaining=0 runtime=0:39:25 sanity=False started=100000 stopped=20260623-002751 submitted=20260622-234826 timeout=5400 username=praza-stalepeer-v41m4-on7.3.77-c37d2130797b99acc9da0313be445d751fee5d62

Perf cluster: saw up to a 100x reduction in CTO/min aggregated across clients (CTO = connection timeout).

[foundationdb-ci]

Result of foundationdb-pr on Linux RHEL 9

• Commit ID: 884238e
• Duration 0:08:14
• Result: ❌ FAILED
• Error: Error while executing command: ninja -v -C build_output -j ${NPROC} all packages strip_targets. Reason: exit status 1
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-clang on Linux RHEL 9

• Commit ID: 884238e
• Duration 0:08:22
• Result: ❌ FAILED
• Error: Error while executing command: ninja -v -C build_output -j ${NPROC} all packages strip_targets. Reason: exit status 1
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-macos-m1 on macOS 14.x

• Commit ID: 884238e
• Duration 0:08:56
• Result: ❌ FAILED
• Error: Error while executing command: ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i ${HOME}/.ssh_key ec2-user@${MAC_EC2_HOST} /opt/homebrew/bin/bash --login ./build_pr_macos.sh. Reason: exit status 1
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-macos on macOS 14.x

• Commit ID: 884238e
• Duration 0:11:21
• Result: ❌ FAILED
• Error: Error while executing command: ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i ${HOME}/.ssh_key ec2-user@${MAC_EC2_HOST} /usr/local/bin/bash --login ./build_pr_macos.sh. Reason: exit status 1
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-clang-73 on Linux CentOS 7

• Commit ID: 884238e
• Duration 0:40:32
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-73 on Linux CentOS 7

• Commit ID: 884238e
• Duration 0:50:29
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-cluster-tests-73 on Linux CentOS 7

• Commit ID: 884238e
• Duration 0:58:02
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)
• Cluster Test Logs zip file of the test logs (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-clang on Linux RHEL 9

• Commit ID: 6730470
• Duration 0:07:27
• Result: ❌ FAILED
• Error: Error while executing command: ninja -v -C build_output -j ${NPROC} all packages strip_targets. Reason: exit status 1
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr on Linux RHEL 9

• Commit ID: 6730470
• Duration 0:07:41
• Result: ❌ FAILED
• Error: Error while executing command: ninja -v -C build_output -j ${NPROC} all packages strip_targets. Reason: exit status 1
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-macos-m1 on macOS 14.x

• Commit ID: 6730470
• Duration 0:08:00
• Result: ❌ FAILED
• Error: Error while executing command: ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i ${HOME}/.ssh_key ec2-user@${MAC_EC2_HOST} /opt/homebrew/bin/bash --login ./build_pr_macos.sh. Reason: exit status 1
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-macos on macOS 14.x

• Commit ID: 6730470
• Duration 0:24:36
• Result: ❌ FAILED
• Error: Error while executing command: ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i ${HOME}/.ssh_key ec2-user@${MAC_EC2_HOST} /usr/local/bin/bash --login ./build_pr_macos.sh. Reason: exit status 1
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-clang-73 on Linux CentOS 7

• Commit ID: 6730470
• Duration 0:39:54
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-73 on Linux CentOS 7

• Commit ID: 6730470
• Duration 0:44:46
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-cluster-tests-73 on Linux CentOS 7

• Commit ID: 6730470
• Duration 0:57:57
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)
• Cluster Test Logs zip file of the test logs (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-clang on Linux RHEL 9

• Commit ID: e71bffc
• Duration 0:07:31
• Result: ❌ FAILED
• Error: Error while executing command: ninja -v -C build_output -j ${NPROC} all packages strip_targets. Reason: exit status 1
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr on Linux RHEL 9

• Commit ID: e71bffc
• Duration 0:07:58
• Result: ❌ FAILED
• Error: Error while executing command: ninja -v -C build_output -j ${NPROC} all packages strip_targets. Reason: exit status 1
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-macos-m1 on macOS 14.x

• Commit ID: e71bffc
• Duration 0:08:25
• Result: ❌ FAILED
• Error: Error while executing command: ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i ${HOME}/.ssh_key ec2-user@${MAC_EC2_HOST} /opt/homebrew/bin/bash --login ./build_pr_macos.sh. Reason: exit status 1
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-macos on macOS 14.x

• Commit ID: e71bffc
• Duration 0:10:52
• Result: ❌ FAILED
• Error: Error while executing command: ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i ${HOME}/.ssh_key ec2-user@${MAC_EC2_HOST} /usr/local/bin/bash --login ./build_pr_macos.sh. Reason: exit status 1
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[ploxiln]

please merge in latest release-7.3 branch to get fixes for more-modern gcc and clang

[foundationdb-ci]

Result of foundationdb-pr-clang-73 on Linux CentOS 7

• Commit ID: e71bffc
• Duration 0:35:45
• Result: ❌ FAILED
• Error: Error while executing command: python3 -m joshua.joshua start --tarball $(find build_output/packages -name correctness\*.tar.gz) --username ${CORRECTNESS_USERNAME} --max-runs 10000. Reason: exit status 1
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-73 on Linux CentOS 7

• Commit ID: e71bffc
• Duration 0:42:52
• Result: ❌ FAILED
• Error: Error while executing command: python3 -m joshua.joshua start --tarball $(find build_output/packages -name correctness\*.tar.gz) --username ${CORRECTNESS_USERNAME} --max-runs 10000. Reason: exit status 1
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-macos-m1 on macOS 14.x

• Commit ID: 94159a8
• Duration 1:00:04
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr on Linux RHEL 9

• Commit ID: 94159a8
• Duration 1:03:55
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-73 on Linux CentOS 7

• Commit ID: 94159a8
• Duration 1:04:28
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr on Linux RHEL 9

• Commit ID: 2a8a2f5
• Duration 1:06:32
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-73 on Linux CentOS 7

• Commit ID: 2a8a2f5
• Duration 1:09:09
• Result: ❌ FAILED
• Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-cluster-tests-73 on Linux CentOS 7

• Commit ID: 2a8a2f5
• Duration 1:14:30
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)
• Cluster Test Logs zip file of the test logs (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-macos on macOS 14.x

• Commit ID: 94159a8
• Duration 1:13:38
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-cluster-tests-73 on Linux CentOS 7

• Commit ID: 94159a8
• Duration 1:14:22
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)
• Cluster Test Logs zip file of the test logs (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-macos on macOS 14.x

• Commit ID: 2a8a2f5
• Duration 1:57:16
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[alecgrieser]

LGTM!

[alecgrieser]

Do you think this one we'd want to turn on in 7.4? Unlike the location cache sweep or the shrink proxy fix in general, I could see it being more debatable. Maybe we want to to be more lazy with this update, but I'm not sure. My guess is it probably doesn't make too much of a difference in production workloads, though

[gxglass]

AI review attached. The thing it calls B1 may be pretty harmless, I can't really tell. My suggestion is either take its advice on the assignment logic or alternatively if there is a very simple thing that can be done to suppress this class of false positive (a comment above the assignment, maybe) then do it. Alternatively just say "the AI is being too wishy washy about future risks here" and we move on.

pr13367-review.md

[gxglass]

I don't fully understand the tracking (off by default) or the test case, but the production logic seems pretty tightly constrained and anyway can be disabled, so in the interest of progress I am going to click Approve.

[spraza]

Made a small change: 231f326

@alecgrieser and @gxglass - can you reapprove when you get a chance?

[foundationdb-ci]

Result of foundationdb-pr-macos-m1 on macOS 14.x

• Commit ID: 231f326
• Duration 0:33:55
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-clang-73 on Linux CentOS 7

• Commit ID: 231f326
• Duration 0:49:14
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-clang-73 on Linux CentOS 7

• Commit ID: 231f326
• Duration 0:50:22
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-clang on Linux RHEL 9

• Commit ID: 231f326
• Duration 0:55:10
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-clang on Linux RHEL 9

• Commit ID: 231f326
• Duration 0:57:57
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr on Linux RHEL 9

• Commit ID: 231f326
• Duration 1:08:14
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-macos on macOS 14.x

• Commit ID: 231f326
• Duration 1:08:25
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-73 on Linux CentOS 7

• Commit ID: 231f326
• Duration 1:08:28
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr on Linux RHEL 9

• Commit ID: 231f326
• Duration 1:09:49
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-73 on Linux CentOS 7

• Commit ID: 231f326
• Duration 1:10:45
• Result: ❌ FAILED
• Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-cluster-tests-73 on Linux CentOS 7

• Commit ID: 231f326
• Duration 1:13:23
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)
• Cluster Test Logs zip file of the test logs (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-cluster-tests-73 on Linux CentOS 7

• Commit ID: 231f326
• Duration 1:13:47
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)
• Cluster Test Logs zip file of the test logs (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-macos-m1 on macOS 14.x

• Commit ID: 231f326
• Duration 1:52:24
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-macos on macOS 14.x

• Commit ID: 231f326
• Duration 3:11:51
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)