[nick-boss-tech]

🤖 AI text below 🤖 (reviewed by Nick Shanin)

Ready for review, but held as a draft due to the project's open-PR cap; I'll mark it open on request.

A safe-leader test fails when the client sends an update to a replica the monkey stops. ChaosMonkeySafeLeaderTest and ChaosMonkeySafeLeaderWithPullReplicasTest stop replicas but spare leaders, then assert that no update failed (line 209). The cloud client in these tests is built with a random choice between sending updates to shard leaders only and sending them to any replica (line 370). In the second mode an update can go to a pull replica, and the monkey can stop that node while the request is in flight. The request then fails with ClosedChannelException and the check reports expected:<0> but was:<1>. On main with seed E4EFD2B86C0B81D4 on local disk, a deleteById sent to collection1_shard1_replica_p11 failed 4 ms after the monkey stopped that node.

The safe-leader tests now use a leaders-only client. Both tests override createCloudClient to build the client with shardLeadersOnly set. Leaders are the one kind of node these tests never stop, so an update that fails now is a real failure. Nothing else changes. The NothingIsSafe tests keep the random client.

Proof. The failure is a timing race, so no single run fails on main. Forcing the any-replica client in ChaosMonkeySafeLeaderWithPullReplicasTest failed 1 of 4 random runs of ChaosMonkeySafeLeaderWithPullReplicasTest with expected:<0> but was:<1>. Its failed request was an update to collection1_shard1_replica_p14, a pull replica. With the leaders-only client (compare, head 60eb0e7), 8 runs of the two tests passed: ChaosMonkeySafeLeaderWithPullReplicasTest three times with the seed above and three times with random seeds, and ChaosMonkeySafeLeaderTest twice with random seeds. Nightly tests on, local disk, 2026-10-23. The runs were repeated on a second host and file system: 9 of 9 passed, including three runs of ChaosMonkeySafeLeaderWithPullReplicasTest at the seed above and three random-seed runs of each test. The full module check passes at the branch head.

The choice. Three routes were available:

1. Use a leaders-only client in the safe-leader tests (this PR).
2. Leave failures caused by a stopped node out of the failure count. This also loosens the failure tolerance in the NothingIsSafe tests that read the same count.
3. Retry the cut-off update in the client. The load-balancing client does not retry an update after an I/O error, because the server may already have applied it (line 576).

Context: the test name says leaders are safe, and a leader is the node every leaders-only update goes to. A retry on a non-idempotent update could apply it twice.

Ladder: Does the test promise no failed update? Yes, but only for updates sent to a node it spares. Is a failed update to a node the monkey stops a defect? No. Does the any-replica client lose coverage? The NothingIsSafe tests still run it, with a failure tolerance.

Recommendation: route 1. It keeps the zero-failure check strict, and it changes only the two tests that need it.

Limits. The safe-leader tests no longer send updates through a non-leader replica under chaos. Whether the client should retry an update that fails with ClosedChannelException is a separate question; a follow-up ticket can be opened on request. The Too many links error that a network file system returned during an index copy (IndexFetcher line 1224) is not addressed. The timeout in ChaosMonkeyNothingIsSafeWithPullReplicasTest appeared only on that file system.

AI-generated description. Nick Shanin directed the work and takes responsibility for this contribution.