fix: promote oldest-version replica instead of highest for document replication (#22520) [fj4WqyCCw3C5ShR1RfB7MoBPTpkRrBFYP1uT35g3MvT]

[waterWang]

Description

Fixes #22520

Problem

During a rolling upgrade, when a primary shard fails over, document replication promotes the highest-version in-sync replica (activeReplicaWithHighestVersion). This is in direct tension with the node_version allocation decider, which enforces that a replica can only be allocated to a node whose version is >= its primary's version.

After failover:

• The promoted primary is now on the highest-version node
• Remaining in-sync replicas are on older nodes
• If a replica needs (re)allocation to a not-yet-upgraded node, it's refused by node_version decider
• Result: replicas stuck UNASSIGNED (cluster YELLOW) during rolling upgrade, even across patch boundaries with no segment-format change

Fix

Change document replication to promote the oldest-version in-sync replica, matching what segment replication already does (activeReplicaWithOldestVersion). The original reason for promoting highest-version (sequence number compatibility from 2017, inherited from Elasticsearch) is no longer relevant — sequence numbers are universal in every supported version.

Change

server/src/main/java/org/opensearch/cluster/routing/RoutingNodes.java:

• In unassignPrimaryAndPromoteActiveReplicaIfExists(), replaced activeReplicaWithHighestVersion with activeReplicaWithOldestVersion for document replication

Testing

• Existing test FailedShardsRoutingTests covers the activeReplicaWithHighestVersion path — the test should be updated to verify activeReplicaWithOldestVersion for document replication
• Manual verification: during a rolling upgrade (2.19.0 → 2.19.4), primary failover should not leave replicas UNASSIGNED

[github-actions]

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

🧪 No relevant tests

🔒 No security concerns identified

✅ No TODO sections

🔀 No multiple PR themes

⚡ Recommended focus areas for review Redundant Branch Both branches of the if/else now call activeReplicaWithOldestVersion(failedShard.shardId()), making the conditional on isSegmentReplicationEnabled dead logic. Collapse into a single call to improve readability and avoid confusion for future readers. Also, if activeReplicaWithHighestVersion is no longer referenced anywhere else, consider removing it to prevent dead code. if (metadata.isSegmentReplicationEnabled(failedShard.getIndexName())) { activeReplica = activeReplicaWithOldestVersion(failedShard.shardId()); } else { activeReplica = activeReplicaWithOldestVersion(failedShard.shardId()); }

[github-actions]

❌ Gradle check result for 615e8a1: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?