conn pool: add NotifyPeerAndDrainExisting drain behavior

[ivpr]

This PR is part of a 3-PR stack. PR2 and PR3 are out-of-tree against ivpr/envoy and exist to demonstrate a concrete consumer of the API added here — they do not need to land in envoyproxy/envoy for this PR to be useful on its own.

#	Where	What
PR1 (this PR)	envoyproxy/envoy	Core: DrainBehavior::NotifyPeerAndDrainExisting enum value + virtual hook ActiveClient::notifyPeerAndDrain() + ClusterManager::drainConnections(cluster, predicate, drain_behavior) overload.
PR2 → ivpr/envoy#1	ivpr/envoy (stacked on PR1)	Upstream consumer: reverse-tunnel acceptor extension registers a POST /reverse_tunnel/drain_cluster admin endpoint that fans out the new drain behavior to every envoy.clusters.reverse_connection cluster. Opt-in via envoy.reloadable_features.reverse_tunnel_drain_with_goaway.
PR3 → ivpr/envoy#2	ivpr/envoy (stacked on PR2)	Downstream consumer: drain-aware HCM observes peer GOAWAY on reverse-tunnel sockets and asks the initiator IOHandle to drop the tunnel from tracking and dial a replacement. Same runtime guard.

End-to-end flow this PR enables

sequenceDiagram
    autonumber
    participant Op as Operator
    participant US as Upstream Envoy
    participant Tun as Reverse tunnel (HTTP/2)
    participant DS as Downstream Envoy

    Note over US,DS: Steady state. DS holds a long-lived HTTP/2 tunnel. <br/>US owns the HTTP/2 client codec via the reverse_connection cluster.

    Op->>US: POST /reverse_tunnel/drain_cluster
    rect rgba(150,200,255,0.15)
        Note over US: PR2 consumer. Admin handler iterates envoy.clusters.reverse_connection clusters <br/>and fans out the new drain behavior.
        US->>US: drainConnections(cluster, nullptr, NotifyPeerAndDrainExisting)
    end

    rect rgba(255,210,150,0.25)
        Note over US: PR1 (this PR). Pool snapshots ready, busy and early-data clients, <br/>then calls MultiplexedActiveClientBase.notifyPeerAndDrain on each, <br/>which calls codec_client_.goAway and transitions to Draining if any streams are active. <br/>is_draining_for_deletion_ stays false so the pool stays usable for new streams via fresh clients.
        US-->>Tun: HTTP/2 GOAWAY
    end

    Tun-->>DS: HTTP/2 GOAWAY

    rect rgba(180,255,180,0.20)
        Note over DS: PR3 consumer. Drain-aware HCM observes peer GOAWAY, <br/>asks the reverse-tunnel IOHandle to drop this tunnel from tracking <br/>and kick maintenance to dial a replacement.
        DS->>DS: markTunnelDrainingAndDialReplacement(key)
        DS-->>Tun: dial new tunnel to sibling US replica
    end

    Note over US,DS: In-flight HTTP/2 streams on the old tunnel complete normally. <br/>The drained client auto-closes once its last stream finishes (onStreamClosed line 311-313). <br/>The tunnel TCP-closes after that.

Loading

PR1 is the only piece that lands in envoyproxy/envoy core. PR2 and PR3 are in reverse tunnel code and are linked above so you can see exactly what a consumer of this API looks like — both how it's invoked (PR2) and how the multiplexed-peer GOAWAY observation can be wired on the receiving side (PR3). Since PR2 and PR3 depend on PR1, I will raise them as new PRs in envoy proxy once this PR is merged.

What PR2 and PR3 say about the shape of the new core API

• DrainBehavior::NotifyPeerAndDrainExisting is strictly opt-in. Existing callers of drainConnections(DrainAndDelete) and friends see no behavior change. PR2 / PR3 also keep the behavior runtime-guarded so end-user reverse-tunnel deployments do not see it until they explicitly enable it.
• ClusterManager::drainConnections(cluster, predicate, drain_behavior) is the three-arg overload that lets an extension target a single cluster with a specific behavior. PR2 uses it to target only envoy.clusters.reverse_connection clusters; everything else is left alone.
• MultiplexedActiveClientBase::notifyPeerAndDrain() handles the "preserve in-flight streams" semantics: if codec_client_->numActiveRequests() > 0, the client transitions to Draining instead of closing; otherwise it closes immediately. This is the property PR2's scenario depends on — sibling-replica failover should not kill the requests already on the wire.

How drained connections eventually close (no new flag required)

The new branch in drainConnectionsImpl does not set is_draining_for_deletion_. Clients close via existing pre-existing paths instead:

1. Idle-at-drain-time clients close immediately. MultiplexedActiveClientBase::notifyPeerAndDrain calls codec_client_->close() directly when numActiveRequests() == 0. The fall-through into closeIdleConnectionsForDrainingPool() flushes connecting_clients_.
2. Busy-then-drained clients close on last-stream-finish. They sit in Draining state while in-flight streams complete. When the last stream finishes, the existing block at conn_pool_base.cc lines 311-313 inside onStreamClosed closes the client. This is the same path the existing MaxConnectionDuration and MaxStreamsPerConnection drains rely on today.
3. Peer-initiated close on receipt of the GOAWAY shows up via the normal ConnectionEvent::RemoteClose handling.
4. Server drain timeout (--drain-time-s) is the last-resort safety net.

The pool itself stays alive (intentionally — that is why we are renaming away from …AndDelete). New requests for the same host land in newStreamImpl and pass through the !is_draining_for_deletion_ ASSERT (no crash); they find no ready clients (all drained), so tryCreateNewConnections() spins up a fresh non-Draining client.

---

Commit message body

Add a new ConnectionPool::DrainBehavior::NotifyPeerAndDrainExisting value that mirrors DrainExistingConnections semantically but first notifies the peer that the pool is draining. For multiplexed pools (HTTP/2, HTTP/3) the notification is a protocol-level GOAWAY; for HTTP/1 and raw TCP there is no peer-level signal and the behavior degrades to plain DrainExistingConnections (the peer learns about the drain when the TCP socket closes). Unlike DrainAndDelete, the pool stays usable: new streams are served by fresh clients while drained clients finish their in-flight work and close on idle.

What this change introduces

• A new enum value ConnectionPool::DrainBehavior::NotifyPeerAndDrainExisting.
• A virtual hook ConnectionPool::ActiveClient::notifyPeerAndDrain() with a default no-op implementation (so HTTP/1 + TCP pools support the new behavior without any code change — they just don't emit a peer signal).
• An override on MultiplexedActiveClientBase that calls codec_client_->goAway() and transitions the client to Draining if any streams are active, otherwise closes the client immediately.
• A branch in ConnPoolImplBase::drainConnectionsImpl() that snapshots ready / busy / early-data clients, invokes the new hook on each, and falls through into the existing DrainExistingConnections body so closeIdleConnectionsForDrainingPool() handles the connecting-clients cleanup. The branch does NOT set is_draining_for_deletion_, so newStreamImpl continues to accept new requests after the drain (served by fresh clients).
• A new three-arg ClusterManager::drainConnections(cluster, predicate, drain_behavior) overload (parallel to the existing two-arg overload) so callers can target a single cluster with a specific behavior.

Why this isn't GoAwayAndDrainAndDelete

Earlier revisions of this PR named the value GoAwayAndDrainAndDelete and modeled it after DrainAndDelete (set is_draining_for_deletion_ = true, mark the pool for deletion). That was incorrect for the motivating use case (reverse-connection clusters), because the cluster's hosts remain LB-selectable after the drain — a new request after drain would trip ASSERT(!is_draining_for_deletion_) in newStreamImpl and crash the process. The new semantics keep the pool usable; closure happens naturally on stream completion via the existing onStreamClosed close-on-idle logic.

GoAway was also HTTP/2-specific vocabulary in the protocol-agnostic envoy/common/conn_pool.h enum. NotifyPeer generalizes cleanly — HTTP/1 and TCP inherit the no-op virtual without any per-protocol branching.

Risk Level

Low. The new enum value is opt-in; existing DrainBehavior values and their code paths are untouched. The new virtual on ActiveClient has a no-op default, so non-multiplexed clients see no behavior change. No structural changes to ConnectivityGrid (the renamed value does not set the grid's draining_=true flag because the pool stays usable).

Testing

Six unit tests in test/common/http/http2/conn_pool_test.cc exercising NotifyPeerAndDrainExisting on the HTTP/2 pool:

• sends GOAWAY then drains a ready client with an active stream
• closes an idle client immediately after GOAWAY
• does not send GOAWAY on a connecting client
• idempotent on repeat calls
• handles multiple clients (mixed busy + idle)
• NEW regression test: a new stream issued after drain creates a fresh client and completes successfully (would crash on the old GoAwayAndDrainAndDelete semantics with ASSERT(!is_draining_for_deletion_))

Existing conn_pool_base_test and conn_pool_grid_test pass unchanged.

Manual validation (in PR2/PR3 stack)

End-to-end validated with a 2-Envoy reverse-tunnel setup: US drains its reverse_connection cluster via POST /reverse_tunnel/drain_cluster, the pool fires GOAWAY on every reverse tunnel, DS observes GOAWAY and dials replacement tunnels, in-flight streams complete on the old tunnels, US stays alive and serves new requests via fresh clients — no crash.

Docs

n/a (header doc comments updated for the new enum value, virtual hook, and ClusterManager overload).

Release Notes

Added a new connection-pool drain behavior NotifyPeerAndDrainExisting that sends a protocol-level GOAWAY to each multiplexed client before draining the pool. Unlike DrainAndDelete, the pool remains usable for new streams via fresh clients while existing clients drain in the background.

[ivpr]

/retest

[basundhara-c]

@botengyao could you have a look at the overall approach for this?

[RyanTheOptimist]

I'm trying to understand how NotifyPeerAndDrainExisting differs from DrainExistingNonMigratableConnections. Can you explain the difference?

[ivpr]

From what I checked, DrainExistingNonMigratableConnections is for HTTP3 QUIC to support skipping the draining when source/downstream network changes which allows connections to be kept alive.

NotifyPeerAndDrainExisting is specifically meant to actually drain the existing connection but with a notification(eg: HTTP2 GOAWAY) to support use-cases like reverse connections where connection can only be initiated from the listener/downstream side and needs a notification to be sent for grace draining.

[ivpr]

Closing this PR in favour of #45632 which solves it a bit more holistically