Add transport failure and connect latency counters to node stats - #1
Draft
jbacchus126 wants to merge 4 commits into
Draft
Add transport failure and connect latency counters to node stats#1jbacchus126 wants to merge 4 commits into
jbacchus126 wants to merge 4 commits into
Conversation
jbacchus126
force-pushed
the
jbacchus/tcp-observability
branch
from
August 4, 2026 19:20
cfcb4e6 to
9ca395a
Compare
Expose counters for transport connect-time and request-time failures via _nodes/stats: connect_failures, connect_time_millis, connect_time_millis_max, channel_close_by_type, outgoing_timeouts, and requests_failed_on_disconnect. The connect-time counters cover the connection-open path, which no existing metric observes: a slow or hung connect blocks the cluster applier thread without incrementing any request-level stat. channel_close_by_type attributes socket closes to individual channel types, which distinguishes impairment affecting only some types from a whole-node failure. Also log connect latency, send failures, and per-channel socket closes at DEBUG to aid diagnosis without adding INFO-level noise. Signed-off-by: Janae Bacchus <jbacchus@uber.com> Co-authored-by: Cursor <cursoragent@cursor.com>
The new transport counters were serialized unconditionally, so a node running this change would read six fields that an older node never wrote, desynchronizing the stream and failing _nodes/stats in a mixed-version cluster or against an older remote cluster. Gate them on 3.8.0 so older nodes exchange only the original counters and report the new ones as zero. Add a TransportStats.Builder copy constructor and use it in TransportService.stats(), which previously restated all twelve fields by hand to override two. Any field added in future was silently dropped there. Cover the wire format with TransportStatsTests, including both older-node directions, and exercise the new fields in NodeStatsTests. Signed-off-by: Janae Bacchus <jbacchus@uber.com> Co-authored-by: Cursor <cursoragent@cursor.com>
Closing one socket tears down its whole connection, which closes every remaining socket, and each of those fired the counting listener. A single failure was therefore reported as a failure of all six channel types at once, and a deliberate disconnect looked identical to a hard failure, leaving channel_close_by_type with no diagnostic value. The counting and teardown listeners are now one listener that records the close before closing the connection, so only the socket that caused the teardown is counted and deliberate closes are not counted at all. A socket carrying several channel types still counts against each of them, since losing it affects all of them, so the total across types is not the number of sockets closed. Say so on the field. connect_failures missed sockets that fail to open, because that path returns before any connect listener is registered. Unresolvable hosts and exhausted file descriptors went uncounted. connect_time_millis now starts its clock before the first socket is opened rather than after, so it measures the whole open path, and the field comments say what the latency covers and that the maximum is a high watermark that is never reset. A connect attempt rejected before it starts never records a start time, so the failure log computed an elapsed time from zero, which is meaningless and can be negative because nanoTime has no defined origin. Report the duration only when an attempt actually ran. Channel type keys are lowercased to match REST field conventions and sorted so field order is stable whether the counters were collected locally or read off the wire. Co-authored-by: Cursor <cursoragent@cursor.com> Signed-off-by: Janae Bacchus <jbacchus@uber.com>
A teardown attributes exactly one socket, the first to close. Sockets failing at almost the same moment are not counted separately, because once teardown is under way a closing socket cannot be distinguished from one the teardown itself closed. Sampling the first failure still shows which channel types are unhealthy, which is the point of the breakdown, but the limit was not written down anywhere. Also note that total_outbound_connections counts successful opens, so dividing connect_time_millis by it gives a valid average. Co-authored-by: Cursor <cursoragent@cursor.com> Signed-off-by: Janae Bacchus <jbacchus@uber.com>
jbacchus126
force-pushed
the
jbacchus/tcp-observability
branch
from
August 5, 2026 17:37
ef1f318 to
c11f2a6
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
_nodes/statscurrently reports transport volume only —rx_count,rx_size_in_bytes,tx_count,tx_size_in_bytes,server_open, andtotal_outbound_connections. There is no always-available signal for transport failure or connection-open latency, which makes a class of network faults hard to diagnose on a running cluster: the volume counters keep incrementing normally while requests silently time out or connections take seconds to establish.The connection-open path is the largest gap. A slow or hung connect blocks the cluster applier thread, and no existing metric observes it at all —
total_outbound_connectionsis incremented only after a handshake succeeds, so a node struggling to open connections looks identical to a healthy idle node.This PR adds six counters to
TransportStats:connect_failuresconnect_time_millisconnect_time_millis / total_outbound_connectionsconnect_time_millis_maxtransport.connect_timeoutis safechannel_close_by_typeRECOVERY,BULK,REG,STATE,PING,STREAM)outgoing_timeoutsrequests_failed_on_disconnectchannel_close_by_typeis the counter that distinguishes partial impairment from a whole-node failure. Because a connection profile assigns separate sockets per channel type,REG/BULKdegrading whilePING/STATEstay healthy is a materially different fault from a node going away, and today the two are indistinguishable from node stats.These live in
_nodes/statsrather than the telemetry metrics registry so they are available on any cluster with no plugin installed and no configuration, which is what makes them usable during an active incident. Publishing the same valuesthrough
MetricsRegistryis a reasonable follow-up, along the lines ofNodeRuntimeMetrics(opensearch-project#20844), which exposesJvmStatsdata as pull-based gauges while_nodes/statsremains the source of truth.Where the counters are incremented
TcpTransport.ChannelsConnectedListener— connect latency is measured from listener construction to successful handshake;closeAndFailincrementsconnect_failures. ThecountDown.fastForward()guard on the failure and timeout paths means a single attempt cannot be counted twice.TcpTransport.NodeChannels— a per-channel close listener attributes socket closes to the channel types sharing that socket. The reverse index is derived fromConnectionProfile.ConnectionTypeHandle's ownoffset/lengthrange, so attribution matches the channel thatgetChannelactually selects.TransportService.onConnectionClosed— counts pruned in-flight handlers.TransportService.TimeoutHandler— counts expired requests.None of the counters sit on the per-request send path, so there is no added contention at request volume.
DEBUG logging
Also adds DEBUG logging for connect latency, send failures, and per-channel socket closes.
NodeConnectionsServicenow reports elapsed time on both successful and failed connects, sofailed to connect to {} after [Nms]distinguishes animmediate refusal from a full connect-timeout stall. Nothing is added at INFO, so production log volume is unchanged.
Sample output
GET _nodes/stats/transport:Backwards compatibility
New fields are written and read only when the negotiated stream version is 3.8.0 or later, so mixed-version clusters and cross-cluster connections to older clusters are unaffected — older nodes exchange only the pre-existing counters and report the new ones as zero.
This is purely additive to the
_nodes/statsresponse; no existing field changes name, type, or meaning. An emptychannel_close_by_typeis omitted rather than rendered as an empty object, so stats built without it are byte-identical to the previous response shape.TransportStatsalso gains aBuilder(TransportStats)copy constructor.TransportService.stats()previously restated all twelve fields by