Skip to content

Add transport failure and connect latency counters to node stats - #1

Draft
jbacchus126 wants to merge 4 commits into
mainfrom
jbacchus/tcp-observability
Draft

Add transport failure and connect latency counters to node stats#1
jbacchus126 wants to merge 4 commits into
mainfrom
jbacchus/tcp-observability

Conversation

@jbacchus126

Copy link
Copy Markdown
Owner

Description

_nodes/stats currently reports transport volume only — rx_count, rx_size_in_bytes, tx_count, tx_size_in_bytes, server_open, and total_outbound_connections. There is no always-available signal for transport failure or connection-open latency, which makes a class of network faults hard to diagnose on a running cluster: the volume counters keep incrementing normally while requests silently time out or connections take seconds to establish.

The connection-open path is the largest gap. A slow or hung connect blocks the cluster applier thread, and no existing metric observes it at all — total_outbound_connections is incremented only after a handshake succeeds, so a node struggling to open connections looks identical to a healthy idle node.

This PR adds six counters to TransportStats:

Field Meaning
connect_failures Outbound connection opens that failed or timed out
connect_time_millis Cumulative time spent on connection opens that succeeded. Average open latency is connect_time_millis / total_outbound_connections
connect_time_millis_max Worst-case single connection open since node start. Useful for judging whether lowering transport.connect_timeout is safe
channel_close_by_type Individual channel sockets closed, keyed by channel type (RECOVERY, BULK, REG, STATE, PING, STREAM)
outgoing_timeouts Outbound requests that received no response within their timeout
requests_failed_on_disconnect In-flight requests cancelled because the connection closed before a response arrived

channel_close_by_type is the counter that distinguishes partial impairment from a whole-node failure. Because a connection profile assigns separate sockets per channel type, REG/BULK degrading while PING/STATE stay healthy is a materially different fault from a node going away, and today the two are indistinguishable from node stats.

These live in _nodes/stats rather than the telemetry metrics registry so they are available on any cluster with no plugin installed and no configuration, which is what makes them usable during an active incident. Publishing the same values
through MetricsRegistry is a reasonable follow-up, along the lines of NodeRuntimeMetrics (opensearch-project#20844), which exposes JvmStats data as pull-based gauges while _nodes/stats remains the source of truth.

Where the counters are incremented

  • TcpTransport.ChannelsConnectedListener — connect latency is measured from listener construction to successful handshake; closeAndFail increments connect_failures. The countDown.fastForward() guard on the failure and timeout paths means a single attempt cannot be counted twice.
  • TcpTransport.NodeChannels — a per-channel close listener attributes socket closes to the channel types sharing that socket. The reverse index is derived from ConnectionProfile.ConnectionTypeHandle's own offset/length range, so attribution matches the channel that getChannel actually selects.
  • TransportService.onConnectionClosed — counts pruned in-flight handlers.
  • TransportService.TimeoutHandler — counts expired requests.

None of the counters sit on the per-request send path, so there is no added contention at request volume.

DEBUG logging

Also adds DEBUG logging for connect latency, send failures, and per-channel socket closes. NodeConnectionsService now reports elapsed time on both successful and failed connects, so failed to connect to {} after [Nms] distinguishes an
immediate refusal from a full connect-timeout stall. Nothing is added at INFO, so production log volume is unchanged.

Sample output

GET _nodes/stats/transport:

"transport": {
  "server_open": 26,
  "total_outbound_connections": 143,
  "rx_count": 1043321,
  "rx_size_in_bytes": 884736102,
  "tx_count": 1043298,
  "tx_size_in_bytes": 901324887,
  "channel_close_by_type": {
    "RECOVERY": 0,
    "BULK": 12,
    "REG": 12,
    "STATE": 0,
    "PING": 0,
    "STREAM": 0
  },
  "outgoing_timeouts": 37,
  "requests_failed_on_disconnect": 214,
  "connect_failures": 6,
  "connect_time_millis": 8412,
  "connect_time_millis_max": 4903
}

Backwards compatibility

New fields are written and read only when the negotiated stream version is 3.8.0 or later, so mixed-version clusters and cross-cluster connections to older clusters are unaffected — older nodes exchange only the pre-existing counters and report the new ones as zero.

This is purely additive to the _nodes/stats response; no existing field changes name, type, or meaning. An empty channel_close_by_type is omitted rather than rendered as an empty object, so stats built without it are byte-identical to the previous response shape.

TransportStats also gains a Builder(TransportStats) copy constructor.
TransportService.stats() previously restated all twelve fields by

@jbacchus126
jbacchus126 force-pushed the jbacchus/tcp-observability branch from cfcb4e6 to 9ca395a Compare August 4, 2026 19:20
jbacchus26 and others added 4 commits August 5, 2026 10:37
Expose counters for transport connect-time and request-time failures via
_nodes/stats: connect_failures, connect_time_millis,
connect_time_millis_max, channel_close_by_type, outgoing_timeouts, and
requests_failed_on_disconnect.

The connect-time counters cover the connection-open path, which no
existing metric observes: a slow or hung connect blocks the cluster
applier thread without incrementing any request-level stat.
channel_close_by_type attributes socket closes to individual channel
types, which distinguishes impairment affecting only some types from a
whole-node failure.

Also log connect latency, send failures, and per-channel socket closes
at DEBUG to aid diagnosis without adding INFO-level noise.

Signed-off-by: Janae Bacchus <jbacchus@uber.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
The new transport counters were serialized unconditionally, so a node
running this change would read six fields that an older node never
wrote, desynchronizing the stream and failing _nodes/stats in a
mixed-version cluster or against an older remote cluster. Gate them on
3.8.0 so older nodes exchange only the original counters and report the
new ones as zero.

Add a TransportStats.Builder copy constructor and use it in
TransportService.stats(), which previously restated all twelve fields by
hand to override two. Any field added in future was silently dropped
there.

Cover the wire format with TransportStatsTests, including both
older-node directions, and exercise the new fields in NodeStatsTests.

Signed-off-by: Janae Bacchus <jbacchus@uber.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Closing one socket tears down its whole connection, which closes every
remaining socket, and each of those fired the counting listener. A
single failure was therefore reported as a failure of all six channel
types at once, and a deliberate disconnect looked identical to a hard
failure, leaving channel_close_by_type with no diagnostic value. The
counting and teardown listeners are now one listener that records the
close before closing the connection, so only the socket that caused the
teardown is counted and deliberate closes are not counted at all.

A socket carrying several channel types still counts against each of
them, since losing it affects all of them, so the total across types is
not the number of sockets closed. Say so on the field.

connect_failures missed sockets that fail to open, because that path
returns before any connect listener is registered. Unresolvable hosts
and exhausted file descriptors went uncounted.

connect_time_millis now starts its clock before the first socket is
opened rather than after, so it measures the whole open path, and the
field comments say what the latency covers and that the maximum is a
high watermark that is never reset.

A connect attempt rejected before it starts never records a start time,
so the failure log computed an elapsed time from zero, which is
meaningless and can be negative because nanoTime has no defined origin.
Report the duration only when an attempt actually ran.

Channel type keys are lowercased to match REST field conventions and
sorted so field order is stable whether the counters were collected
locally or read off the wire.

Co-authored-by: Cursor <cursoragent@cursor.com>
Signed-off-by: Janae Bacchus <jbacchus@uber.com>
A teardown attributes exactly one socket, the first to close. Sockets
failing at almost the same moment are not counted separately, because
once teardown is under way a closing socket cannot be distinguished from
one the teardown itself closed. Sampling the first failure still shows
which channel types are unhealthy, which is the point of the breakdown,
but the limit was not written down anywhere.

Also note that total_outbound_connections counts successful opens, so
dividing connect_time_millis by it gives a valid average.

Co-authored-by: Cursor <cursoragent@cursor.com>
Signed-off-by: Janae Bacchus <jbacchus@uber.com>
@jbacchus126
jbacchus126 force-pushed the jbacchus/tcp-observability branch from ef1f318 to c11f2a6 Compare August 5, 2026 17:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants