Skip to content

fix(network): close peer connections that go idle (backport of #4562) #4568

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
stevenvegt wants to merge 1 commit into
base: V5.4
Choose a base branch
from
Open

fix(network): close peer connections that go idle (backport of #4562) #4568

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion docs/pages/deployment/cli-reference.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,14 +46,15 @@ The following options apply to the server commands below:
--http.default.log string What to log about HTTP requests. Options are 'nothing', 'metadata' (log request method, URI, IP and response code), and 'metadata-and-body' (log the request and response body, in addition to the metadata). (default "metadata")
--http.default.tls string Whether to enable TLS for the default interface, options are 'disabled', 'server', 'server-client'. Leaving it empty is synonymous to 'disabled',
--internalratelimiter When set, expensive internal calls are rate-limited to protect the network. Always enabled in strict mode. (default true)
--jsonld.contexts.localmapping stringToString This setting allows mapping external URLs to local files for e.g. preventing external dependencies. These mappings have precedence over those in remoteallowlist. (default [https://nuts.nl/credentials/v1=assets/contexts/nuts.ldjson,https://www.w3.org/2018/credentials/v1=assets/contexts/w3c-credentials-v1.ldjson,https://w3c-ccg.github.io/lds-jws2020/contexts/lds-jws2020-v1.json=assets/contexts/lds-jws2020-v1.ldjson,https://schema.org=assets/contexts/schema-org-v13.ldjson])
--jsonld.contexts.localmapping stringToString This setting allows mapping external URLs to local files for e.g. preventing external dependencies. These mappings have precedence over those in remoteallowlist. (default [https://schema.org=assets/contexts/schema-org-v13.ldjson,https://nuts.nl/credentials/v1=assets/contexts/nuts.ldjson,https://www.w3.org/2018/credentials/v1=assets/contexts/w3c-credentials-v1.ldjson,https://w3c-ccg.github.io/lds-jws2020/contexts/lds-jws2020-v1.json=assets/contexts/lds-jws2020-v1.ldjson])
--jsonld.contexts.remoteallowlist strings In strict mode, fetching external JSON-LD contexts is not allowed except for context-URLs listed here. (default [https://schema.org,https://www.w3.org/2018/credentials/v1,https://w3c-ccg.github.io/lds-jws2020/contexts/lds-jws2020-v1.json])
--loggerformat string Log format (text, json) (default "text")
--network.bootstrapnodes strings List of bootstrap nodes ('<host>:<port>') which the node initially connect to.
--network.connectiontimeout int Timeout before an outbound connection attempt times out (in milliseconds). (default 5000)
--network.enablediscovery Whether to enable automatic connecting to other nodes. (default true)
--network.enabletls Whether to enable TLS for gRPC connections, which can be disabled for demo/development purposes. It is NOT meant for TLS offloading (see 'tls.offload'). Disabling TLS is not allowed in strict-mode. (default true)
--network.grpcaddr string Local address for gRPC to listen on. If empty the gRPC server won't be started and other nodes will not be able to connect to this node (outbound connections can still be made). (default ":5555")
--network.idletimeout duration Period without any received message after which a connection to a peer is closed and re-established (in Golang duration format, e.g. '2m'). Specify 0 to disable. (default 2m0s)
--network.maxbackoff duration Maximum between outbound connections attempts to unresponsive nodes (in Golang duration format, e.g. '1h', '30m'). (default 1h0m0s)
--network.nodedid string Specifies the DID of the organization that operates this node, typically a vendor for EPD software. It is used to identify the node on the network. If the DID document does not exist of is deactivated, the node will not start.
--network.protocols ints Specifies the list of network protocols to enable on the server. They are specified by version (1, 2). If not set, all protocols are enabled.
Expand Down
3 changes: 2 additions & 1 deletion docs/pages/deployment/server_options.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,14 +52,15 @@
http.default.auth.type Whether to enable authentication for the default interface, specify 'token_v2' for bearer token mode or 'token' for legacy bearer token mode.
http.default.cors.origin [] When set, enables CORS from the specified origins on the default HTTP interface.
**JSONLD**
jsonld.contexts.localmapping [https://nuts.nl/credentials/v1=assets/contexts/nuts.ldjson,https://www.w3.org/2018/credentials/v1=assets/contexts/w3c-credentials-v1.ldjson,https://w3c-ccg.github.io/lds-jws2020/contexts/lds-jws2020-v1.json=assets/contexts/lds-jws2020-v1.ldjson,https://schema.org=assets/contexts/schema-org-v13.ldjson] This setting allows mapping external URLs to local files for e.g. preventing external dependencies. These mappings have precedence over those in remoteallowlist.
jsonld.contexts.localmapping [https://www.w3.org/2018/credentials/v1=assets/contexts/w3c-credentials-v1.ldjson,https://w3c-ccg.github.io/lds-jws2020/contexts/lds-jws2020-v1.json=assets/contexts/lds-jws2020-v1.ldjson,https://schema.org=assets/contexts/schema-org-v13.ldjson,https://nuts.nl/credentials/v1=assets/contexts/nuts.ldjson] This setting allows mapping external URLs to local files for e.g. preventing external dependencies. These mappings have precedence over those in remoteallowlist.
jsonld.contexts.remoteallowlist [https://schema.org,https://www.w3.org/2018/credentials/v1,https://w3c-ccg.github.io/lds-jws2020/contexts/lds-jws2020-v1.json] In strict mode, fetching external JSON-LD contexts is not allowed except for context-URLs listed here.
**Network**
network.bootstrapnodes [] List of bootstrap nodes ('<host>:<port>') which the node initially connect to.
network.connectiontimeout 5000 Timeout before an outbound connection attempt times out (in milliseconds).
network.enablediscovery true Whether to enable automatic connecting to other nodes.
network.enabletls true Whether to enable TLS for gRPC connections, which can be disabled for demo/development purposes. It is NOT meant for TLS offloading (see 'tls.offload'). Disabling TLS is not allowed in strict-mode.
network.grpcaddr \:5555 Local address for gRPC to listen on. If empty the gRPC server won't be started and other nodes will not be able to connect to this node (outbound connections can still be made).
network.idletimeout 2m0s Period without any received message after which a connection to a peer is closed and re-established (in Golang duration format, e.g. '2m'). Specify 0 to disable.
network.maxbackoff 1h0m0s Maximum between outbound connections attempts to unresponsive nodes (in Golang duration format, e.g. '1h', '30m').
network.nodedid Specifies the DID of the organization that operates this node, typically a vendor for EPD software. It is used to identify the node on the network. If the DID document does not exist of is deactivated, the node will not start.
network.protocols [] Specifies the list of network protocols to enable on the server. They are specified by version (1, 2). If not set, all protocols are enabled.
Expand Down
1 change: 1 addition & 0 deletions docs/pages/release_notes.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ Unreleased
- Network: a peer that rejects an outbound connection with ``already connected`` is now retried with exponential backoff instead of every 1 to 5 seconds. Previously such a peer was retried every 1 to 5 seconds indefinitely, and never connected. By @stevenvegt in https://github.com/nuts-foundation/nuts-node/pull/4467
- Network: the default of ``network.maxbackoff`` is lowered from ``24h`` to ``1h``. The backoff is persisted across restarts and only reset when a peer's NutsComm address changes, so a peer that was unreachable for a few days could previously go unattempted for up to a day after it came back.
- Network: failed connection attempts are now logged at debug level instead of warning level. A warning is logged once, on the attempt that reaches ``network.maxbackoff``, so an unreachable peer no longer repeats the same warning on every retry.
- Network: connections on which no message was received for ``network.idletimeout`` (default ``2m``) are now closed and re-established. Peers send gossip and diagnostics messages every few seconds, so a silent connection is a dead one: typically a half-open TCP connection or a reverse proxy that kept the stream open after the other side went away. Previously such connections lingered until the proxy or node was restarted, and the peer holding the stale connection rejected new connections with ``already connected``. Set ``network.idletimeout`` to ``0`` to disable. By @stevenvegt in https://github.com/nuts-foundation/nuts-node/pull/4562

## Security

Expand Down
1 change: 1 addition & 0 deletions network/cmd/cmd.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,7 @@ func FlagSet() *pflag.FlagSet {
"(outbound connections can still be made).")
flagSet.Int("network.connectiontimeout", defs.ConnectionTimeout, "Timeout before an outbound connection attempt times out (in milliseconds).")
flagSet.Duration("network.maxbackoff", defs.MaxBackoff, "Maximum between outbound connections attempts to unresponsive nodes (in Golang duration format, e.g. '1h', '30m').")
flagSet.Duration("network.idletimeout", defs.IdleTimeout, "Period without any received message after which a connection to a peer is closed and re-established (in Golang duration format, e.g. '2m'). Specify 0 to disable.")
flagSet.StringSlice("network.bootstrapnodes", defs.BootstrapNodes, "List of bootstrap nodes ('<host>:<port>') which the node initially connect to.")
flagSet.Bool("network.enablediscovery", defs.EnableDiscovery, "Whether to enable automatic connecting to other nodes.")
flagSet.String("network.nodedid", defs.NodeDID, "Specifies the DID of the organization that operates this node, typically a vendor for EPD software. It is used to identify the node on the network. If the DID document does not exist of is deactivated, the node will not start.")
Expand Down
4 changes: 4 additions & 0 deletions network/config.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,9 @@ type Config struct {
ConnectionTimeout int `koanf:"connectiontimeout"`
// MaxBackoff specifies the maximum backoff for outbound connections
MaxBackoff time.Duration `koanf:"maxbackoff"`
// IdleTimeout specifies the period without any received message after which a connection to a peer is closed.
// Peers send gossip and diagnostics messages at a fixed interval, so a silent connection is a dead one. Zero disables the check.
IdleTimeout time.Duration `koanf:"idletimeout"`
// Public address of this nodes other nodes can use to connect to this node.
BootstrapNodes []string `koanf:"bootstrapnodes"`
// Protocols is the list of network protocols to enable on the server. They are specified by version (v1, v2).
Expand Down Expand Up @@ -65,6 +68,7 @@ func DefaultConfig() Config {
GrpcAddr: ":5555",
ConnectionTimeout: 5000,
MaxBackoff: time.Hour,
IdleTimeout: 2 * time.Minute,
ProtocolV2: v2.DefaultConfig(),
EnableDiscovery: true,
}
Expand Down
1 change: 1 addition & 0 deletions network/config_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@ func TestDefaultConfig(t *testing.T) {
defs := DefaultConfig()
assert.Equal(t, ":5555", defs.GrpcAddr)
assert.Equal(t, time.Hour, defs.MaxBackoff, "a peer that comes back after a long outage should be retried within the hour")
assert.Equal(t, 2*time.Minute, defs.IdleTimeout)
}

func TestConfig_IsProtocolEnabled(t *testing.T) {
Expand Down
1 change: 1 addition & 0 deletions network/network.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -241,6 +241,7 @@ func (n *Network) Configure(config core.ServerConfig) error {
if n.connectionManager == nil {
grpcOpts := []grpc.ConfigOption{
grpc.WithConnectionTimeout(time.Duration(n.config.ConnectionTimeout) * time.Millisecond),
grpc.WithIdleTimeout(n.config.IdleTimeout),
grpc.WithBackoff(func() grpc.Backoff {
return grpc.BoundedBackoff(time.Second, n.config.MaxBackoff)
}),
Expand Down
Loading
Loading