device: add repro for permanent device leak via stale indexTable entry + queue finalizer cycle#65
Draft
lkosewsk wants to merge 2 commits into
Draft
device: add repro for permanent device leak via stale indexTable entry + queue finalizer cycle#65lkosewsk wants to merge 2 commits into
lkosewsk wants to merge 2 commits into
Conversation
To quote Claude here, because this is unfamiliar code to me: There's a leak as a result of a check-then-act race in wireguard-go. - Every timer-arming .Mod() is guarded by timersActive() (isRunning && device.isUp()), but the guard and the .Mod() aren't atomic with Peer.Stop(). - A send/handshake goroutine that passes the guard, then gets preempted while Stop() runs (isRunning=false, timersStop()), then runs its .Mod(), re-arms a timer nothing will ever stop (second Stop() early-returns on !isRunning). - That pending AfterFunc(time.Hour) pins peer → device → 64 KiB buffer pools. No goroutine leaks (so goroutine counts stay flat) and -race is clean (it's a logical race, not a data race) The problem was discovered by an Aperture issue where an instance with a bad configuration was stuck in a restart loop (the fact that happened is *Aperture's* problem), but as a result of that loop, wireguard via tsnet kept memory leaking, quickly piling on RSS - see the TIMER_LEAK_REPRO.md file for more details and how to run the test to prove the point. Signed-off-by: Luke Kosewski <lkosewsk@tailscale.com>
Signed-off-by: Luke Kosewski <lkosewsk@tailscale.com>
bradfitz
reviewed
Jun 3, 2026
Comment on lines
+5
to
+6
| // peer's autodraining queues carry a finalizer that captures *device. A | ||
| // finalizable object trapped in a reference cycle is never collected by Go, so |
Member
There was a problem hiding this comment.
A finalizable object trapped in a reference cycle is never collected by Go
nit: Go’s docs say this is not guaranteed, not formally "never".
"Finalizers are run in dependency order: if A points at B, both have finalizers, and they are otherwise unreachable, only the finalizer for A runs; once A is freed, the finalizer for B can run. If a cyclic structure includes a block with a finalizer, that cycle is not guaranteed to be garbage collected and the finalizer is not guaranteed to run, because there is no ordering that respects the dependencies."
bradfitz
added a commit
that referenced
this pull request
Jun 4, 2026
In #65, @lkosewsk reproduced a memory leak seen in prod with lots of wireguard-go instances being created and destroyed, where they were still being retained forever due to cycles in the runtime.SetFinalizer reference graph. Really we shouldn't be using runtime.SetFinalizer anywhere. But we still use it on mobile platforms in WaitPool. But those platforms don't have thousands of tsnet.Server instances coming & going, so this is a half fix: avoid the finalizer registration on Linux, etc where the queue doesn't need to be drained and there's no WaitPool accounting. Just let GC handle it, without adding finalizer cycle complexity. Updates tailscale/corp#42776 Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This draft adds a failing test (device/leakrepro_test.go) that reproduces a
permanent device leak, so it's easy to check out and run: go test ./device/
-run TestLeak -v. See ./LEAK_REPRO.md for more.
The leak: tearing a peer down while a handshake is in flight can leave a stale
entry in device.indexTable pointing at the removed peer. Since each peer's
autodraining queues carry a finalizer that captures *device
(runtime.SetFinalizer(q, device.flushInboundQueue) in channels.go), that stale
device → peer → queue → finalizer → device reference is a cycle containing a
finalizable object, which Go never collects. The whole *Device leaks — its 64
KiB PopulatePools buffers and everything else. It's permanent (not
timer-bounded), leaks no goroutine, and -race stays silent, so it's easy to
miss; the symptom is unbounded RSS growth when devices are repeatedly brought
up and torn down (e.g. a tsnet Server.Close() that races an in-flight
connection).
Why it happens: Peer.Stop() early-returns on !isRunning, so its indexTable
cleanup (ZeroAndFlushAll) only runs on the first stop. Meanwhile registering
an index entry (SendHandshakeInitiation → CreateMessageInitiation, and
Timer.Mod) is guarded by timersActive() but not atomic with Stop(), so a
handshake send — directly, or via a leaked pending handshake timer firing
after the stop — can register a fresh entry after the cleanup ran. isRunning
is now false, so no later Stop() ever clears it.
The tests pin down the mechanism:
even a bare index entry with no crypto) leaks; clearing the queue finalizers
frees it either way.
fire; keepalive/zero-key timers self-heal.
leaked state ~32k/100k iterations; clean under -race.
Likely fix suggested by Claude (not included here — this is just the repro): clear the peer's
indexTable entries unconditionally on removal (in
removePeerLocked/RemoveAllPeers) instead of relying on Stop()'s
isRunning-gated path, so the device → peer edge can't survive removal.