Skip to content

lock metric reads in Store.Gc and json marshal to fix data race - #1014

Open
sayed0200 wants to merge 1 commit into
google:mainfrom
sayed0200:metric-labelvalues-race
Open

sayed0200 wants to merge 1 commit into
google:mainfrom
sayed0200:metric-labelvalues-race

Conversation

@sayed0200

Copy link
Copy Markdown

mtail turns captured log fields into metric label values, and a new label-value tuple gets appended to Metric.LabelValues under the metric write lock inside GetDatum every time a program sees one it has not recorded before. The periodic Store.Gc loop and Metric.RemoveOldestDatum walk that same slice to trim expired or over-limit entries, but they hold only the store search lock, never the metric own lock, so a GC tick racing with a steady stream of fresh labels from the logs is a real data race on the slice header that go test -race reports. The default reflection marshalling of a Metric reads LabelValues the same unlocked way, so scraping /json or /debug/vars while the VM is updating metrics trips it too, and a torn read of the header can index past the backing array rather than just return stale data. I ran into it driving a writer that adds labels against the store tests under the race detector. The fix reads LabelValues under m.RLock() in Gc and RemoveOldestDatum, collecting the entries to drop and releasing the lock before the Remove* helpers reacquire it as writers, and adds a Metric.MarshalJSON that takes the read lock and marshals through a method-less alias so the output stays byte-identical. A -race regression test runs GetDatum against Gc and MarshalJSON concurrently.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant