Conversation
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
mtail turns captured log fields into metric label values, and a new label-value tuple gets appended to Metric.LabelValues under the metric write lock inside GetDatum every time a program sees one it has not recorded before. The periodic Store.Gc loop and Metric.RemoveOldestDatum walk that same slice to trim expired or over-limit entries, but they hold only the store search lock, never the metric own lock, so a GC tick racing with a steady stream of fresh labels from the logs is a real data race on the slice header that go test -race reports. The default reflection marshalling of a Metric reads LabelValues the same unlocked way, so scraping /json or /debug/vars while the VM is updating metrics trips it too, and a torn read of the header can index past the backing array rather than just return stale data. I ran into it driving a writer that adds labels against the store tests under the race detector. The fix reads LabelValues under m.RLock() in Gc and RemoveOldestDatum, collecting the entries to drop and releasing the lock before the Remove* helpers reacquire it as writers, and adds a Metric.MarshalJSON that takes the read lock and marshals through a method-less alias so the output stays byte-identical. A -race regression test runs GetDatum against Gc and MarshalJSON concurrently.