feat: Adds `observability` domain with Sentry, Tempo, telemetry cost control skills by MajorLift · Pull Request #76 · MetaMask/skills

MajorLift · 2026-07-27T13:58:45Z

Motivation

The observability domain covers a) system telemetry via Sentry, Grafana Tempo, and b) traffic control for span volume.

Overview

Adds the observability domain with six skills on Sentry, Grafana Tempo, the Sentry span quota and release performance attribution.

sentry-quota reviews code and PRs for span instrumentation that risks the quota. It checks each span for three problems: a) fan-outs (loop, poller, selector, or trace callback forwarded into a MetaMask/core controller that emits multiple spans per trace), b) always-on with no per-trace sub-sampling gate, and c) no env-flag kill switch. All three together form an anti-pattern that causes anomalous span volume and quota breach.
- domains/observability/knowledge/span-sub-sampling.md: Knowledge file on the per-trace sub-sampling gate: a sub-rate keyed on the trace id and applied to a high-frequency custom span, so every gated span in a trace is kept or dropped together.
sentry-mcp-queries queries Sentry via MCP for triage, volume, release comparison and replay.
grafana-tempo-queries queries backend traces with TraceQL. Before trusting a result, it runs two checks. It confirms the Grafana session is live, because an expired session returns an empty body that reads as "no traces". It confirms the TraceQL filter is actually applied, by running a query whose filter can match no trace: if traces come back, the filter was ignored.
sentry-grafana-correlation joins a trace across Sentry and Tempo, and diagnoses why a trace appears in only one of them.
instrumentation creates and updates Sentry spans, MetaMetrics and Segment events.
performance-attribution attributes release-over-release p75/p95 movement in Sentry transactions, such as UI Startup, to specific code changes.

Public repo: hosts, datasource uids, org slugs and project ids are read from environment variables (GRAFANA_HOST, GRAFANA_ORG, TEMPO_UID, SENTRY_ORG, SENTRY_PROJECT_ID), not hardcoded.

sentry-mcp-queries: document data-fidelity loss on older releases for longer-range (30D+) queries (sample-rate drift, extrapolation hiding thin samples, retention downsampling) and percentile (p75+) sample-size/quality filtering; add table rows + pitfalls for stored-span-count and superseded-patch releases. performance-attribution: new skill for attributing release-over-release p75/p95 movements to code changes via black-box diff analysis, with an extension repo file (Trace Explorer queries, key transactions, highest-sample-patch version selection, 90d-vs-30d empirics, hot-path files, core-package changelog analysis, worked v13.11->v13.15 catalogue). analytics-instrumentation: cross-link volume-estimation caveats to the new fidelity guidance. CHANGELOG updated.

…n-out, latent trace? param, trace-arg PR-review scan, per-name budget

…cidental instrumentation)

CHANGELOG.md tracks consumer-facing changes to the `@metamask/skills` package, per CONTRIBUTING's "CLI / tooling changes" section. No merged skill-only PR adds an entry (#80, #78, #70, #62, #61 all touch zero changelog lines). Carrying one here bought nothing and was the sole source of this branch's conflict with `main`, since every skill PR edits the same `[Unreleased]` block. Restoring the file to its merge-base state makes the branch conflict-free without a merge commit.

MajorLift · 2026-07-30T20:43:14Z

Context budget

What this PR costs an agent, measured from an install, with tokens estimated as UTF-8 bytes/4. All six are maturity: experimental, so an install at the default --maturity stable gets none of them and pays nothing unless it names them with --include, which bypasses the maturity filter. An install into metamask-extension that opts in, with --maturity experimental or --include, pays in three tiers for each skill it installs.

Skill	Description (always loaded)	Selected (by the agent)	Selected + knowledge file
`grafana-tempo-queries`	713 bytes (~178 tok)	~2,220 tok	~3,269 tok
`instrumentation`	113 bytes (~28 tok)	~2,209 tok	~3,258 tok
`performance-attribution`	113 bytes (~28 tok)	~2,826 tok	~3,875 tok
`sentry-grafana-correlation`	870 bytes (~218 tok)	~2,566 tok	~3,615 tok
`sentry-mcp-queries`	92 bytes (~23 tok)	~3,453 tok	~4,502 tok
`sentry-quota`	138 bytes (~35 tok)	~3,361 tok	~4,410 tok
All six	2,039 bytes (~510 tok)	n/a, paid per selected skill	n/a, paid per selected skill

Description is the only tier an opted-in install pays unconditionally. The mms- output name is loaded beside each description and adds ~35 tok across the six, which this column leaves out.

Selected is the whole installed SKILL.md, frontmatter included, so it counts the description a second time on top of the Description tier. It is paid only when the agent selects the skill. Selected + knowledge file adds the observability domain's knowledge file, span-sub-sampling.md (~1,049 tok), if the agent then reads it. Only sentry-quota names the file, so for the other five skills nothing in the skill body points the agent at it.

Method

tools/install --repo metamask-extension --target <metamask-extension checkout> --maturity experimental --domain observability against this branch at 9939cce, measured on each of the six skills' installed directories. Four of the six (instrumentation, performance-attribution, sentry-mcp-queries, sentry-quota) have an overlay only for metamask-extension, so an install into any other repo skips them and gets only grafana-tempo-queries and sentry-grafana-correlation, at the figures above (~396 tok of descriptions).

The squash commit a31f624 carries the same six skill files, overlays, knowledge file and tools/install as the commit above, so these figures describe what merged until a later commit changes any of them.

The skill lives in the `analytics` domain, so the prefix repeated what the path and every discovery surface already show. Installed as `mms-instrumentation`.

"Cross-ref" describes the mechanism and reads as a generic pointer. The skill produces a correlation: one trace joined across two stores by `trace_id`, plus a reading of which absences are expected. The vendor names stay. They are load-bearing here rather than incidental — the skill is about the split between Sentry and Grafana Tempo specifically, and is not source-agnostic.

The setup block asked for `grafana_session` from an authenticated browser and offered no alternative, so following the skill as written meant extracting a session credential and exporting it into a shell. Sentry in the neighbouring skill already uses a scoped bearer token, which is the shape this should have had. A service account token, Viewer role, covers every query here. It is scoped to what the skill needs, revocable on its own, and attributable in an audit log to a purpose rather than to a person. The cookie stays as a documented fallback for instances that disallow service accounts, with the cost stated where someone deciding between them will read it: a session carries the whole of your Grafana authority rather than one datasource's read access, expires on a schedule you do not control, cannot be revoked without ending your own session, and is indistinguishable from you afterwards. The correlation skill's credential note said "an expired Grafana session"; it now names a token without datasource scope, so the two skills describe the same auth path.

## Summary - Adds `privacy-egress-diligence` to `security` — review for changes to what user data leaves the device. - Fills the one proof-standard domain with no lane today. Raised during [ADR-0058 review](MetaMask/decisions#173): security, observability/data, and ci/devex/build all have coverage; compliance/privacy had none. ## Why this surface `app/scripts/constants/sentry-state.ts` decides what egresses. Measured on `metamask-extension` at the current head: - **116 fields set to `true`** — the real value is copied and sent to Sentry. - Edited inside **ordinary feature PRs** — recent changes ride onboarding, swaps, rewards, and analytics-controller work. - **No CODEOWNERS entry**, so no privacy reviewer is automatically tagged. A file governing data egress, changed casually, reviewed by whoever happened to review the feature. ## Shape Deliberately the same as `lavamoat-policy`: detection is mechanical, judgement is not. `maskObject` (`shared/lib/object.utils.ts`) makes the risk one-way — unlisted fields degrade to a `typeof` string, so only a field promoted to `true` widens egress, and `git diff` finds those exactly. **The trap it names:** "the field is in the mask, so someone decided it was fine." The mask *is* the decision. Presence proves authorship, not review. So the deliverable is what each field holds at runtime — a mask path cannot distinguish `selectedTab` from `selectedAddress` — sorted into: | Bucket | Action | |---|---| | Safe | note the type that makes it safe (bounded enum, boolean, count) | | Needs narrowing | propose a nested mask keeping the shape, dropping the leaves | | Must not egress | propose `false` or a non-identifying substitute | It also covers the sibling pipes a PR widens at the same time: new MetaMetrics/Segment properties, and error strings interpolating runtime values, both of which leak regardless of the mask. **No accept/reject verdict.** Whether a field is acceptable to collect belongs to privacy and legal; a confident reviewer "this is fine" is exactly the artifact that lets an unreviewed field through. ## Notes - Experimental. One skill, no knowledge files. - Lands in `security` rather than a new `privacy` domain — a one-skill domain is premature, and the diligence pattern already lives there. Say the word if you'd rather it were separate. - Complements `instrumentation` (#76), which covers whether an event is correctly *identified* and consent-*gated*. This covers whether its payload is *sendable*. - No `CHANGELOG.md` entry: skill-only, no CLI change. --------- Co-authored-by: abretonc7s <107169956+abretonc7s@users.noreply.github.com> Co-authored-by: behroozreview <behrooz.aghakhanian@gmail.com>

…o `data` `analytics` covered two subjects under one name. Five of the six skills and `span-sub-sampling` are Sentry, Tempo and Grafana — traces, errors and the span budget, which is observability. The MetaMetrics and Segment material is product analytics, and with MetaMetrics migrating to `AnalyticsController` in both clients it needs a name that survives that. `instrumentation` stays in `observability`: it names Sentry spans alongside MetaMetrics and Segment events, but the body runs about three to one on spans. Known consequence, not yet resolved. `instrumentation` cites `metrametrics-identity` and `segment-governance`, and `sentry-quota` cites `segment-governance` — all three now cross a domain boundary. The installer copies domain knowledge per skill, so `data` shipping no skill of its own means that knowledge installs nowhere until it gains one.

`span-sub-sampling` still declared `domain: analytics` after the rename. `lint-skill-entry` requires `domain` to match the parent directory, so the rename shipped a lint error — caught by checking the merge into `main`, not by checking this branch, which predates `tools/skill-schema.mjs` and cannot run the linter at all. `domains/data/` moves to #143 with the same fix applied.

Both platform teams co-own the domain, as they do `coding`, `general`, `performance` and `pr-workflow`. A client's overlay is owned by that client's platform team, placed directly under the domain line so it wins on last-match and so each domain PR inserts at its own anchor.

metamask-extension#44451 (reduce Sentry trace sampling) lowered the fallback `tracesSampleRate` from 0.0075 to 0.005, and `tracesSampler` takes precedence over it. Span `count()` is already extrapolated, so multiplying it by the inverse rate counted twice. The Sentry project is `metamask`. A backend span that starts after its supposed parent ended is not its child. Routing to Sentry is decided per trace, and the background continues UI contexts as sampled. The removed `isOptIn` rule is replaced by the `excludeMetaMetricsId` behavior, and knowledge that moved to the `data` domain is named by its path there.

Check the stored `count_sample()` before trusting an extrapolated count, run an unfiltered control before reading a zero, and treat an issue as a grouping bucket rather than a fault. The serialized trace context carries no sampled flag, UI and background clocks can disagree by an hour, and a dependency or SDK-integration change can add span volume with no `trace(` in the diff. Release filters, ingestion lag and error metering get their limits stated, and the correlation skill names phantom parents, window traps and the time a CDN or queued connection adds.

A trace name a PR adds or reroutes may already be live in the target project, and then the review is a change to measured volume. For a name new to the target, the other client where it already ships is the reference class, and only the names the diff can actually start there carry over.

…-description-over-1024 * origin/main: feat: add navigation skill with MetaMask Mobile overlay (#148) feat: add feature-flags skill with MetaMask Mobile overlay (#147) feat: add analytics skill with MetaMask Mobile overlay (#140) feat: Adds `observability` domain with Sentry, Tempo, telemetry cost control skills (#76) refactor(perps): review skill composes the perps library instead of carrying knowledge copies (#159) # Conflicts: # README.md

MajorLift added 7 commits June 5, 2026 14:52

Add analytics domain: Sentry quota, MCP queries, instrumentation

4a4cd85

Merge branch 'main' into add/analytics

b72ded1

sentry-quota: catch incidental instrumentation — memoized-selector fa…

837ba90

…n-out, latent trace? param, trace-arg PR-review scan, per-name budget

CHANGELOG: note expanded sentry-quota detection (selector fan-out, in…

0061f40

…cidental instrumentation)

feat(analytics): add grafana-tempo-queries skill

71bf451

feat(analytics): add sentry-grafana-cross-ref skill

5590e2d

MajorLift mentioned this pull request Jul 30, 2026

feat: Adds security domain - /supply-chain-audit, /lavamoat-policy skills #83

Open

MajorLift changed the title ~~feat(analytics): add grafana-tempo-queries and sentry-grafana-cross-ref skills~~ feat(analytics): add analytics domain — Sentry, MetaMetrics, Segment, Grafana Tempo Jul 30, 2026

MajorLift changed the title ~~feat(analytics): add analytics domain — Sentry, MetaMetrics, Segment, Grafana Tempo~~ feat(analytics): add Sentry, MetaMetrics, Segment, and Tempo skills Jul 30, 2026

MajorLift marked this pull request as draft July 30, 2026 14:03

MajorLift marked this pull request as ready for review July 30, 2026 18:15

MajorLift mentioned this pull request Jul 30, 2026

feat: Adds security domain - /privacy-egress-diligence skill #90

Merged

MajorLift added 2 commits July 31, 2026 08:33

Rename analytics-instrumentation to instrumentation

4d330e4

The skill lives in the `analytics` domain, so the prefix repeated what the path and every discovery surface already show. Installed as `mms-instrumentation`.

This was referenced Aug 1, 2026

feat: init perps sentry implementation MetaMask/metamask-extension#41528

Merged

feat: assets unify balance and traces MetaMask/metamask-extension#44978

Merged

[DO-NOT-MERGE] trial branch: all skill PRs merged, for install and testing #108

Draft

MajorLift changed the title ~~feat(analytics): add Sentry, MetaMetrics, Segment, and Tempo skills~~ feat: Adds analytics domain - Sentry, MetaMetrics, Segment, Tempo query, instrumentation, quota skills Aug 31, 2026

MajorLift mentioned this pull request Aug 31, 2026

feat: Adds agentic domain - /agent-run-cost skill #91

Open

MajorLift changed the title ~~feat: Adds analytics domain - Sentry, Tempo, MetaMetrics, Segment skills and knowledges~~ feat: Adds observability and data domains - Sentry, Tempo, Grafana skills; product-analytics knowledge Sep 1, 2026

MajorLift mentioned this pull request Sep 1, 2026

feat(platform): add the metamask-extension overlay for analytics #143

Draft

MajorLift changed the title ~~feat: Adds observability and data domains - Sentry, Tempo, Grafana skills; product-analytics knowledge~~ feat: add an observability domain — Sentry, Tempo and span-budget skills Sep 1, 2026

MajorLift changed the title ~~feat: add an observability domain — Sentry, Tempo and span-budget skills~~ feat: Adds observability domain with Sentry, Tempo, telemetry cost control skills Sep 1, 2026

MajorLift changed the title ~~feat: Adds observability domain with Sentry, Tempo, span-budget skills~~ feat: Adds observability domain with Sentry, Tempo, telemetry cost control skills Sep 1, 2026

MajorLift added 2 commits September 14, 2026 07:00

Merge main so CODEOWNERS changes land on its structure

a2d1a2e

MajorLift mentioned this pull request Sep 14, 2026

ci: default CODEOWNER for repo overlays go to respective platform teams #160

Draft

MajorLift added 2 commits September 14, 2026 08:08

Eamonn-O-Connor requested a review from rvelaz September 14, 2026 16:04

rvelaz approved these changes Sep 15, 2026

View reviewed changes

MajorLift self-assigned this Sep 15, 2026

MajorLift enabled auto-merge (squash) September 15, 2026 10:56

Merge branch 'main' into add/analytics-tracing-cross-ref

8e2762e

MajorLift merged commit a31f624 into MetaMask:main Sep 15, 2026
28 checks passed

Navigation Menu

Uh oh!

feat: Adds `observability` domain with Sentry, Tempo, telemetry cost control skills - #76

feat: Adds `observability` domain with Sentry, Tempo, telemetry cost control skills#76
MajorLift merged 19 commits into
MetaMask:mainfrom
MajorLift:add/analytics-tracing-cross-ref

MajorLift commented Jul 27, 2026 •

edited

Loading

Uh oh!

MajorLift commented Jul 30, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

MajorLift commented Jul 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Overview

Uh oh!

MajorLift commented Jul 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Context budget

Method

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

MajorLift commented Jul 27, 2026 •

edited

Loading

MajorLift commented Jul 30, 2026 •

edited

Loading