feat: Adds observability domain with Sentry, Tempo, telemetry cost control skills

.github/CODEOWNERS

@@ -16,6 +16,9 @@
 /domains/assets/         @MetaMask/metamask-assets
 /domains/coding/         @MetaMask/extension-platform @MetaMask/mobile-platform @MetaMask/core-platform
 /domains/general/        @MetaMask/extension-platform @MetaMask/mobile-platform
+/domains/observability/  @MetaMask/extension-platform @MetaMask/mobile-platform
+/domains/observability/skills/*/repos/metamask-extension.md @MetaMask/extension-platform
+/domains/observability/skills/*/repos/metamask-mobile.md    @MetaMask/mobile-platform
 /domains/performance/    @MetaMask/extension-platform @MetaMask/mobile-platform
 /domains/perps/          @MetaMask/perps
 /domains/pr-workflow/    @MetaMask/extension-platform @MetaMask/mobile-platform

domains/observability/knowledge/span-sub-sampling.md

@@ -0,0 +1,80 @@
+---
+name: span-sub-sampling
+domain: observability
+description: Deterministic per-trace sub-sampling for high-frequency custom spans — span sub-rate, traceId-hash bucketed
+---
+
+# Span Sub-Sampling
+
+Durable fix for a custom span that fans out and eats the span budget. Gate the span with a per-trace sub-rate, keyed on the trace id so every gated span in a trace is kept-or-dropped together. Source: [PR #39891](https://github.com/MetaMask/metamask-extension/pull/39891) (`shared/lib/wrapper-sampling.ts`).
+
+## Rate Math
+
+```
+effective rate = sample rate of the trace context the span joins × span sub-rate
+```
+
+- A span that joins its own realm's head-sampled trace inherits the global `tracesSampleRate` (extension prod: 0.5%), so the sub-rate cuts on top: `0.5% × 1% = 0.005%`.
+- A span that joins a context propagated as sampled inherits a rate of 1, not the global rate. The extension background continues every UI context it receives as sampled, so its `rpc.handler` spans are kept at the sub-rate alone unless a per-name override or the remote `sentry.tracesSampleRate` ceiling applies.
+- PR #39891 ships a sub-rate of 0.5% (`WRAPPER_SAMPLE_RATE = 0.005`) — a conservative pilot — and names 5% as the step-up once the denylist is confirmed effective in production.
+
+Pick the sub-rate from how many sampled traces the metric needs to stay useful — not from the quota alone. Too low and the metric goes dark.
+
+## Pattern
+
+```ts
+const WRAPPER_SAMPLE_RATE = 0.005;
+
+// Deterministic: same answer for the same traceId, so every gated span in a
+// trace is kept or dropped together. The trace's root is sampled separately.
+export function shouldSampleWrappers(traceId: string | undefined): boolean {
+  if (!traceId || traceId.length < 8) {
+    return false;
+  }
+  const hashBucket = parseInt(traceId.slice(0, 8), 16) % 10000;
+  return hashBucket < WRAPPER_SAMPLE_RATE * 10000;
+}
+```
+
+**Why deterministic, not `Math.random()` per call:** independent per-span sampling shreds a trace into partial waterfalls (some spans present, siblings missing) — useless for attribution. Hashing the trace id makes keep/drop the same for every gated span in the trace. It does not tie them to the trace's root, which head sampling draws on its own: with a 0.5% root rate, about 99.5% of traces that keep wrapper spans are expected to lack their root.
+
+## Gate Order (cheapest check first)
+
+```ts
+let traceId: string | undefined;
+try {
+  traceId = sentryGetActiveSpan()?.spanContext().traceId;
+} catch {
+  // The span may have ended or be invalid. A tracing failure must not escape
+  // into the work itself, so fall through with no trace id.
+}
+if (!traceId || isReadOnlyAction(action) || !shouldSampleWrappers(traceId)) {
+  return doWorkWithoutSpan();
+}
+return trace({ name, op, data }, doWorkWithSpan);
+```
+
+1. No active trace → no span.
+2. Denylist → skip noise (below).
+3. Sub-sample miss → skip this trace's spans.
+
+## Denylist: cut before you sample
+
+Drop spans with no timing/attribution signal before sub-sampling. In PR #39891, read-only verbs are ~90% of `messenger.call` volume:
+
+```ts
+const READ_ONLY_VERB = /^(?:get|has|find|is|peek)(?:[A-Z]|$)/u;
+```
+
+The denylist covers `messenger.call` only. `rpc.handler` spans reach the sub-sample gate with no denylist, and 34.9% of them were read-only in one recorded measurement.
+
+Removing ~90% of volume before the sample multiplies headroom — a higher sub-rate then yields the same span budget, so kept traces are denser and more useful.
+
+## Where the Gate Goes
+
+- **Consumer (extension):** spans go through `trace()`. Gate at the call site, or for a whole span family inside the wrapper. `traceId` from `sentryGetActiveSpan()?.spanContext().traceId`.
+- **Controller package (core):** controllers call an injected `trace` callback. Gate in the package's trace util or the callback so every consumer inherits the cap. Pull the trace id from the controller's tracing context, not a fresh Sentry import.
+
+## Kill Switch
+
+Ship every always-on span family with an env disable flag (PR #39891: `SENTRY_DISTRIBUTED_TRACING_DISABLED` returns the messenger un-wrapped). It turns a future emergency cut into a config flip instead of a cherry-pick.

domains/observability/skills/grafana-tempo-queries/skill.md

@@ -0,0 +1,143 @@
+---
+name: grafana-tempo-queries
+description: Query backend traces in Grafana Tempo with TraceQL — find traces by service or span attribute, fetch a trace by id, inspect its span tree, and enumerate tag values. Covers the datasource-proxy access path, the credential-expiry failure that returns empty results indistinguishable from "no data", the negative control that proves a filter actually applied, and the id/kind/base64 decoding quirks in the response. Use when investigating backend latency, checking what the backend recorded for a request, or establishing which infrastructure tiers a trace reaches. Triggers on Tempo, TraceQL, Grafana traces, backend span inspection, "does the backend have this trace", or tracing a request past the API boundary.
+maturity: experimental
+---
+
+# grafana-tempo-queries
+
+Tempo holds **backend** spans. Client spans from the extension and mobile go to Sentry via the SDK's own transport and never appear here — so a Tempo trace normally starts at an inbound server span, and a missing root is expected rather than broken. Mobile sends no trace context to the backend: its `tracePropagationTargets` name no backend host and it does not set `propagateTraceparent`, so a backend trace for a mobile request shares no trace id with the mobile client's spans. To join the two halves, see `sentry-grafana-correlation`.
+
+## Setup
+
+Everything goes through Grafana's datasource proxy. Authenticate with a service account token. Keep the host, datasource uid, org id, and token in your environment — this repository is public, so never commit them.
+
+```bash
+# Set these once per shell, from your own Grafana instance:
+#   GRAFANA_HOST   e.g. https://grafana.<your-org-domain>
+#   TEMPO_UID      the Tempo datasource uid (see discovery below)
+#   GRAFANA_ORG    the numeric org id the datasource belongs to
+#   GRAFANA_TOKEN  a service account token, Viewer role, scoped to the Tempo datasource
+BASE="$GRAFANA_HOST/api/datasources/proxy/uid/$TEMPO_UID"
+AUTH=(-H "Authorization: Bearer $GRAFANA_TOKEN" -H "X-Grafana-Org-Id: $GRAFANA_ORG")
+```
+
+Create the token under Administration, Service accounts. A Viewer-role account is
+sufficient for every query in this skill, and the token is revocable on its own without
+disturbing anything else you have open.
+
+If your Grafana disallows service accounts and there is genuinely no token path, a browser
+`grafana_session` cookie works in the same header slot:
+
+```bash
+AUTH=(-H "Cookie: grafana_session=$GRAFANA_SESSION" -H "X-Grafana-Org-Id: $GRAFANA_ORG")
+```
+
+Reach for that only after confirming a token cannot be issued. A session cookie carries
+your whole Grafana authority rather than one datasource's read access, expires on a
+schedule you do not control, and cannot be revoked without ending your own session. It is
+also indistinguishable from you in an audit log.
+
+Discover the datasource uid rather than guessing it:
+
+```bash
+curl -s "$GRAFANA_HOST/api/datasources" "${AUTH[@]}" \
+  | node -e 'JSON.parse(require("fs").readFileSync(0)).filter(d=>d.type==="tempo").forEach(d=>console.log(d.uid,d.name))'
+```
+
+## Check the instrument before believing a result
+
+**A stale session returns HTTP 401 with an empty body, and a naive parser reports that as zero results** — indistinguishable from "this data does not exist". This is the single most expensive failure mode here: it produces confident negative conclusions about instrumentation coverage.
+
+```bash
+# 1. Prove you are authenticated. Do this first, every session.
+curl -s -o /dev/null -w 'grafana auth: HTTP %{http_code}\n' "$GRAFANA_HOST/api/user" "${AUTH[@]}"
+
+# 2. Prove the filter is actually being applied, with a query that must match nothing.
+curl -s -G "$BASE/api/search" "${AUTH[@]}" \
+  --data-urlencode 'q={span.db.system = "not-a-real-db-xyz"}' \
+  --data-urlencode "start=$START" --data-urlencode "end=$NOW" \
+  | node -e 'const j=JSON.parse(require("fs").readFileSync(0));console.log("control traces:",(j.traces||[]).length,"(must be 0)")'
+```
+
+If several different filters all return exactly your `limit`, the filter is not being applied — treat the results as unfiltered until the negative control returns 0.
+
+## Core queries
+
+Every endpoint wants an explicit epoch-seconds window. Omitting it on a by-id lookup makes the request hunt across all blocks and hit a context deadline. A window is not always enough: a by-id lookup of a session-length trace has returned 500 with one.
+
+```bash
+NOW=$(date +%s); START=$((NOW-3600))
+```
+
+**Search by TraceQL.** Returns trace summaries plus the spans that matched.
+
+```bash
+curl -s -G "$BASE/api/search" "${AUTH[@]}" \
+  --data-urlencode 'q={resource.service.name="my-service"}' \
+  --data-urlencode "start=$START" --data-urlencode "end=$NOW" \
+  --data-urlencode "limit=20"
+```
+
+**Fetch one trace in full** (OTLP JSON: resource batches → scope spans → spans).
+
+```bash
+curl -s "$BASE/api/traces/$TRACE_ID?start=$START&end=$NOW" "${AUTH[@]}"
+```
+
+**Enumerate values for a tag** — useful for inventorying what a fleet emits. Expect a `502` on high-cardinality tags; fall back to inspecting individual traces rather than concluding the tag is unused.
+
+```bash
+curl -s -G "$BASE/api/v2/search/tag/span.db.system/values" "${AUTH[@]}" \
+  --data-urlencode "start=$START" --data-urlencode "end=$NOW"
+```
+
+## TraceQL patterns worth knowing
+
+| Goal | Query |
+| --- | --- |
+| One service | `{resource.service.name="svc-name"}` |
+| Several services | `{resource.service.name=~"(svc-a|svc-b)-prd"}` |
+| Attribute present at all | `{span.db.system != nil}` |
+| Attribute absent, which a `!=` comparison skips | `{span.db.system = nil}` |
+| Span kind | `{kind=server}`, `{kind=client}` |
+| Slow spans | `{duration > 1s}` |
+| **Two conditions anywhere in the same trace** | `{resource.service.name="svc-a"} && {span.db.system != nil}` |
+
+The last one is the important one: `&&` between two brace groups is a **trace-level** conjunction, not a single-span filter. It answers "does a request into this service reach a database at all", which is how you map how deep a trace goes without reading traces one at a time.
+
+## Reading the response
+
+- **Span and trace ids are base64**, not hex. Decode before comparing them to anything from a header or from Sentry: `Buffer.from(id,"base64").toString("hex")`.
+- **`kind` is a string** (`SPAN_KIND_SERVER`, `SPAN_KIND_CLIENT`, `SPAN_KIND_INTERNAL`), not the numeric enum. Filtering on `sp.kind === 2` silently matches nothing.
+- **Search results drop leading zeros from trace ids.** A 31-character id is a 32-character id with a leading zero; zero-pad before using it anywhere else, or the lookup fails for a reason that looks like absence.
+- **`rootServiceName: "<root span not yet received>"`** means the trace's root is not in Tempo. For client-originated requests that is the normal case — the root is a client span living in Sentry — and it is the marker for finding them.
+- Resource attributes carry deployment context (`service.name`, kubernetes pod/namespace/cluster, region); span attributes carry the request (`http.*`, `net.*`, `db.*`).
+
+## Deep links for sharing
+
+A link is more useful than a pasted id. Build a Grafana Explore URL with the query pre-filled:
+
+```bash
+node -e '
+const left={datasource:process.env.TEMPO_UID,
+  queries:[{refId:"A",datasource:{type:"tempo",uid:process.env.TEMPO_UID},queryType:"traceql",query:process.argv[1]}],
+  range:{from:"now-6h",to:"now"}};
+console.log(`${process.env.GRAFANA_HOST}/explore?orgId=${process.env.GRAFANA_ORG}&left=${encodeURIComponent(JSON.stringify(left))}`);
+' '<trace-id-or-traceql>'
+```
+
+Prefer an absolute `from`/`to` when the link needs to outlive the event; a relative window slides off it and the reader opens an empty result. The Explore state in a link can be dropped across an SSO redirect, so a reader who is not yet signed in can land on an empty Explore.
+
+## Failure modes
+
+| Symptom | Cause | Response |
+| --- | --- | --- |
+| All queries return 0 | Session expired (401, empty body) | Check `/api/user` first |
+| Every filter returns exactly `limit` | Filter not applied | Run the negative control |
+| By-id lookup times out | No time window | Pass `start`/`end` |
+| Tag-values returns 502 | High cardinality | Inspect traces directly |
+| Cloudflare error 1015 instead of JSON | Parallel requests tripped the rate limit in front of Grafana | Send requests one at a time |
+| Id from search not found elsewhere | Leading zeros stripped | Zero-pad to 32 chars |
+| Kind filter matches nothing | Comparing to a number | Compare to `SPAN_KIND_*` |
+| Trace has no root | Root is a client span | Expected; see `sentry-grafana-correlation` |

domains/observability/skills/instrumentation/repos/metamask-extension.md

@@ -0,0 +1,68 @@
+---
+repo: metamask-extension
+parent: instrumentation
+---
+
+## Key Files
+
+| Content | Path |
+|---------|------|
+| Sentry trace wrapper | `shared/lib/trace.ts` |
+| Trace name enum | `shared/lib/trace.ts` → `TraceName` |
+| MetaMetrics controller | `app/scripts/controllers/metametrics-controller.ts` |
+| Anonymous-event marking (`excludeMetaMetricsId`) | `app/scripts/controllers/analytics/analytics.ts` → `applyAnonymousEventOptions()` |
+| Event enum | `shared/constants/metametrics.ts` → `MetaMetricsEventName` |
+| Sentry setup + sample rate | `app/scripts/lib/setupSentry.js` → `getTracesSampleRate()` |
+| Sentry `user.id` (set to the MetaMetrics `analyticsId`) | `app/scripts/lib/sentry-metametrics.ts` → `metaMetricsIntegration()` |
+| Segment tracking plan | `Consensys/segment-schema` → `tracking-plans/metamask-extension.yaml`, which lists event libraries. Event definitions live in `libraries/events/<library>/` |
+
+## Cross-Process Context (UI → Background)
+
+The extension has two Sentry hubs — one in the UI process and one in the background service worker. A trace starting in UI and continuing in background requires explicit context propagation across the RPC boundary:
+
+```typescript
+// Serialize at UI call site
+const context: SerializedTraceContext = {
+  _name: TraceName.MyOperation,
+  _traceId: span.spanContext().traceId,
+  _spanId: span.spanContext().spanId,
+}
+
+// Background receives context, creates child span
+trace({ name: TraceName.MyOperation, parentContext: context }, async () => { ... })
+```
+
+Without propagation: Sentry shows two disconnected operations. With propagation, the background span joins the UI's trace under whichever UI span was active at the call, which is not necessarily the span that caused it. `submitRequestToBackground` attaches a context only when a UI span is active at call time. Without one, `createMetaRPCHandler` runs the handler with no `rpc.handler` span and outside the UI's trace.
+
+The serialized context carries no sampled flag, so the background continues every propagated trace as sampled. `tracesSampler` then keeps the background span at rate 1 unless a per-name override or a ceiling applies, so background spans can be stored for a trace whose UI root was sampled out.
+
+The UI and background timestamps do not share a clock. In 17 of 96 measured traces they disagreed by up to 67 minutes, so a duration computed across the boundary is not reliable.
+
+## Sentry Sample Rate
+
+```bash
+grep -n "tracesSampleRate" app/scripts/lib/setupSentry.js
+# The fallback rate. tracesSampler (app/scripts/lib/sentry-traces-sampler.ts) takes precedence over it
+```
+
+## Sentry Traces Explorer Query (Volume Estimation)
+
+```
+Environment: production | Time range: 30 days | Mode: aggregate
+Query: span.op:http.client span.description:*{endpoint}*
+Group by: span.description, transaction
+Sort: -count(span.duration)
+```
+
+## Detect `excludeMetaMetricsId` Misuse
+
+```bash
+grep -rn "excludeMetaMetricsId: true" app/ ui/ shared/ --include="*.ts" --include="*.tsx" --include="*.js"
+# Each hit sends its event under the shared anonymous id. Confirm the event must not carry identity.
+# Event names matching /^send|^confirm/iu are anonymous by default: check new event names too.
+```
+
+## Data Council Contact
+
+- Slack: `#metamask-metametrics`
+- Team: `@consensys/data-council`