refactor: metric labels

[alexghr]

Related to A-2039

[greptile-apps]

Confidence Score: 4/5

The PR appears safe to merge, with a non-blocking observability gap that should be corrected so RPC metrics receive the same environment identity as other deployments.

Findings

1.  Missing RPC Environment Label ▶

Summary

• Adds cloud, cluster, deployment, network, and component identity to metrics collectors.
• Propagates shared OpenTelemetry attributes into Aztec node environments.
• Extends the resource setup script with an optional network attribute.
• Leaves RPC telemetry without the otherwise-standard deployment environment label.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  E[RPC environment configuration] --> A[OTEL_RESOURCE_ATTRIBUTES map]
  A --> N[RPC node environment]
  A --> K[Kong metrics collector]
  N --> S[setup-otel-resource.sh]
  S --> O[OTLP resource attributes]
  K --> O
  O --> P[Prometheus resource-label promotion]
  M[Missing deployment.environment.name] -. affects .-> A

Loading

_{Reviews (1) · Last reviewed commit: "refactor: metric labels"}

[greptile-apps]

Missing RPC Environment Label

The new resource attribute map omits deployment.environment.name, even though NAMESPACE is available and every other deployment updated by this refactor supplies that attribute. The same omission appears in the testnet map at spartan/terraform/deploy-rpc/environments/testnet/main.tf:112-118. As a result, both RPC node and Kong metrics lack the environment label that the production metrics processor promotes for filtering and grouping. Add the namespace-derived attribute to both maps.

[spalladino]

Posted by Claude Opus on Santiago's behalf — he asked me to review the diff, these are my findings, not his. Treat them as a bot review: worth checking, but verify before acting.

Edited after reading aztec-labs-eng/aztec-observability#2, which changes the answers below. Item 1 is resolved there; item 2's fix goes the opposite way from what I first wrote; item 3 is settled by that PR's labels.md.

terraform fmt -check -recursive is clean and terraform validate passes in all four roots. I also checked the tricky expressions in terraform console — replace(..., "/-[a-z]$/", "") correctly turns us-west1-a into us-west1 and leaves us-west1 alone, element(reverse(split("_", ...)), 0) pulls the cluster out of a gke_ context, and the compact/join merge in the RPC module behaves. Four things I think are worth a look:

1. Five of the seven new attributes never become metric labels — resolved in aztec-observability#2, with one caveat

My original point: the central collector's transform/promote_resource_attributes only promoted 9 attributes, and the prometheus exporter has resource_to_telemetry_conversion.enabled: false, so project, cloud.provider, cloud.platform, cloud.account.id, cloud.region and k8s.cluster.name would have been dropped.

aztec-labs-eng/aztec-observability#2 extends the promote list to cover all seven attributes this PR emits, and keeps region as an explicitly deprecated alias for cloud.region. That closes it.

The remaining caveat is deploy ownership. aztec-observability is a fresh extraction ("initially extracted from aztec-labs-eng/aztec-node/spartan") and has no .github/workflows yet, while this repo's .github/workflows/metrics-deploy.yml still deploys spartan/metrics/values/prod.yaml from this repo — whose promote list is unchanged. So until either that change also lands here or the deploy pipeline moves to the new repo, the running collector keeps the old list and these labels are still dropped in the interim. Worth confirming which copy is authoritative right now. (aztec-observability#2 is also stacked on stack/feat-deploy-yace rather than main.)

2. network now means two different things in one deployment

spartan/aztec-node/scripts/setup-otel-resource.sh:27-29 sets network from $NETWORK, which comes from global.aztecNetwork = var.NETWORK. The kong collector alongside it keeps network = var.RELEASE_PREFIX (deploy-aztec-infra/main.tf:911), and RELEASE_PREFIX = NAMESPACE. Those diverge:

• staging-internal.env and staging-public.env both set NETWORK=staging with distinct namespaces. Node metrics from the two deployments collapse into one network="staging" series while their kong metrics stay separate, so node and gateway metrics can no longer be joined on network.
• .github/workflows/deploy-network.yml exposes namespace as an override input, but NETWORK is fixed in the env file. An ad-hoc deployment therefore writes into the real network's network="staging" / network="testnet" series.

Correcting my original suggestion: I first proposed deriving network from RELEASE_PREFIX/NAMESPACE. aztec-observability#2's labels.md defines network as "Blockchain network such as mainnet or testnet" and puts namespace identity in k8s_namespace_name ("Namespace is not project ownership or environment"). So the node's network=$NETWORK is the correct side and kong's network = var.RELEASE_PREFIX is what violates the contract — the fix belongs on the kong attribute, not on the nodes.

That leaves the ad-hoc-namespace collision unaddressed by either PR: two deployments of the same chain are distinguishable only by k8s_namespace_name, which is fine as long as queries carry it.

3. deployment.environment.name disagrees for the same network — and labels.md says the RPC side is right

deploy-aztec-infra/main.tf:115 computes var.NETWORK == "mainnet" ? "production" : "staging", so testnet nodes come out as staging. But deploy-rpc/environments/testnet/main.tf:111 hardcodes "production", as does deploy-ethereum. Testnet's nodes and testnet's RPC land in different environments.

aztec-observability#2's labels.md resolves which is correct: "A production service can serve a testnet; blockchain network is a separate label." Testnet nodes should therefore be deployment.environment.name=production with network=testnet, which makes the ternary in deploy-aztec-infra the side to change. Since this attribute is promoted to a label, the split shows up in queries.

4. k8s.cluster.name is string-sliced from the kubeconfig context

K8S_CLUSTER_CONTEXT=$(kubectl config current-context) (deploy_network.sh:307), so the gke_<proj>_<zone>_<cluster> shape only holds when credentials came straight from gcloud container clusters get-credentials. deploy_chaos_mesh.sh:36 only asserts that the context contains $CLUSTER, so a renamed or aliased context silently yields a wrong cluster name. CLUSTER (aztec-gke-public / aztec-gke-private) is authoritative and already in the deploy scripts, just not plumbed into terraform.

Minor, new from reading aztec-observability#2

cloud.availability_zone is now on the promote list and labels.md explicitly asks for zones like us-west1-a to go there. Every spartan/environments/*.env sets GCP_REGION=us-west1-a, and replace(var.GCP_REGION, "/-[a-z]$/", "") throws that zone away rather than emitting it as cloud.availability_zone alongside the stripped region. Cheap to add while the block is being written.

---

Things I checked that look right: the is_kind gate keeps the fake GCP attributes off local kind deployments, the script appends to rather than overwrites a pre-existing OTEL_RESOURCE_ATTRIBUTES so the terraform-injected values survive, and envDetectorSync in yarn-project/telemetry-client/src/otel_resource.ts does read the env var.

[spalladino]

LGTM, but Claude found a few issues that I'm asking it to post. Namely:

• We have resource_to_telemetry_conversion.enabled set to false, so new metrics don't make it. I haven't reviewed the other observability PRs yet, so maybe this is handled in one of the other PRs for the other repos? Edit: yep, https://github.com/aztec-labs-eng/aztec-observability/pull/2/ fixes it.
• Network, environment name, and cluster variables can be set more cleanly.