feat: Mint metrics tokens for Datum - #240
Merged
Merged
Conversation
Datum's metrics gateway is vmauth, which verifies a Central-signed JWT and reads `vm_access.metrics_extra_labels` off it. VictoriaMetrics applies those labels over whatever the producer sent, so a pilot cannot write as another resource even if it claims one in the request body. Central had no token for this. Its existing mints are bench logins, site logins and enrolment, all short-lived and none carrying a resource id. `scope` is what makes this safe to add: every downward token is signed with the same key, so without one the gateway would accept a bench SID as a licence to write metrics. It matches on `scope: datum` and rejects the rest. The resource is bound after Atlas provisions, so minting before then is refused rather than handing out a token whose samples could not be attributed to anything. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
They were the longest in both files -- nine lines where the neighbours run to two or three. The facts that are not obvious from the code stay: what scope guards against, and why minting is refused before Atlas binds the Asset.
No other mint has a test file of its own -- bench login and bootstrap are covered inside test_enroll, site login inside test_atlas_sync. A dedicated 120-line file for this one was out of step, and larger than any test file in the repo. Two tests remain, beside the endpoint's siblings: the token carries the scope and resource label, and minting is refused before Atlas binds the Asset. The signature is not re-tested; test_sso already covers that every minted token verifies against the JWKS.
A day meant every pilot re-fetching daily, and a pilot that cannot reach Central for a day stops reporting metrics -- exactly when you want them. The cost is that a leaked token is good for a year, and there is no revocation list: pulling one back means rotating Central's signing key, which invalidates every token it ever signed. Worth revisiting if pilots grow a way to refresh unattended.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Datum stores metrics behind vmauth, which checks a Central-signed JWT before
anything reaches the store. It reads
vm_access.metrics_extra_labelsoff thetoken and hands them to VictoriaMetrics, which stamps them over whatever the
producer sent — so a pilot can't write as another resource even if it claims
one in the request body.
Central had no token for that. The existing mints are bench logins, site logins
and enrolment: all short-lived, none carrying a resource id.
This adds
mint_metrics_token, and ametrics_tokenendpoint the pilot callswith its usual
X-Pilot-Token:{ "aud": "<pilot_credential_id>", "scope": "datum", "vm_access": {"metrics_extra_labels": ["resource_id=vm-abc123"]} }scopeis the part that matters. Every downward token is signed with the samekey, so without it the gateway would treat a bench SID as permission to write
metrics. vmauth matches on
scope: datumand rejects everything else.The resource id comes from the credential's linked Asset. That's only bound
after Atlas provisions, so a pilot asking too early is told to wait rather than
handed a token whose samples couldn't be attributed to anything.
It's a separate endpoint rather than part of
configbecause it expires — thepilot re-fetches on a TTL, or when the gateway answers 401.
Verified the claim shape against a real vmauth: the token is accepted, and a
body claiming a different
resource_idis overwritten with the one from thetoken.