Investigate public-key and attenuation-capable token formats (Biscuit, macaroons) as an evolution path

[dgenio]

Summary

Run a structured investigation into whether (and how) the kernel's HMAC capability
tokens should gain an optional public-key-verifiable and/or offline-attenuable
format — evaluating Biscuit, macaroon-style HMAC chaining, and PASETO against the
kernel's invariants — producing a design recommendation, not an implementation.

Why this matters

Two roadmap directions strain the current shared-secret HMAC design: delegated/
attenuated grants (#129) want offline attenuation (macaroons/Biscuit's core trick),
and federation across trust boundaries (plus the manifest-signing model documented
in ISSUE 49) wants verification without sharing the signing secret. Choosing a
token evolution path early — even if the answer is "stay HMAC-only, add attenuation
via re-issuance" — prevents an expensive migration later and gives #129 and #103 a
shared foundation. This is framed as an investigation because the right answer
genuinely is not obvious: HMAC's simplicity is a documented strength.

Current evidence

• tokens.py: HMAC-SHA256, single shared secret, no key id (until ISSUE 11), verify requires the signing secret — delegation and cross-party verification are structurally out of reach.
• federation_discovery.py: pairwise shared-secret HMAC manifests — same constraint at the federation layer.
• Open issues [Feature] Delegated, attenuated grants for multi-agent delegation chains #129 (delegated, attenuated grants) and security: production-hardening roadmap + ship the first piece (token signing/encryption, pluggable secrets/auth) #103 (hardening roadmap: "token signing/encryption") both touch token format without settling it.

External context

• Biscuit: public-key verification + offline attenuation via Datalog (https://www.biscuitsec.org/, https://doc.biscuitsec.org/getting-started/introduction).
• Macaroons: HMAC-chained caveats, attenuation without asymmetric crypto (https://github.com/rescrv/libmacaroons).
• Both have mature design literature on the exact tradeoffs (verifier complexity, dependency weight, token size).

Proposed implementation

1. Define evaluation criteria from the kernel's invariants: I-06 binding semantics,
   minimal-dependency policy (Biscuit needs a third-party lib; macaroon-style
   chaining can be stdlib hmac), determinism, token size, verification cost,
   revocation interaction, 0.x migration cost.
2. Prototype the two leading options behind the existing TokenProvider protocol
   (the seam already exists) in throwaway branches — measure, don't speculate.
3. Deliverable: a design doc (ADR per ISSUE 53) recommending one of: (a) status
   quo + re-issuance-based attenuation, (b) macaroon-style chaining in-tree, (c)
   Biscuit behind an extra, with a migration sketch for the chosen path.
4. Feed conclusions into [Feature] Delegated, attenuated grants for multi-agent delegation chains #129 and security: production-hardening roadmap + ship the first piece (token signing/encryption, pluggable secrets/auth) #103 rather than implementing here.

AI-agent execution notes

• Inspect first: tokens.py (TokenProvider protocol, HMACTokenProvider), docs/agent-context/invariants.md (I-06), issues [Feature] Delegated, attenuated grants for multi-agent delegation chains #129/security: production-hardening roadmap + ship the first piece (token signing/encryption, pluggable secrets/auth) #103 text.
• The TokenProvider protocol boundary is the load-bearing fact: any recommendation must slot behind it without kernel-wide changes.
• Do not add dependencies or change production code in this issue.
• Be explicit about what each format costs in auditability (Datalog policies are powerful but reduce explain()-style transparency).

Acceptance criteria

• A written evaluation against the defined criteria, with prototype evidence for at least two options.
• A clear recommendation with migration sketch and explicit rejection rationale for alternatives.
• [Feature] Delegated, attenuated grants for multi-agent delegation chains #129 and security: production-hardening roadmap + ship the first piece (token signing/encryption, pluggable secrets/auth) #103 updated (comments) with the conclusion.

Test plan

Prototype benchmarks (token size, verify latency) recorded in the doc; no production
test changes. Run make ci (unchanged).

Documentation plan

ADR/design doc under docs/ (or docs/adr/ per ISSUE 53); CHANGELOG not required
(no code change).

Migration and compatibility notes

Investigation only; any eventual format change would ship behind the TokenProvider
protocol with a dual-verification window.

Risks and tradeoffs

Risk of analysis paralysis — timebox it. Recommending a heavier format conflicts
with the minimal-deps invariant; the criteria framework makes that tension explicit
rather than implicit.

Suggested labels

investigation, security, architecture