adagents.json discovery: replace follow_redirects=False with same-registrable-domain following (apex→www); refuse on authoritative_location (adcp#5450)

[bokelley]

Context

AdCP issue adcontextprotocol/adcp#5450 and merged spec PR adcontextprotocol/adcp#5472 define the redirect policy for adagents.json discovery. The reference validator in adcontextprotocol/adcp is now aligned; this SDK needs the same behavior so all AdCP implementations resolve discovery identically.

The bug this fixes: a publisher whose apex domain 301-redirects to www (the default on most managed hosting / CDNs) is silently reported unauthorized by validators that refuse all redirects on the discovery fetch.

Policy — two hops, deliberately different

Fetch	Redirect policy
Initial https://{domain}/.well-known/adagents.json	Follow same-registrable-domain redirects (apex↔www, HTTPS-preserving, ≤3 hops), re-validating SSRF (reject non-HTTPS, reject reserved/private IPs, pin connect) on every hop, with the same-domain comparison anchored on the originally-requested domain at every hop (not the previous hop). Refuse cross-registrable-domain redirects — that is an unscoped delegation signal; the publisher should declare delegation via authoritative_location instead.
authoritative_location dereference (2nd hop)	Refuse all redirects. The named URL is the declared authoritative location; a redirect away from it changes that declaration and MUST be treated as an error.

⚠️ Security-critical: use the PSL private section

Registrable-domain comparison MUST use the Public Suffix List including its PRIVATE section. Without it, shared-hosting tenants collapse to one registrable domain — victim.github.io and attacker.github.io both reduce to github.io — and a cross-tenant redirect would be wrongly followed and trusted as authoritative. The same applies to *.pages.dev, *.web.app, *.herokuapp.com, *.wordpress.com, etc. The reference impl uses tldts.getDomain(host, { allowPrivateDomains: true }).

Conformance

Gate on the shared cross-SDK vectors: static/test-vectors/adagents-discovery-redirects.json. They cover apex↔www, subdomain, co.uk (ICANN multi-label), *.github.io (PSL private section), same→cross two-hop (origin-anchored), HTTPS downgrade, hop cap, and the authoritative-location refuse-all rule. The normative assertion per vector is result (resolved vs refused).

This SDK (adcp-client-python)

Per the #5450 report, the .well-known/adagents.json fetch currently passes follow_redirects=False (in adcp/adagents.py), which is exactly what breaks apex→www publishers.

• .well-known fetch → replace follow_redirects=False with a manual same-registrable-domain redirect loop: follow only same-eTLD+1, HTTPS-preserving, ≤3 hops, re-validating SSRF on each hop, anchored on the originally-requested domain; refuse cross-domain.
• authoritative_location dereference → keep redirects refused.
• PSL with private domains: e.g. tldextract with include_psl_private_domains=True, or publicsuffixlist/publicsuffix2 with private rules — so victim.github.io ≠ attacker.github.io.

Acceptance: the discovery path passes every vector in adagents-discovery-redirects.json.

[bokelley]

Triage

Classification: Bug
Bucket(s): signing, validation
Status: ready-for-human
Milestone: (omit — no target version named)

What the experts said:

• security-reviewer: 4 blockers — (1) redirect loop must be manual/hop-by-hop with a fresh _owned_pinned_client per hop (using follow_redirects=True on one pinned transport reopens the DNS-rebinding window); (2) original-domain anchor must be captured before the loop and held fixed — a rolling-anchor implementation is a SSRF mitigation bypass; (3) _validate_redirect_url lacks an explicit scheme == "https" guard (the authoritative_location path at line 694–699 already does this correctly and must be the model); (4) new redirect-loop code must import same_registrable_domain from adcp.signing.etld (already configured with include_psl_private_domains=True) — a second extractor constructed without that flag is a blocker
• ad-tech-protocol-expert: 2 additional blockers — (1) same original-domain-anchor point confirmed as correct spec reading; (2) IDN/punycode normalization required before eTLD+1 comparison (not covered by the test vectors; urlparse returns raw ACE labels which may not match the Unicode original domain); also: confirmed non-breaking for currently-working publishers; separate MAX_WELL_KNOWN_REDIRECT_HOPS = 3 constant recommended (nit); no feature flag needed

My take: The fix is clearly scoped and the spec (adcp#5472) is normative, but both experts raised blockers on the security-sensitive SSRF-mitigation path. The IDN/punycode gap is an additional angle the issue body does not cover. Flagging for @bokelley: security-sensitive changes always go to human review per triage policy.

Implementation brief (for the assignee):

• In _stream_capped / _fetch_adagents_url, replace follow_redirects=False with a manual per-hop redirect loop capped by a new constant MAX_WELL_KNOWN_REDIRECT_HOPS = 3 (separate from MAX_REDIRECT_DEPTH = 5 which governs the authoritative_location chain)
• Per hop: extract Location header → assert scheme == "https" (explicit check, not implicit via _check_safe_host) → _check_safe_host → _dns_validate_host → _owned_pinned_client(location_url, timeout) — new pinned client per hop
• Same-domain gate: import same_registrable_domain from adcp.signing.etld; compare urlparse(location_url).hostname against original_hostname captured before the loop (not against the previous hop's URL)
• IDN: verify whether tldextract normalizes punycode ACE labels internally before same_registrable_domain comparison, or add idna.encode(host, uts46=True).decode("ascii") normalization on both sides; failing the IDN test vectors is a blocker
• authoritative_location dereference in _resolve_direct: no change — keep existing follow_redirects=False
• Acceptance gate: pytest against the vectors in static/test-vectors/adagents-discovery-redirects.json from adcontextprotocol/adcp main (apex↔www, PSL private section, HTTPS downgrade, hop cap, original-domain anchor, authoritative-location refuse-all)

---

Triaged by Claude Code. Session: https://claude.ai/code/session_01UhFtrv7V583KN49V5yJ3gb

---

Generated by Claude Code