Skip to content

fix(setup): stop the update skill reporting the dropped lychee domains as drift - #1321

Merged
potiuk merged 1 commit into
apache:mainfrom
potiuk:fix/update-skill-lychee-domains
Sep 21, 2026
Merged

potiuk merged 1 commit into
apache:mainfrom
potiuk:fix/update-skill-lychee-domains

Conversation

@potiuk

@potiuk potiuk commented Sep 21, 2026

Copy link
Copy Markdown
Member

Summary

Type of change

  • Skill change (.claude/skills/<name>/) — see test plan for why no eval fixture
  • Tool / bridge contract (tools/<system>/*.md)
  • Python package (tools/*/ with pyproject.toml) — tools/sandbox-lint (test only)
  • Groovy reference impl
  • Cross-cutting (RFC, AGENTS.md, sandbox, privacy-LLM)
  • Documentation (docs/, README.md, CONTRIBUTING.md)
  • Project template (projects/_template/)
  • CI / dev loop (prek, workflows, validators)

How it works

The rewritten bullet pair says three things the old one got wrong: the default allows only *.crates.io + static.rust-lang.org (the hosts prek needs to bootstrap rustup and cargo install lychee); the eleven link-target hosts were dropped when the hook went offline, so their presence is dead weight to report, never their absence as drift; and enableWeakerNetworkIsolation stays for native-TLS CLI tools (gh / gcloud / terraform), not for lychee.

tools/sandbox-lint/tests/test_doc_domain_consistency.py is the guard. It parses both lists straight out of the skill — the kept hosts from the "default allows …" sentence, the dropped ones from the "once sat beside them (…)" parenthetical — and asserts the first set is present in .claude/settings.json and the second is absent. Rewording the bullet stays free; the two phrases are the anchors. The failure mode it closes is the one this PR fixes: settings move, prose does not, and nothing notices.

Test plan

  • prek run --all-files passes (32 hooks, exit 0) — including skill-token-count, hence the docs/mode-economics.md row in the diff.
  • uv run --project tools/sandbox-lint pytest tools/sandbox-lint/tests/test_doc_domain_consistency.py — 2 passed.
  • Guard verified in both directions: parsing yields exactly {*.crates.io, static.rust-lang.org} kept and the eleven dropped hosts; injecting *.nist.gov into the settings set makes test_dropped_domains_are_not_allowed_again fail.
  • No eval fixture added. The skill's four eval suites cover runtime routing, snapshot drift, tool freshness and the after-report step; none extracts check 4, and the changed text is a report-what-you-see instruction rather than a classification boundary. The pytest above is the regression test — it catches the actual defect class (prose vs settings divergence), which an eval on fixed fixtures could not.

RFC-AI-0004 compliance

  • Sandbox — no new host access; this narrows what the skill tells adopters to allow.
  • Conversational + correctable — the skill stays read-only and proposal-only; only the reported verdict changes.

Linked issues

Refs #501.

Notes for reviewers (optional)

  • The enableWeakerNetworkIsolation bullet keeps the schema's security warning verbatim; only its rationale changed (it was justified by lychee, which is now offline).
  • Anchoring a test on two prose phrases is a trade. The alternative — duplicating the domain lists in the test — would have made the test agree with itself rather than with the doc, which is the failure this PR is about.

🤖 Generated with Claude Code

…s as drift

Check 4 of `setup-isolated-setup-update` told the agent that the
dogfooded `allowedDomains` default carries eleven wildcard link-target
hosts (`*.apache.org`, `*.nist.gov`, `lychee.cli.rs`, …) and that
"without these, lychee fails the PR-blocking prek check locally on first
run".  Both claims stopped being true in apache#501, which moved the lychee
hook to `offline = true` and dropped those hosts from
`.claude/settings.json` precisely because the hook no longer reaches
them.  `docs/setup/secure-agent-setup.md` was updated in that PR; this
skill was not, so a drift report run against current settings reports
eleven correctly-absent domains as missing defaults and walks the
adopter into re-adding dead allowlist entries.

Rewrite the bullet pair to the post-apache#501 reality: the default allows
only the two hosts prek needs to bootstrap rustup and `cargo install`
lychee, the removed hosts are dead weight to *drop* if still present,
and `enableWeakerNetworkIsolation` stays for native-TLS CLI tools
(`gh` / `gcloud` / `terraform`) rather than for lychee.

Add a guard so the prose cannot silently drift from the settings again:
`tools/sandbox-lint/tests/test_doc_domain_consistency.py` parses both
lists out of the skill and asserts the kept hosts are allowed and the
dropped hosts are not.  Re-adding `*.nist.gov` to the settings, or
removing `static.rust-lang.org`, now fails the workspace pytest hook.

Generated-by: Claude Opus 5
@potiuk potiuk added family:setup setup-* skills capability:platform Framework / agent substrate skills (install, verify, doctor, override, status, setup bootstrap) labels Sep 21, 2026
@potiuk
potiuk merged commit 3f2cff9 into apache:main Sep 21, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

capability:platform Framework / agent substrate skills (install, verify, doctor, override, status, setup bootstrap) family:setup setup-* skills

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant