Add agent-readiness discovery endpoints for AI agents by edmundmiller · Pull Request #478 · nextflow-io/website

edmundmiller · 2026-06-16T14:39:40Z

Summary

Implements the agent-readiness checks reported by isitagentready.com for nextflow.io. The site is a static Astro app on S3 + CloudFront; this PR adds the discovery artifacts, headers, and infrastructure hooks needed for AI agent discovery.

Changes

Discoverability

public/_headers — Link response headers on / pointing to api-catalog, service-doc, and service-desc (Netlify previews)
infra/cloudfront/ — CloudFront Function for markdown negotiation and response headers policy template for production
scripts/set-s3-agent-metadata.sh — Sets correct Content-Type and Link metadata on S3 objects during publish

Content accessibility

public/index.md — Markdown homepage
netlify/edge-functions/markdown-negotiate.js — Serves markdown when Accept: text/markdown is present (with x-markdown-tokens)

Bot access control

public/robots.txt — Added Content-Signal: ai-train=no, search=yes, ai-input=yes

Protocol discovery

/.well-known/api-catalog — RFC 9727 linkset (application/linkset+json)
/openapi.json — Minimal OpenAPI 3.1 spec for public endpoints
/.well-known/oauth-authorization-server — RFC 8414 metadata with agent_auth block (honest stub — no protected APIs today)
/.well-known/oauth-protected-resource — RFC 9728 metadata
/.well-known/jwks.json — Empty JWKS (reserved)
/auth.md — Agent registration discovery document
/.well-known/mcp/server-card.json — MCP server card (SEP-1649)
/.well-known/agent-skills/index.json — Agent skills index synced from nextflow-io/agent-skills at build time (install-nextflow, create-workflow, run-module, launch-workflow) with SHA-256 digests
src/components/WebMcpBridge — WebMCP tools via navigator.modelContext

DNS-AID

infra/dns-aid/nextflow.io.zone — SVCB records for _index._agents and _a2a._agents (requires DNS operator to apply + DNSSEC)

Agent skills

scripts/generate-agent-skills-index.mjs shallow-clones nextflow-io/agent-skills into .cache/agent-skills, copies each skills/*/SKILL.md to public/.well-known/agent-skills/, and regenerates index.json with digests. Runs automatically via prebuild.

Deployment notes

After merge, production needs:

make publish — runs updated metadata script automatically
CloudFront — attach markdown-negotiation.js viewer-request function and response headers policy per infra/cloudfront/README.md
DNS — publish DNS-AID records from infra/dns-aid/nextflow.io.zone and enable DNSSEC

Netlify PR previews get Link headers and markdown negotiation automatically via _headers and the edge function.

Testing

npx astro check && npx astro build — passes
node scripts/generate-agent-skills-index.mjs — syncs 4 skills from nextflow-io/agent-skills

Publish RFC 8288 Link headers, RFC 9727 API catalog, OAuth discovery metadata, MCP server card, agent skills index, auth.md, Content Signals in robots.txt, markdown negotiation, WebMCP tools, and DNS-AID zone templates for nextflow.io agent discovery. Co-authored-by: Edmund Miller <edmundmiller@users.noreply.github.com>

netlify · 2026-06-16T14:39:46Z

✅ Deploy Preview for nextflow-staging ready!

Name	Link
🔨 Latest commit	`3dd2b50`
🔍 Latest deploy log	https://app.netlify.com/projects/nextflow-staging/deploys/6a3c21ddf1368900080c47df
😎 Deploy Preview	https://deploy-preview-478--nextflow-staging.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.
Lighthouse	1 paths audited Performance: 61 (🟢 up 4 from production) Accessibility: 74 (no change from production) Best Practices: 83 (🟢 up 8 from production) SEO: 79 (🔴 down 8 from production) PWA: - View the detailed breakdown and full score reports

To edit notification comments on pull requests, go to your Netlify project configuration.

Replace the local nextflow-docs stub with the four official skills from https://github.com/nextflow-io/agent-skills, synced at build time with SHA-256 digests for the discovery index. Co-authored-by: Edmund Miller <edmundmiller@users.noreply.github.com>

Co-authored-by: Edmund Miller <edmundmiller@users.noreply.github.com>

ewels

Contains a mixture of deployment methods, which is very confusing.

Netlify is only used for deployment previews I think, so does not need to be agent-ready. It's only used for internal review and should not be accessed by an agent.

There are also many many broken URLs referenced in here, a lot of functionality that doesn't exist. Needs self-review before going any further.

- api-catalog: drop 'status' rel pointing at feed.xml (RSS != status) - mcp/server-card.json: replace fake streamable-http /mcp endpoint with webmcp transport pointing at the homepage (WebMcpBridge uses navigator.modelContext, not an HTTP server) - oauth-authorization-server: remove authorization_endpoint, token_endpoint, jwks_uri and related fields that all 404 - Delete jwks.json empty stub (no keys, no real OAuth flow) - _headers: remove jwks.json entry

Remove the AI-generated OpenAPI spec and its references from api-catalog (service-desc), _headers Link rel, auth.md, and index.md. nextflow.io is a marketing site, not an API; the spec was unnecessary overhead that produced a dead URL.

ewels · 2026-06-24T06:14:31Z

URL validation review

I checked every URL referenced in the changed files (api-catalog, openapi.json, _headers, both oauth-* files, mcp/server-card.json, agent-skills/index.json, auth.md, index.md, robots.txt, the DNS zone, and the sync script). Results are against current head 8ac2e45.

✅ Valid / correct

All committed discovery endpoints (/openapi.json, /index.md, /auth.md, /.well-known/api-catalog, /.well-known/oauth-authorization-server, /.well-known/oauth-protected-resource, /.well-known/mcp/server-card.json, /.well-known/agent-skills/index.json) are backed by files in the PR and will resolve once deployed.
docs.seqera.io/nextflow/, github.com/nextflow-io/agent-skills (exists, contains all 4 referenced skills), github.com/workos/auth.md — all real.
schemas.agentskills.io/discovery/0.2.0/schema.json — does not resolve, but this is correct: it's the canonical schema identifier from the agent-skills 0.2.0 discovery spec, which explicitly states the URI need not be resolvable. Not a broken link.
rfc-editor.org/rfc/rfc8414 (OAuth 2.0 Authorization Server Metadata) and rfc9728 (OAuth 2.0 Protected Resource Metadata) — correct references for the respective OAuth files.
index.md quick-links (community.seqera.io/tag/nextflow, training.nextflow.io/latest/, marketplace.visualstudio.com/...nextflow.nextflow, seqera.io/ask-ai/, nf-co.re/, seqera.io/pipelines/, github.com/nextflow-io/nextflow/issues) — all live.

⚠️ Issues to address

/agent/register and /agent/claim 404. public/.well-known/oauth-authorization-server still advertises register_uri and claim_uri pointing at these, and auth.md references them, but neither has any backing file or implementation. Advertising non-resolvable endpoints in a live discovery document is the same class of issue raised earlier in review — recommend removing them until they're real.
Stale http:// links in index.md: http://docker.io and http://singularity.lbl.gov/. Both resolve but are plain http and out of date — suggest https://www.docker.com/ and https://apptainer.org/.
SKILL.md build dependency. index.json is committed with hardcoded sha256 digests, but the four */SKILL.md files are only produced by scripts/generate-agent-skills-index.mjs at build time. Please confirm that script runs in the deploy pipeline; otherwise the index points at 404s and/or digest mismatches.

Everything else checks out.

- oauth-authorization-server: remove register_uri and claim_uri — no backing implementation, same class of hallucinated-endpoint issue - auth.md: remove 'Agent registration' section referencing /agent/register and /agent/claim; simplify to reflect anonymous read-only reality - index.md: update http://docker.io → https://www.docker.com/ and http://singularity.lbl.gov/ → https://apptainer.org/ (Singularity was renamed to Apptainer)

edmundmiller · 2026-06-24T06:15:55Z

Thanks for the thorough URL audit! Addressed in c5410c9:

/agent/register + /agent/claim — removed register_uri and claim_uri from oauth-authorization-server; simplified auth.md to remove the whole "Agent registration" section. No stubs advertised anymore.
Stale http:// links — index.md updated: http://docker.io → https://www.docker.com/ and http://singularity.lbl.gov/ → https://apptainer.org/.
SKILL.md build dependency — confirmed: package.json has "prebuild": "node scripts/generate-agent-skills-index.mjs", so the script runs automatically before every astro build. The committed index.json is just a snapshot; the build always regenerates it (and the SKILL.md files) from the live nextflow-io/agent-skills repo before deploy.

Generated by Claude Code

ewels

I still think we should strip all the auth / oauth stuff. It's not designed for a website like this and although most of it is technically correct I guess, I think having it there at all is just going to be a red herring and do more harm than good.

Please remove it entirely..

…gests - Remove oauth-authorization-server, oauth-protected-resource, auth.md entirely — auth machinery is a red herring on a public marketing site - Revert netlify.toml edge_functions entries — Netlify config is internal - Remove sha256 digests from agent-skills/index.json and the generator script (generate-agent-skills-index.mjs) - Clean up _headers and index.md of now-deleted file references

ewels · 2026-06-24T19:15:57Z

Can remove, as specifically for netlify

ewels · 2026-06-24T19:16:08Z

+
+The function rewrites `/` and `/index.html` to `/index.md` when `Accept: text/markdown` is present.
+
+## Netlify previews


Don't want to touch netlify

ewels · 2026-06-24T19:16:29Z

+    "Items": [
+      {
+        "Header": "Link",
+        "Value": "</.well-known/api-catalog>; rel=\"api-catalog\", </docs.seqera.io/nextflow/>; rel=\"service-doc\", </openapi.json>; rel=\"service-desc\"",


still referencing openai even though it's gone

ewels · 2026-06-24T19:17:04Z

Don't want any netlify stuff so can remove this

ewels · 2026-06-24T19:17:43Z

 Disallow: /logs/
 Disallow: /tests/
 Disallow: /releases/
+Content-Signal: ai-train=no, search=yes, ai-input=yes


We want to be in training sets..

Suggested change

Content-Signal: ai-train=no, search=yes, ai-input=yes

Content-Signal: ai-train=yes, search=yes, ai-input=yes

ewels · 2026-06-24T19:20:13Z

+      training: "https://training.nextflow.io/latest/",
+      community: "https://community.seqera.io/tag/nextflow",
+      examples: "https://nextflow.io/examples.html",
+      api_catalog: "https://nextflow.io/.well-known/api-catalog",


This just links to the docs again, I think it can be dropped

ewels · 2026-06-24T19:20:27Z

+      { title: "Basic pipeline", url: "https://nextflow.io/basic-pipeline.html" },
+      { title: "Mixing scripting languages", url: "https://nextflow.io/mixing-scripting-languages.html" },
+      { title: "BLAST pipeline", url: "https://nextflow.io/blast-pipeline.html" },
+      { title: "RNA-seq pipeline", url: "https://nextflow.io/rna-seq-pipeline.html" },
+      { title: "Machine learning pipeline", url: "https://nextflow.io/machine-learning-pipeline.html" },


I don't love hardcoding these, could go stale easily

ewels · 2026-06-24T19:21:24Z

How is this used? Would end users / agents have to install an MCP to use the Nextflow website? I can't ever see that happening if so tbh. If it's anything other than pure raw "better discovery" then we should cull this as unnecessary complexity.

ewels · 2026-06-24T19:21:47Z


+    <link rel="api-catalog" href="/.well-known/api-catalog" />
+    <link rel="service-doc" href="https://docs.seqera.io/nextflow/" />
+    <link rel="service-desc" href="/openapi.json" />


openapi again

ewels · 2026-06-24T19:22:34Z

    "docs": "bash build_docs.sh",
+    "prebuild": "node scripts/generate-agent-skills-index.mjs",
    "build": "astro check && astro build && bash build_docs.sh",
+    "generate:agent-skills-index": "node scripts/generate-agent-skills-index.mjs",


It'd be good if this ran on CI somewhere to auto-update (or at least check in CI so that we force it to stay up to date).

Either that or add it onto the build command that runs during deployment (in which case, probably don't need the file itself under source control?)

cursoragent and others added 2 commits June 16, 2026 14:46

Apply prettier formatting for pre-commit CI

c050bab

Co-authored-by: Edmund Miller <edmundmiller@users.noreply.github.com>

edmundmiller marked this pull request as ready for review June 16, 2026 15:21

ewels requested changes Jun 16, 2026

View reviewed changes

edmundmiller requested a review from ewels June 17, 2026 13:36

ewels requested changes Jun 24, 2026

View reviewed changes

Comment thread netlify.toml

Comment thread public/.well-known/agent-skills/index.json Outdated

edmundmiller requested a review from ewels June 24, 2026 18:28

ewels requested changes Jun 24, 2026

View reviewed changes


		The function rewrites `/` and `/index.html` to `/index.md` when `Accept: text/markdown` is present.

		## Netlify previews

	Content-Signal: ai-train=no, search=yes, ai-input=yes
	Content-Signal: ai-train=yes, search=yes, ai-input=yes

Uh oh!

Conversation

edmundmiller commented Jun 16, 2026 • edited by cursor Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Discoverability

Content accessibility

Bot access control

Protocol discovery

DNS-AID

Agent skills

Deployment notes

Testing

Uh oh!

netlify Bot commented Jun 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for nextflow-staging ready!

Uh oh!

ewels left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ewels commented Jun 24, 2026

URL validation review

✅ Valid / correct

⚠️ Issues to address

Uh oh!

edmundmiller commented Jun 24, 2026

Uh oh!

ewels left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

edmundmiller commented Jun 16, 2026 •

edited by cursor Bot

Loading

netlify Bot commented Jun 16, 2026 •

edited

Loading