A proposal for a web content architecture where markdown is the source of truth, agents are first-class citizens, and access policy is expressed through price.
Status: Draft (v0.2.0) · Seeking community feedback
Authors: Gary Walker / BitCryptic™ · Graham Hall / Slepner
AI agents are now among the most frequent consumers of web content, yet the web serves them HTML — a format built for human eyes. Most agents compensate by stripping navigation, ads, scripts and layout markup client-side. That works, until it doesn't: the result depends entirely on how well a given page yields to a heuristic extractor, and the page has to be downloaded in full either way. Meanwhile the person who wrote the content receives nothing for it. MDF proposes a simple, open architecture addressing both halves: markdown as the canonical source, HTTP content negotiation for delivery, and an optional payment layer that doubles as access policy — giving creators a way to be paid for agent consumption of their work rather than watching it scraped and propagated for free.
- Markdown-first authoring — markdown is the source of truth; HTML is rendered from it for browsers, not the other way around
- Native agent serving —
Accept: text/markdownreturns clean markdown at the same URL, no conversion middleware required - Structured discovery —
/mdf.jsonadvertises capabilities, pricing, and content signals in a machine-readable format that agents can query before fetching - Price as access policy — a single price field unifies open access, micropayment, and private/authenticated access into one continuous spectrum, using the x402 (EVM/stablecoin) and L402 (Bitcoin/Lightning) payment standards
Efficiency, where it applies. Serving markdown means the HTML never crosses the wire. Whether it also saves context-window tokens depends on the page: boilerplate-heavy commercial and editorial pages retain a large saving even against a good client-side extractor, while clean documentation may save almost nothing. Where MDF is unconditionally better is determinism — the publisher declares what the content is, rather than an extractor guessing, so JS-rendered, paginated and table-dense pages stop failing silently.
Compensation, always. This half doesn't depend on markdown being smaller. Today a creator's options are to be scraped for free or to retreat behind a login and disappear from the open web. MDF offers a third: stay open and discoverable, and be paid at the point of machine consumption. At higher price tiers the same payment mechanism issues an access credential, giving a small publisher the practical equivalent of a subscription wall without operating one — and giving agents a lawful route to content that was previously unavailable at any price.
| Price | What it means |
|---|---|
$0.00 |
Open — serve immediately, no payment required |
$0.0001 |
Micropayment — small per-fetch fee offsets creator costs; on boilerplate-heavy pages, typically cheaper for AI operators than downloading and extracting the HTML |
$1.00+ |
Premium — meaningful payment for gated content |
$100.00+ |
Private — payment triggers an auth token issuance rather than immediate delivery |
Servers performing on-server markdown conversion may also report the size reduction between the original HTML and the served markdown directly in the response — giving agents a concrete efficiency signal alongside the price itself, rather than requiring them to price a request in the dark. See the concept document's Response Value Signalling section for details.
The concept document covers the full architecture, existing partial solutions, open questions, and the reference implementation plan.
A concept document for an MDF reference client: an MCP gateway that lets any MCP-capable agent runtime discover, negotiate, evaluate and pay for MDF content. Concept stage, not built.
- Concept document published
-
mdf.schema.json— JSON Schema for/mdf.json - Reference implementation —
bitcryptic/mdf-serveron Docker Hub (bitcryptic-gw/mdf-reference-server) - Demo site — https://mdf-demo.bitcryptic.com — live end-to-end demonstration of all three payment tiers
- x402 payment verification stub (EVM/stablecoin rail)
- L402 payment verification — live Lightning invoice settlement (Bitcoin/Lightning rail)
- Atom feed with
mdf:change_typenamespace — live at https://mdf-demo.bitcryptic.com/feed.xml - Validator CLI —
bitcryptic-gw/mdf-validator - On-chain x402 payment verification (stub → production)
- Lightning invoice verification for L402 — production, real transactions settled
-
mdf-402.schema.json— JSON Schema for the 402 response body - Reference client — MDF MCP gateway (concept published, not built)
A reference implementation is publicly deployed at https://mdf-demo.bitcryptic.com
Try it now:
# Discover the site's MDF capabilities
curl https://mdf-demo.bitcryptic.com/mdf.json
# Fetch the agent index
curl https://mdf-demo.bitcryptic.com/llms.txt
# Request markdown directly (agent-style)
curl -H "Accept: text/markdown" https://mdf-demo.bitcryptic.com/
# Free content — no payment required
curl -H "Accept: text/markdown" https://mdf-demo.bitcryptic.com/docs/getting-started
# Paid content — returns 402 with payment instructions
curl -H "Accept: text/markdown" https://mdf-demo.bitcryptic.com/premium/deep-dive
# Micropayment tier over Lightning — returns 402 with an L402 invoice
curl -H "Accept: text/markdown" https://mdf-demo.bitcryptic.com/micropayment/intro
# Private content — returns 402 with auth endpoint hint
curl https://mdf-demo.bitcryptic.com/private/internals
# Atom feed with mdf:change_type metadata
curl https://mdf-demo.bitcryptic.com/feed.xmlThe demo site exercises every payment tier and the full auth-via-payment flow. The L402 (Lightning) rail performs real invoice verification — a /micropayment/ fetch settles a genuine Lightning payment. The x402 (EVM) rail is still stubbed, so the /premium/ and /private/ tiers can be explored without an on-chain transaction.
A WordPress plugin that tracks AI agent traffic and Accept: text/markdown requests to your site, with a dashboard showing estimated earnings from MDF-enabled content. Since v0.1.4 it also serves markdown natively: published posts and pages are pre-converted to CommonMark and returned at the same URL to agents that negotiate for them. Wallet connection and paid markdown remain future work.
→ bitcryptic-gw/mdf-analytics-wp
This is an early-stage community proposal. The open questions section of the concept document is a good starting point for discussion.
- Discuss: Open an issue
- Implement: Build an MDF-compatible server or client and link it here
- Challenge: If you think this is wrong, redundant, or misses something — say so
MIT