diff --git a/CLAUDE.md b/CLAUDE.md index e3a631485..8e15329aa 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1,78 +1,121 @@ -# CLAUDE.md - Repository Assistant Guide +# CLAUDE.md + +This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. ## Repository Overview -This is the ar-io/docs repository, which manages the documentation for the ar.io Developer Platform. It references various services, SDKs and tools for building on and with ar.io. It also includes documentation for interacting with ArDrive - a flagship consumer dApp that stores data on Arweave. + +Documentation site for the ar.io Developer Platform, covering services, SDKs, and tools for building on ar.io and Arweave. Built with [Fumadocs](https://fumadocs.dev/) on Next.js. Live site: [docs.ar.io](https://docs.ar.io) + +## Key Commands + +```bash +npm run dev # Start dev server (Turbo, standalone mode, no trailing slashes) +npm run build # Production build (static export to out/, trailing slashes) +npm run lint # ESLint (src/ and content/, .ts/.tsx/.mdx) +npx tsc --noEmit # TypeScript type checking +npm run check-links # Validate internal/external links + +# Content generation +npm run generate-api-docs # Generate OpenAPI docs from ar-io-node + turbo services +npm run generate-sdk-docs # Generate SDK reference from external repo READMEs +npm run generate-llm-text # Generate public/llms-full.txt +npm run generate-sdk-llm-texts # Generate per-SDK llm.txt files +npm run generate-all-docs # Run all generation scripts (except API docs) +``` + +Note: The `postinstall` script runs `fumadocs-mdx` to generate the `.source/` directory (content type definitions and index). This must run before the dev server or build will work. If `.source/` is missing, run `npm install` or `npx fumadocs-mdx`. The package manager is **yarn 1.22.22** (specified in `package.json`). ## Architecture -This repository is built with [Fumadocs](https://fumadocs.dev/), a documentation framework. The general architecture is as follows: -### UI Components -All UI should use Fumadocs available UI components by default. This ensures consistency with the framework's design system and maintains a cohesive user experience throughout the documentation. +### Build Modes +- **Development** (`npm run dev`): Standalone Next.js server with Turbo, hot reload, no trailing slashes. Redirects from `redirects.mjs` work here. +- **Production** (`npm run build`): Static export (`output: "export"`) to `out/`, trailing slashes enabled, unoptimized images. ESLint errors are ignored during production builds. Redirects from `redirects.mjs` do **not** work (static export limitation). -All icons should use lucide-react for consistency across the documentation site. +### Routing +- `src/app/[[...slug]]/` - Single catch-all route handles all documentation pages +- `src/app/[[...slug]]/page.tsx` - Renders MDX content, generates static params, handles metadata/OG tags +- `src/app/api/search/route.ts` - Orama full-text search endpoint +- 404s redirect to `/learn` as a fallback -## Key Commands -- **Linting**: `npm run lint` - Run ESLint to check code quality -- **Type checking**: `npm run typecheck` - Run TypeScript type checking -- **Development server**: `npm run dev` - Start the development server -- **Build**: `npm run build` - Build the documentation site - -## Repository Structure -- `/content/` - Main documentation content - - `/content/docs/` - Core documentation pages - - `/content/api/` - API documentation - - `/content/guides/` - User guides and tutorials -- `/public/` - Static assets -- `/src/` - Source code for documentation site -- `/scripts/` - Build and utility scripts - -## Documentation Guidelines -- Documentation is written in MDX format (Markdown with JSX) -- Use semantic headings (H1 for page title, H2 for main sections, etc.) -- Include code examples with proper syntax highlighting -- Add metadata frontmatter to all documentation pages -- Follow existing documentation patterns and structure - -### Documentation Philosophy -These docs intend to model documentation sites of well-established platforms like Stripe, Coinbase, and ChatGPT - where users can natively find things logically, and patterns are consistent throughout. Key principles: -- Logical information architecture -- Consistent navigation patterns -- Clear categorization of content -- Predictable documentation structure -- User-friendly search and discovery - -## Working with OpenAPI -The repository supports OpenAPI documentation import (see recent commit 33af22f2e). When working with API documentation: -- OpenAPI specs should be properly formatted YAML or JSON -- Use the import scripts to convert OpenAPI to documentation format -- Maintain consistency with existing API documentation structure - -## Git Workflow -- Main branch: `main` -- Create feature branches for new work -- Use descriptive commit messages -- Run linting and type checking before committing - -## Community Resources -- **Discord**: https://discord.com/invite/HGG52EtTc2 - Join the ar.io community for updates and discussions - -## Important Notes -- This is a documentation repository - focus on content clarity and accuracy -- Always verify technical details match the actual ar.io implementation -- Maintain consistent terminology throughout documentation -- Test all code examples before including them in documentation - -## Recent Updates -- **Access Data Documentation**: Restructured `/content/build/access/` pages with consistent patterns: - - Updated index.mdx to show three access methods (Find Data, Fetch Data, ArNS) with enhanced cards - - Renamed and reorganized find-data.mdx (GraphQL) and fetch-data.mdx (REST API) for clarity - - Enhanced ArNS documentation with practical SDK examples - - Used native Fumadocs cards with icons for consistency - - Replaced axios with fetch API in all examples - - Added proper call-to-action sections directing users to next steps -- **Upload Data Documentation**: Enhanced `/content/build/upload/index.mdx` with native cards and clear Turbo recommendation -- **Code Standards**: - - Use `fetch` instead of `axios` for HTTP requests - - Use `ARIO.mainnet()` from '@ar.io/sdk' for ArNS operations - - Call `setRecord` on ANT instances, not ARIO instances - - ArNS undernames use underscores (e.g., `api_myapp.arweave.net`) not periods +### Content Source Pipeline +- `source.config.ts` - Fumadocs MDX configuration: registers remark plugins (Mermaid, math) and rehype plugins (KaTeX) +- `src/lib/source.ts` - Content loader using `fumadocs-core`'s `loader()`, with: + - OpenAPI transformer for API reference pages + - Alphabetical sorting for `build/guides` folder (via `ALPHABETICALLY_SORTED_FOLDERS`) + - Icon resolution from `meta.json` - supports Lucide icon names, SVG/PNG paths, and the special `ThemeIcon` for dark/light variants +- `.source/` (generated) - TypeScript types and content index produced by `fumadocs-mdx` +- `src/mdx-components.tsx` - All MDX components registered here, including 18 Lucide icons available as JSX in MDX files +- `src/lib/layout.shared.tsx` - Shared layout options (nav, theme, conditional links) + +### Content Organization +- `content/` - All documentation in MDX format + - `content/learn/` - Conceptual documentation (ArNS, gateways, token, etc.) + - `content/build/` - Developer guides (access data, upload, run gateway) + - `content/sdks/` - SDK documentation (ar-io-sdk, turbo-sdk, ardrive-cli, wayfinder) + - `content/apis/` - API reference (ar-io-node, turbo services) +- `content/meta.json` - Root navigation structure +- Each folder uses `meta.json` to define page order and navigation +- Parenthesized folders like `(introduction)` are route groups (removed from URL) + +### Path Aliases +- `@/*` → `./src/*` +- `@/.source` → `./.source/index.ts` (Fumadocs generated content) + +### Key Components +- **Ask Arie** (`src/components/ask-arie/`) - AI chat widget with session persistence (sessionStorage), thread management, health checks against 3 backend APIs, and citation rendering +- **Page Actions** (`src/components/page-actions.tsx`) - LLM copy button (fetches raw markdown from GitHub) and "Open in AI" dropdown (ChatGPT, Claude, etc.) +- **Search** (`src/components/search.tsx`) - Orama full-text search on a static index built at build time + +### Styling +- Tailwind CSS v4 with CSS-driven configuration (no `tailwind.config.*` file) +- Theme customization in `src/app/global.css` via `@theme` directive +- Brand colors: ar.io purple `#5427C8`, accent `#DFD6F7` + +## Code Standards + +### MDX Content +- Use Fumadocs UI components: ``, ``, ``, ``, ``, `` +- Icons from lucide-react only (imported in mdx-components.tsx) +- Frontmatter required: `title`, `description` +- Optional frontmatter: `image`, `icon`, `keywords`, `author`, `full` (full-width layout) +- Custom components available: ``, ``, ``, ``, ``, `` +- Math/LaTeX: Use `$inline$` and `$$block$$` syntax (remark-math + rehype-katex) +- ESLint is relaxed in `.mdx` files: unused vars, unescaped entities, and `` elements are allowed + +### Code Examples +- Use `fetch` instead of `axios` for HTTP requests +- Use `ARIO.mainnet()` from '@ar.io/sdk' for ArNS operations +- Call `setRecord` on ANT instances, not ARIO instances +- ArNS undernames use underscores: `api_myapp.gateway-url.tld` (not periods) + +### Navigation +- `meta.json` files control page ordering in each directory +- Format: `{ "pages": ["page-slug", "folder-name", "..."] }` +- Use `"..."` for auto-discovery of remaining pages +- Use `"!folder-name"` to exclude a folder from navigation +- Optional fields: `"icon"` (Lucide icon name), `"defaultOpen"` (expand on load) +- `build/guides` is automatically sorted alphabetically (configured in `src/lib/source.ts`) + +### ESLint +- Extends: `next/core-web-vitals`, `next/typescript`, `plugin:mdx/recommended` +- Custom plugin: `validate-jsx-nesting` (enforces proper JSX nesting in components) +- MDX files have relaxed rules (see MDX Content section above) + +## OpenAPI Integration + +The site pulls OpenAPI specs from external sources via `scripts/generate-api-docs.ts`: +- ar-io-node: `https://raw.githubusercontent.com/ar-io/ar-io-node/refs/heads/main/docs/openapi.yaml` +- turbo upload-service and payment-service + +Runtime rendering uses `src/lib/openapi.ts` which fetches the ar-io-node spec directly. Generated API docs go to `content/apis/`. Use `` component in MDX for rendering. + +## LLM Text Generation + +Scripts generate AI-friendly text files from docs: +- `public/llms-full.txt` - Complete site content +- `content/sdks/*/llm.txt` - Per-SDK content (also copied to `public/sdks/`) + +## Deployment + +- **Production**: Deploys to Arweave via GitHub Actions (`.github/workflows/deploy-to-arweave.yaml`) on pushes to main. Uses permaweb-deploy. Supports manual dispatch with custom ArNS undername. +- **PR Previews**: `.github/workflows/pr-preview.yaml` deploys previews for PRs that change `content/` or `src/`. Only runs for PRs from the main repo (not forks). diff --git a/content/apis/ar-io-node/arns.mdx b/content/apis/ar-io-node/arns.mdx index 8a4e142d0..f9c0daa4b 100644 --- a/content/apis/ar-io-node/arns.mdx +++ b/content/apis/ar-io-node/arns.mdx @@ -13,7 +13,7 @@ _openapi: {/* This file was generated by Fumadocs. Do not edit this file directly. Any changes should be made by running the generation command again. */} -Get data from the ar.io Gateway Arweave Name System +Get data from the ar.io Gateway Ar.io Name System \ No newline at end of file diff --git a/content/build/access/arns.mdx b/content/build/access/arns.mdx index e1fb79634..5d671365a 100644 --- a/content/build/access/arns.mdx +++ b/content/build/access/arns.mdx @@ -1,5 +1,5 @@ --- -title: "Arweave Name System (ArNS)" +title: "Ar.io Name System (ArNS)" description: "Assign ArNS names to transaction IDs" --- @@ -11,8 +11,8 @@ ArNS is a naming system that allows you to register human-readable names that po **Example:** -- **Before:** `https://arweave.net/bVLEkL1SOPFCzIYi8T_QNnh17VlDp4RylU6YTwCMVRw` -- **After:** `https://myapp.arweave.net` +- **Before:** `https://turbo-gateway.com/bVLEkL1SOPFCzIYi8T_QNnh17VlDp4RylU6YTwCMVRw` +- **After:** `https://myapp.ar.io` **Learn More:** For detailed information about ArNS architecture and how it works, see our [ArNS Documentation](/learn/arns). @@ -26,10 +26,8 @@ The easiest way to get an ArNS name is via [arns.ar.io](https://arns.ar.io), whi - **Turbo Credits** - Use existing Turbo credits - **ARIO tokens** - Pay with ARIO cryptocurrency -**Alternative registration methods:** +**Alternative registration method:** -- **[Wander Chrome Extension](https://chrome.google.com/webstore/detail/wander)** - Browser-based registration -- **Wander Mobile App** - Register on iOS and Android - **ar.io SDK** - Programmatic registration using the `buyRecord` API ### Using the ar.io SDK @@ -61,7 +59,7 @@ Once you've set up your ArNS name, fetch data using standard HTTP requests: ```js // Fetch content from your ArNS name -const response = await fetch("https://my-data.arweave.net"); +const response = await fetch("https://my-data.ar.io"); if (!response.ok) { throw new Error(`HTTP error! status: ${response.status}`); @@ -83,7 +81,7 @@ ArNS provides significant advantages for accessing data on Arweave: **Flexible Data Management** - **Permanent references** - Keep stable URLs even when updating underlying data - **Replaceable data** - Point names to new transaction IDs as content evolves -- **Undernames** - Organize related content under a single name using underscores (e.g., `v2_myapp.arweave.net`, `docs_myapp.arweave.net`) +- **Undernames** - Organize related content under a single name using underscores (e.g., `v2_myapp.ar.io`, `docs_myapp.ar.io`) **Supporting Network Decentralization** - ArNS purchases contribute to the protocol balance @@ -95,7 +93,7 @@ ArNS provides significant advantages for accessing data on Arweave: } > diff --git a/content/build/access/fetch-data.mdx b/content/build/access/fetch-data.mdx index b62d01823..d19a6770d 100644 --- a/content/build/access/fetch-data.mdx +++ b/content/build/access/fetch-data.mdx @@ -32,8 +32,8 @@ https:/// **Examples:** -- `https://arweave.net/bVLEkL1SOPFCzIYi8T_QNnh17VlDp4RylU6YTwCMVRw` -- `https://permagate.io/FguFk5eSth0wO8SKfziYshkSxeIYe7oK9zoPN2PhSc0` +- `https://turbo-gateway.com/bVLEkL1SOPFCzIYi8T_QNnh17VlDp4RylU6YTwCMVRw` +- `https://ardrive.net/FguFk5eSth0wO8SKfziYshkSxeIYe7oK9zoPN2PhSc0` ### Raw Data Endpoint @@ -63,8 +63,8 @@ Ar.io gateways implement security measures by redirecting requests to sandbox su **What to Expect:** -- Initial request: `https://arweave.net/transaction-id` -- Redirects to: `https://sandbox.arweave.net/transaction-id` (or similar) +- Initial request: `https://turbo-gateway.com/transaction-id` +- Redirects to: `https://sandbox.turbo-gateway.com/transaction-id` (or similar) - Final content served from sandbox subdomain **Important:** Always follow redirects in your applications - the final sandbox URL contains the actual content. @@ -81,7 +81,7 @@ Ar.io gateways implement security measures by redirecting requests to sandbox su ```js // Fetch data from Arweave (follows redirects automatically) -const response = await fetch("https://arweave.net/your-transaction-id", { +const response = await fetch("https://turbo-gateway.com/your-transaction-id", { redirect: "follow", // Follow redirects automatically }); @@ -97,7 +97,7 @@ console.log(data); ```html -My Image +My Image ``` ## Manifests @@ -105,14 +105,14 @@ console.log(data); For organized file collections, use manifests to create friendly path-based URLs: ``` -https://arweave.net//path/to/file +https://turbo-gateway.com//path/to/file ``` **Example:** -- `https://arweave.net/X8Qm…AOhA/index.html` -- `https://arweave.net/X8Qm…AOhA/styles.css` -- `https://arweave.net/X8Qm…AOhA/assets/logo.png` +- `https://turbo-gateway.com/X8Qm…AOhA/index.html` +- `https://turbo-gateway.com/X8Qm…AOhA/styles.css` +- `https://turbo-gateway.com/X8Qm…AOhA/assets/logo.png` [Learn more about manifests](/build/upload/manifests) diff --git a/content/build/access/find-data.mdx b/content/build/access/find-data.mdx index 985f3fbc1..01cf4054a 100644 --- a/content/build/access/find-data.mdx +++ b/content/build/access/find-data.mdx @@ -24,7 +24,7 @@ This separation allows for powerful data discovery while keeping data retrieval ## GraphQL Providers -- **arweave.net** - `https://arweave.net/graphql` - Comprehensive indexing of all Arweave data +- **turbo-gateway.com** - `https://turbo-gateway.com/graphql` - Comprehensive indexing of all Arweave data - **Goldsky** - `https://arweave-search.goldsky.com/graphql` - High-performance GraphQL service with full data coverage @@ -39,7 +39,7 @@ This separation allows for powerful data discovery while keeping data retrieval The easiest way to get started is using the interactive GraphQL playground: -1. Navigate to [https://arweave.net/graphql](https://arweave.net/graphql) in your browser +1. Navigate to [https://turbo-gateway.com/graphql](https://turbo-gateway.com/graphql) in your browser 2. Enter your GraphQL query in the interface 3. Press the "play" button to execute and see results @@ -102,7 +102,7 @@ query { } `; - const response = await fetch("https://arweave.net/graphql", { + const response = await fetch("https://turbo-gateway.com/graphql", { method: "POST", headers: { "Content-Type": "application/json", @@ -140,7 +140,7 @@ query { } `; - const response = await fetch("https://arweave.net/graphql", { + const response = await fetch("https://turbo-gateway.com/graphql", { method: "POST", headers: { "Content-Type": "application/json", @@ -172,7 +172,7 @@ query { } `; - const response = await fetch("https://arweave.net/graphql", { + const response = await fetch("https://turbo-gateway.com/graphql", { method: "POST", headers: { "Content-Type": "application/json", @@ -213,7 +213,7 @@ query { } `; - const response = await fetch("https://arweave.net/graphql", { + const response = await fetch("https://turbo-gateway.com/graphql", { method: "POST", headers: { "Content-Type": "application/json", @@ -250,7 +250,7 @@ query { } `; - const nextResponse = await fetch("https://arweave.net/graphql", { + const nextResponse = await fetch("https://turbo-gateway.com/graphql", { method: "POST", headers: { "Content-Type": "application/json", @@ -271,16 +271,13 @@ query { ## Pagination -As of September 2025, GraphQL endpoints have different limits: +Different GraphQL providers may enforce varying limits on the maximum number of items returned per query. -- **arweave.net**: Supports queries up to **1,000 items at once** -- **Goldsky**: Supports queries up to **100 items at once** - -For larger datasets, use cursor-based pagination to navigate through results. +For larger datasets, use cursor-based pagination techniques to navigate through the complete set of results. **How Pagination Works:** -- Use `first` parameter to specify page size (max 1,000 for arweave.net, max 100 for Goldsky) +- Use `first` parameter to specify page size - Use `pageInfo.hasNextPage` to check if more results exist - Use `cursor` from the last item with `after` parameter for the next page @@ -314,7 +311,7 @@ while (hasNextPage) { } `; - const response = await fetch("https://arweave.net/graphql", { + const response = await fetch("https://turbo-gateway.com/graphql", { method: "POST", headers: { "Content-Type": "application/json", diff --git a/content/build/access/index.mdx b/content/build/access/index.mdx index 617e059ab..955fc41af 100644 --- a/content/build/access/index.mdx +++ b/content/build/access/index.mdx @@ -34,7 +34,7 @@ Different methods serve different needs. Each provides unique capabilities for r

Retrieve data bytes from Arweave

REST API endpoints
-
GET arweave.net/[txId]
+
GET turbo-gateway.com/[txId]
Returns raw data/files
@@ -68,7 +68,7 @@ Different methods serve different needs. Each provides unique capabilities for r **Fetching Data** - Retrieve the actual files/data using transaction IDs -- Access data via REST API: `GET arweave.net/[txId]` +- Access data via REST API: `GET turbo-gateway.com/[txId]` - Stream large files efficiently **Naming with ArNS** @@ -108,7 +108,7 @@ query { Use the transaction ID to retrieve the actual data: ```bash -curl https://arweave.net/[transaction-id-from-above] +curl https://turbo-gateway.com/[transaction-id-from-above] ```
diff --git a/content/build/advanced/arfs/data-model.mdx b/content/build/advanced/arfs/data-model.mdx index f2f9943c6..b29976090 100644 --- a/content/build/advanced/arfs/data-model.mdx +++ b/content/build/advanced/arfs/data-model.mdx @@ -3,7 +3,7 @@ title: "Data Model" description: "Understanding how ArFS organizes data hierarchically using entity relationships" --- -Because of Arweave's permanent and immutable nature, traditional file structure operations such as renaming and moving files or folders cannot be accomplished by simply updating on-chain data. ArFS works around this by defining an append-only transaction data model based on the metadata tags found in the Arweave [Transaction Headers.](https://docs.arweave.org/developers/server/http-api#transaction-format) +Because of Arweave's permanent and immutable nature, traditional file structure operations such as renaming and moving files or folders cannot be accomplished by simply updating onchain data. ArFS works around this by defining an append-only transaction data model based on the metadata tags found in the Arweave [Transaction Headers.](https://docs.arweave.org/developers/server/http-api#transaction-format) This model uses a bottom-up reference method, which avoids race conditions in file system updates. Each file contains metadata that refers to the parent folder, and each folder contains metadata that refers to its parent drive. A top-down data model would require the parent model (i.e. a folder) to store references to its children. diff --git a/content/build/advanced/arfs/index.mdx b/content/build/advanced/arfs/index.mdx index cb51bae0a..30cbc2345 100644 --- a/content/build/advanced/arfs/index.mdx +++ b/content/build/advanced/arfs/index.mdx @@ -5,7 +5,7 @@ description: "A decentralized file system on Arweave for structured data storage Arweave File System, or "ArFS" is a data modeling, storage, and retrieval protocol designed to emulate common file system operations and to provide aspects of mutability to your data hierarchy on [Arweave](/learn/what-is-arweave)'s otherwise permanent, immutable data storage blockweave. -Due to Arweave's permanent, immutable and public nature traditional file system operations such as permissions, file/folder renaming and moving, and file updates cannot be done by simply updating the on-chain data model. +Due to Arweave's permanent, immutable and public nature traditional file system operations such as permissions, file/folder renaming and moving, and file updates cannot be done by simply updating the onchain data model. ArFS works around this by implementing a privacy and encryption pattern and defining an append-only transaction data model using tags within [Arweave Transaction headers](https://docs.arweave.org/developers/server/http-api#transaction-format). diff --git a/content/build/advanced/arfs/upgrading-drives.mdx b/content/build/advanced/arfs/upgrading-drives.mdx index 3c9c5bbf6..ddbc3c581 100644 --- a/content/build/advanced/arfs/upgrading-drives.mdx +++ b/content/build/advanced/arfs/upgrading-drives.mdx @@ -81,19 +81,19 @@ The upgrade process has been made simple by using the [ArDrive app](https://app. If the connected wallet has V1 private drives that need to be updated, a banner will appear at the top of the screen. -![ArDrive Upgrade Banner](https://arweave.net/kJzzrYY4KIHLTC9VOECfzvVAjaO1_FOkejWgsRNbLx4) +![ArDrive Upgrade Banner](https://turbo-gateway.com/kJzzrYY4KIHLTC9VOECfzvVAjaO1_FOkejWgsRNbLx4) ### Step 2: Click "Update Now!" This will open a modal listing the drives that need to be updated, and linking to more information about the upgrade process. -![ArDrive Upgrade Modal](https://arweave.net/Qa-qeKkr1flXl1-fdLapjKNmme0lKA43YQbjmMvnF-U) +![ArDrive Upgrade Modal](https://turbo-gateway.com/Qa-qeKkr1flXl1-fdLapjKNmme0lKA43YQbjmMvnF-U) ### Step 3: Click "Update" The process of upgrading the private drives will begin, and involve signing messages depending on how many drives are being upgraded. When the process is complete, a new modal will appear listing the drives that have been successfully updated. -![ArDrive Upgrade Complete](https://arweave.net/qEwT3oFZbFDpmRQpw9j1c_okN5pIkFqRuoSwBzhc3HQ) +![ArDrive Upgrade Complete](https://turbo-gateway.com/qEwT3oFZbFDpmRQpw9j1c_okN5pIkFqRuoSwBzhc3HQ) ## Manual Upgrade Process diff --git a/content/build/advanced/ethareum.mdx b/content/build/advanced/ethareum.mdx index 551f86401..e4da24238 100644 --- a/content/build/advanced/ethareum.mdx +++ b/content/build/advanced/ethareum.mdx @@ -64,12 +64,6 @@ The Arweave ecosystem primarily uses **keyfiles** rather than seed phrases for w - **Keyfile**: JSON file containing a Json Web Key (JWK) that acts as private keys - **Seed Phrase**: Supported but not universally implemented across all dApps - - **Keyfile Security**: Always treat your keyfile with the same care as you - would private keys for an Ethereum wallet. Learn more about keyfiles in the - [Arweave Cookbook](https://cookbook.arweave.dev/). - - ### Accessing Your Keys Both keyfile and seed phrase are available for download in most dApps: diff --git a/content/build/advanced/normalized-addresses.mdx b/content/build/advanced/normalized-addresses.mdx index cc5a92517..0c34a8902 100644 --- a/content/build/advanced/normalized-addresses.mdx +++ b/content/build/advanced/normalized-addresses.mdx @@ -5,11 +5,11 @@ description: "Learn about normalized addresses - how different blockchain wallet ## Overview -Different blockchains use different formats for the [public keys](/glossary) of wallets, and the [native addresses](/glossary) for those wallets. In most cases, when a system in the Arweave ecosystem needs to display the wallet address of a wallet from a different blockchain, for instance in the `Owner.address` value of an AO process spawned by an ETH wallet, that address will be normalized into the format recognized by Arweave. Specifically, a 43 character base64url representation of the sha256 hash of the public key. This is done to prevent potential errors by systems in the Arweave ecosystem that expect these values to be a certain size and conform to a specific format. +Different blockchains use different formats for the [public keys](/glossary) of wallets, and the [native addresses](/glossary) for those wallets. In most cases, when a system in the Arweave ecosystem needs to display the wallet address of a wallet from a different blockchain, for instance in the `Owner.address` value of a data item signed by an ETH or Solana wallet, that address will be normalized into the format recognized by Arweave. Specifically, a 43 character base64url representation of the sha256 hash of the public key. This is done to prevent potential errors by systems in the Arweave ecosystem that expect these values to be a certain size and conform to a specific format. Essentially, normalized addresses are a way to represent public keys and wallet addresses from other blockchains in a way that is familiar to systems in the Arweave ecosystem. -A tool for easily obtaining a normalized addresses from public keys can be found at [ar://normalize-my-key](https://normalize-my-key.arweave.net/) +A tool for easily obtaining a normalized addresses from public keys can be found at [ar://normalize-my-key](https://normalize-my-key.turbo-gateway.com/) ## At A Glance @@ -89,9 +89,9 @@ Again, this value is used for the GraphQl tag `owner` when uploading data. It ca ## Discovering Public Keys from On-Chain Transactions -In many cases, you may only have a wallet's native address and need to derive its normalized address. Since normalized addresses are derived from public keys, and public keys are not always readily available, you can recover them from on-chain transaction signatures. +In many cases, you may only have a wallet's native address and need to derive its normalized address. Since normalized addresses are derived from public keys, and public keys are not always readily available, you can recover them from onchain transaction signatures. -The [ar://normalize-my-key](https://normalize-my-key.arweave.net/) tool implements this discovery process, allowing you to look up an address or transaction hash/signature to automatically recover the public key and derive the normalized address. +The [ar://normalize-my-key](https://normalize-my-key.turbo-gateway.com/) tool implements this discovery process, allowing you to look up an address or transaction hash/signature to automatically recover the public key and derive the normalized address. ### EVM Chains (Ethereum, Polygon, Base) diff --git a/content/build/advanced/sandboxing.mdx b/content/build/advanced/sandboxing.mdx index e7cdb6a7a..a617170fe 100644 --- a/content/build/advanced/sandboxing.mdx +++ b/content/build/advanced/sandboxing.mdx @@ -6,7 +6,7 @@ description: "Learn about browser sandboxing in ar.io gateways - how security is ## Overview -Browser sandboxing allows data requests to a gateway node to benefit from the security advantages of using a browser's same-origin policy by redirecting the requests to a pseudo-unique subdomain of the gateway's apex domain. For example, an attempt to access `https://arweave.net/gnWKBqFXMJrrksEWrXLQRUQQQeFhv4uVxesHBcT8i6o` would redirect to `https://qj2yubvbk4yjv24syelk24wqivcbaqpbmg7yxfof5mdqlrh4rova.arweave.net/gnWKBqFXMJrrksEWrXLQRUQQQeFhv4uVxesHBcT8i6o` +Browser sandboxing allows data requests to a gateway node to benefit from the security advantages of using a browser's same-origin policy by redirecting the requests to a pseudo-unique subdomain of the gateway's apex domain. For example, an attempt to access `https://turbo-gateway.com/gnWKBqFXMJrrksEWrXLQRUQQQeFhv4uVxesHBcT8i6o` would redirect to `https://qj2yubvbk4yjv24syelk24wqivcbaqpbmg7yxfof5mdqlrh4rova.turbo-gateway.com/gnWKBqFXMJrrksEWrXLQRUQQQeFhv4uVxesHBcT8i6o` Two DNS records are required to link a domain to an Arweave transaction on a gateway node. For example, `www.mycustomsite.com` would need the following records to link it to `www.arweave-gateway.net`: @@ -29,7 +29,7 @@ The integration of TLS is crucial for the implementation of browser sandboxing. Ar.io nodes generate browser sandbox values deterministically. Because of this, it is possible to calculate ahead of time what that value will be for a particular transaction id. -Sandbox values are a Base32 encoding of the transaction ID. ar.io gateways use the following code snippet to accomplish the encoding: +Sandbox values are a Base32 encoding of the transaction ID. Ar.io gateways use the following code snippet to accomplish the encoding: ```typescript const expectedTxSandbox = (id: string): string => { diff --git a/content/build/extensions/bundler.mdx b/content/build/extensions/bundler.mdx index 962fb5fe2..941cf290b 100644 --- a/content/build/extensions/bundler.mdx +++ b/content/build/extensions/bundler.mdx @@ -179,12 +179,6 @@ Now that you have a bundler set up to accept data uploads, continue building you description="Improve query performance with ClickHouse and Parquet integration" href="/build/extensions/clickhouse" /> - } - title="Run Compute Unit" - description="Execute AO processes locally for maximum efficiency" - href="/build/extensions/compute-unit" - /> } title="Buy an ArNS Name" diff --git a/content/build/extensions/clickhouse.mdx b/content/build/extensions/clickhouse.mdx index c66c028bb..b0054f7b6 100644 --- a/content/build/extensions/clickhouse.mdx +++ b/content/build/extensions/clickhouse.mdx @@ -1,419 +1,451 @@ --- title: "ClickHouse & Parquet" -description: "Learn how to use Parquet files and ClickHouse to improve performance and scalability of your ar.io gateway for large datasets." +description: "Scale your gateway's GraphQL performance with ClickHouse analytical queries and Parquet-based data export for large datasets" --- -## Overview +import { Callout } from "fumadocs-ui/components/callout"; +import { Steps, Step } from "fumadocs-ui/components/steps"; +import { Card, Cards } from "fumadocs-ui/components/card"; +import Mermaid from "@/components/Mermaid"; -Ar.io gateway Release 33 introduces a new configuration option for using Parquet files and ClickHouse to improve performance and scalability of your ar.io gateway for large datasets. +ClickHouse is an optional analytical database that runs alongside your ar.io gateway to accelerate GraphQL queries on large datasets. It stores stable historical data in a compressed columnar format while SQLite continues handling real-time writes and recent data. -This guide will walk you through the process of setting up ClickHouse with your ar.io gateway, and importing Parquet files to bootstrap your ClickHouse database. - -## What is Parquet? - -Apache Parquet is a columnar storage file format designed for efficient data storage and retrieval. Unlike row-based storage formats like SQLite, Parquet organizes data by column rather than by row, which provides several advantages for analytical workloads: - -- **Efficient compression**: Similar data is stored together, leading to better compression ratios -- **Columnar access**: You can read only the columns you need, reducing I/O operations -- **Predicate pushdown**: Filter operations can be pushed down to the storage layer, improving query performance - -For more information about Parquet, see the [Parquet documentation](https://parquet.apache.org/docs/). - -## Current Integration with ar.io Gateways - -In the current ar.io gateway implementation, Parquet and ClickHouse run alongside SQLite rather than replacing it. This parallel architecture allows each database to handle what it does best: - -- **SQLite** continues to handle transaction writes and updates -- **ClickHouse** with Parquet files is optimized for fast query performance, especially with large datasets - -The gateway continues to operate with SQLite just as it always has, maintaining all of its normal functionality. Periodically, the gateway will export batches of data from SQLite to Parquet files, which are then imported into ClickHouse. This batch-oriented approach is much more efficient than attempting to synchronize the databases in real-time, as it leverages Parquet's strength in handling large, immutable data sets. - -Note that despite Parquet's efficient compression, gateways may not see significant disk space reduction in all cases. While bundled transaction data is exported to Parquet, L1 data remains in SQLite. Without substantial unbundling and indexing filters, minimal data gets exported to Parquet, limiting potential storage savings. - -With ClickHouse integration enabled, GraphQL queries are primarily routed to ClickHouse, leveraging its superior performance for large datasets. This significantly improves response times while maintaining SQLite's reliability for transaction processing. + + ClickHouse requires version **24.8 or later**. Earlier versions have known issues with projections on ReplacingMergeTree tables. The default Docker image uses ClickHouse 26.3. + -For more information about gateway architecture and data processing, see our [Gateway Architecture](/learn/gateways/architecture) documentation. +## How It Works -## Parquet vs. SQLite in ar.io Gateways +The gateway uses a speed/batch layer architecture: -While SQLite is excellent for transactional workloads and small to medium datasets, it faces challenges with very large datasets: +- **SQLite (speed layer)** — Handles all real-time writes, recent/unstable data, and non-GraphQL operations (data retrieval, chunk fetches, admin APIs) +- **ClickHouse (batch layer)** — Stores stable historical data exported as Parquet files, optimized for analytical GraphQL queries -| Feature | SQLite | Parquet + ClickHouse | -| ------------------------ | ----------------------------- | -------------------------------- | -| Storage model | Row-based | Column-based | -| Query optimization | Basic | Advanced analytical optimization | -| Compression | Limited | High compression ratios | -| Scaling | Limited by single file | Distributed processing capable | -| Write speed | Fast for small transactions | Optimized for batch operations | -| Read speed for analytics | Slower for large datasets | Optimized for analytical queries | -| Ideal use case | Recent transaction data, OLTP | Historical data, OLAP workloads | +|writes| B[SQLite] + B -->|stable data export| C[Parquet Files] + C -->|batch import| D[ClickHouse] + E[GraphQL Query] -->|height > boundary| B + E -->|height ≤ boundary| D +`} /> -## Benefits for Gateway Operators +### Query Routing -Implementing Parquet and ClickHouse alongside SQLite in your ar.io gateway offers several key advantages: +When ClickHouse is enabled, GraphQL queries are split by block height: -- **Dramatically improved query performance** for GraphQL endpoints, especially for large result sets -- **Reduced storage requirements** through efficient columnar compression -- **Better scalability** for growing datasets -- **Faster bootstrapping** of new gateways through Parquet file imports -- **Reduced load on SQLite** by offloading query operations to ClickHouse +- **ClickHouse** handles blocks up to `(max_imported_height - 10)` — the bulk of historical data +- **SQLite** handles blocks above that boundary — recent, potentially unstable data -The primary focus of the Parquet/ClickHouse integration is the significant speed improvement for querying large datasets. Gateway operators managing significant volumes of data will notice substantial performance gains when using this configuration. +Results from both backends are merged and deduplicated before being returned to the client. If either backend is unavailable, a circuit breaker degrades gracefully rather than failing the entire query. -## Storage Considerations +### When to Enable ClickHouse -While Parquet files offer more efficient compression for the data they contain, it's important to understand the storage impact: +Enable ClickHouse if your gateway: -- Bundled transaction data is exported to Parquet and removed from SQLite, potentially saving space -- L1 data remains in SQLite regardless of Parquet configuration -- Space savings are highly dependent on your unbundling filters - without substantial unbundling configurations, minimal data gets exported to Parquet -- The more data you unbundle and export to Parquet, the greater the potential storage efficiency +- Serves heavy GraphQL traffic on historical data +- Indexes millions of data items (large unbundling filters) +- Needs faster bootstrapping than a full SQLite reindex (import pre-built Parquet snapshots) +- Would benefit from 3-5x storage compression on historical data -For gateway operators, this means proper filter configuration is crucial to realize storage benefits. The primary advantage remains significantly improved query performance for large datasets, with potential space savings as a secondary benefit depending on your specific configuration. +Keep it disabled for small deployments, gateways that only serve data retrieval (not GraphQL), or single-node low-resource environments. -The following sections will guide you through setting up ClickHouse with your ar.io gateway, exporting data from SQLite to Parquet, and importing Parquet files to bootstrap your ClickHouse database. +## Quick Start - -The below instructions are designed to be used in a linux environment. Windows and MacOS users must modify the instructions to use the appropriate package manager/ command syntax for their platform. + + +### Configure Environment -Unless otherwise specified, all commands should be run from the root directory of the gateway. +Add ClickHouse settings to your `.env` file: - +```bash +CLICKHOUSE_URL=http://clickhouse:8123 +CLICKHOUSE_PASSWORD=your-password +CLICKHOUSE_SQLITE_MIN_HEIGHT_ENABLED=true +ADMIN_API_KEY=your-admin-key +``` -## Installing ClickHouse +If using a non-default username: -ClickHouse is a powerful, open-source analytical database that excels at handling large datasets and complex queries. It is the tool used by the gateway to integrate with the Parquet format. +```bash +CLICKHOUSE_USER=your-username +``` -For more information about ClickHouse, see the [ClickHouse documentation](https://clickhouse.com/docs/). + - -### Add ClickHouse Repository +### Configure Unbundling Filters -It is recommended to use [official pre-compiled deb packages for Debian or Ubuntu](https://clickhouse.com/docs/install#quick-install). Run these commands to install packages: +ClickHouse stores data items exported from SQLite. The more you unbundle, the more data flows into ClickHouse. At minimum, configure filters that match the pre-built snapshot you plan to import. -```bash -sudo apt-get install -y apt-transport-https ca-certificates curl gnupg -curl -fsSL 'https://packages.clickhouse.com/rpm/lts/repodata/repomd.xml.key' | sudo gpg --dearmor -o /usr/share/keyrings/clickhouse-keyring.gpg +The ArDrive snapshot includes all ArDrive-uploaded data items. To match this: -ARCH=$(dpkg --print-architecture) -echo "deb [signed-by=/usr/share/keyrings/clickhouse-keyring.gpg arch=${ARCH}] https://packages.clickhouse.com/deb stable main" | sudo tee /etc/apt/sources.list.d/clickhouse.list -sudo apt-get update +```bash +ANS104_UNBUNDLE_FILTER='{ "and": [ { "not": { "or": [ { "tags": [ { "name": "Bundler-App-Name", "value": "Warp" } ] }, { "tags": [ { "name": "Bundler-App-Name", "value": "Redstone" } ] }, { "tags": [ { "name": "Bundler-App-Name", "value": "KYVE" } ] }, { "tags": [ { "name": "Bundler-App-Name", "value": "AO" } ] }, { "attributes": { "owner_address": "-OXcT1sVRSA5eGwt2k6Yuz8-3e3g9WJi5uSE99CWqsBs" } }, { "attributes": { "owner_address": "ZE0N-8P9gXkhtK-07PQu9d8me5tGDxa_i4Mee5RzVYg" } }, { "attributes": { "owner_address": "6DTqSgzXVErOuLhaP0fmAjqF4yzXkvth58asTxP3pNw" } } ] } }, { "tags": [ { "name": "App-Name", "valueStartsWith": "ArDrive" } ] } ] }' +ANS104_INDEX_FILTER='{ "tags": [ { "name": "App-Name", "value": "ArDrive-App" } ] }' ``` -This will verify the installation package from official sources and enable installation via `apt-get`. +See the [Filters guide](/build/run-a-gateway/manage/filters) for building custom filter configurations. -### Install ClickHouse +### Start ClickHouse + +ClickHouse runs as a Docker profile alongside your gateway: ```bash -sudo apt-get install -y clickhouse-client +docker compose --profile clickhouse up -d ``` -This will perform the actual installation of the ClickHouse server and client. - -During installation, you will be prompted to set a password for the `default` user. This is required to connect to the ClickHouse server. - -Advanced users may also choose to create a designated user account in clickhouse for the gateway to use, but the default gateway configuration will assume the `default` user. +This starts both the ClickHouse server and the auto-import daemon. - - -## Configure Gateway to use ClickHouse - -### Set Basic ClickHouse Configuration - -Because the gateway will be accessing ClickHouse, host address andthe password for the selected user must be provided. This is done via the `CLICKHOUSE_PASSWORD` environment variable. +### Import a Parquet Snapshot -Update your .env file with the following: +Download and import a pre-built snapshot to bootstrap your ClickHouse instance: ```bash -CLICKHOUSE_URL="http://clickhouse:8123" -CLICKHOUSE_PASSWORD= -``` +# Download the ArDrive snapshot (~3.5GB, current to April 23, 2025) +curl -L https://turbo-gateway.com/JVmsuD2EmFkhitzWN71oi9woADE4WUfvrbBYgremCBM -o ardrive-parquet.tar.gz -If you set a specific user account for the gateway to use, you can set the `CLICKHOUSE_USER` environment variable to the username. +# Extract +tar -xzf ardrive-parquet.tar.gz -```bash -CLICKHOUSE_USER= +# Move into the data directory +mkdir -p data/parquet +mv 2025-04-23-ardrive-ans104-parquet/* data/parquet/ + +# Run the import script +./scripts/clickhouse-import ``` -If omitted, the gateway will use the `default` user. +The import takes 10-20 minutes depending on hardware. -### Configure Unbundling Filters +### Verify -Additionally, The Parquet file provided below contains an unbundled data set that includes all data items uploaded via an ArDrive product, including Turbo. Because of this, it is recommended to include unbundling filters that match, or expand, this configuration. +Check the transaction count: ```bash -ANS104_UNBUNDLE_FILTER='{ "and": [ { "not": { "or": [ { "tags": [ { "name": "Bundler-App-Name", "value": "Warp" } ] }, { "tags": [ { "name": "Bundler-App-Name", "value": "Redstone" } ] }, { "tags": [ { "name": "Bundler-App-Name", "value": "KYVE" } ] }, { "tags": [ { "name": "Bundler-App-Name", "value": "AO" } ] }, { "attributes": { "owner_address": "-OXcT1sVRSA5eGwt2k6Yuz8-3e3g9WJi5uSE99CWqsBs" } }, { "attributes": { "owner_address": "ZE0N-8P9gXkhtK-07PQu9d8me5tGDxa_i4Mee5RzVYg" } }, { "attributes": { "owner_address": "6DTqSgzXVErOuLhaP0fmAjqF4yzXkvth58asTxP3pNw" } } ] } }, { "tags": [ { "name": "App-Name", "valueStartsWith": "ArDrive" } ] } ] }' -ANS104_INDEX_FILTER='{ "tags": [ { "name": "App-Name", "value": "ArDrive-App" } ] }' +docker compose exec clickhouse clickhouse-client \ + --password your-password \ + -q 'SELECT COUNT(DISTINCT id) FROM transactions' ``` - +Expected result: `32712311` (for the ArDrive snapshot). - -### Set Admin API Key - -Lastly, you must have a gateway admin password set. This is used for the periodic export of data from SQLite to Parquet. +Test a GraphQL query: ```bash -ADMIN_API_KEY= +curl -g -X POST \ + -H "Content-Type: application/json" \ + -d '{"query":"query { transactions(ids: [\"YSNwoYB01EFIzbs6HmkGUjjxHW3xuqh-rckYhi0av4A\"]) { edges { node { block { height } bundledIn { id } } } } }"}' \ + http://localhost:3000/graphql ``` -Once the .env file is updated, restart the gateway to apply the changes. - -## Downloading and Importing the Parquet File +## Auto-Import - - -### Download the Parquet File +The `clickhouse-auto-import` container runs a continuous loop that exports new stable data from SQLite to Parquet and imports it into ClickHouse. This keeps ClickHouse up to date without manual intervention. -A Parquet archive file is available for download from [ar://JVmsuD2EmFkhitzWN71oi9woADE4WUfvrbBYgremCBM](https://arweave.net/JVmsuD2EmFkhitzWN71oi9woADE4WUfvrbBYgremCBM). This file contains an unbundled data set that includes all data items uploaded via an ArDrive product, current to April 23, 2025, and compressed using tar.gz. +Each cycle: +1. Reloads TTL rules from `config/clickhouse-ttl-rules.yaml` +2. Advances the export window to the latest stable height +3. Exports new Parquet partitions from SQLite +4. Imports Parquet into ClickHouse staging tables +5. Migrates staging to the final `transactions` table (applying TTL rules) -To download the file, run the following command: +### Auto-Import Configuration -```bash -curl -L https://arweave.net/JVmsuD2EmFkhitzWN71oi9woADE4WUfvrbBYgremCBM -o 2025-04-23-ardrive-ans104-parquet.tar.gz -``` +| Variable | Default | Description | +|----------|---------|-------------| +| `CLICKHOUSE_AUTO_IMPORT_SLEEP_INTERVAL` | `3600` | Seconds between import cycles | +| `CLICKHOUSE_AUTO_IMPORT_HEIGHT_INTERVAL` | `1000` | Blocks per Parquet partition | +| `CLICKHOUSE_AUTO_IMPORT_MAX_ROWS_PER_FILE` | - | Maximum rows per Parquet file | -or visit the url [https://arweave.net/JVmsuD2EmFkhitzWN71oi9woADE4WUfvrbBYgremCBM](https://arweave.net/JVmsuD2EmFkhitzWN71oi9woADE4WUfvrbBYgremCBM) and download the file manually. +The default 1-hour interval is conservative. For gateways with high unbundling throughput, reduce to 600-900 seconds to keep ClickHouse more current. - -If downloaded manually, it will download as a binary file named `JVmsuD2EmFkhitzWN71oi9woADE4WUfvrbBYgremCBM`. This is normal and must be converted to a tar.gz file by renaming it to `2025-04-23-ardrive-ans104-parquet.tar.gz`. +## TTL Rules (Data Retention) -It should also be placed in the root directory of the gateway. +TTL rules control how long data items remain in ClickHouse before automatic expiration. Use them to manage storage costs by expiring ephemeral or low-value data while retaining important content indefinitely. - +### Configuration -The downloaded file will be approximately 3.5GB in size. +Copy the example template and edit: - +```bash +cp config/clickhouse-ttl-rules.example.yaml config/clickhouse-ttl-rules.yaml +``` - -### Extract the Parquet Files +Example configuration: + +```yaml +# Keep L1 transactions forever, expire unmatched data items after 30 days +default_ttl_seconds: 2592000 +l1_never_expires: true + +rules: + # Keep ArDrive content forever + - tag_name: App-Name + tag_value: ArDrive + match: prefix + never_expire: true + + # Expire ephemeral chat messages after 1 day + - tag_name: App-Name + tag_value: ephemeral-chat + ttl_seconds: 86400 + + # Expire test uploads after 1 hour + - tag_name: App-Name + tag_value: test- + match: prefix + ttl_seconds: 3600 + + # Expire specific owner's data after 7 days + - field: owner_address + value: abcDEF0123xyz + ttl_seconds: 604800 +``` -With the parquet file downloaded and placed in the root directory of the gateway, you can extract the file and import it into ClickHouse. +### Rule Precedence -```bash -tar -xzf 2025-04-23-ardrive-ans104-parquet.tar.gz -``` +When multiple rules could apply, the first matching branch wins: -This will extract the file into a directory named `2025-04-23-ardrive-ans104-parquet`, and take a while to complete. +1. `l1_never_expires: true` AND row is L1 transaction → kept forever +2. Any matching rule with `never_expire: true` → kept forever +3. One or more matching TTL rules → shortest `ttl_seconds` wins +4. `default_ttl_seconds` set → applied to all unmatched rows +5. Otherwise → kept indefinitely (no expiry) - +### Match Modes - -### Prepare the Data Directory +- **`exact`** (default) — O(1) dictionary lookup. Use for specific tag values. +- **`prefix`** — Matches the beginning of the value. Use for `Content-Type` with parameters (e.g., `image/gif` catches `image/gif; charset=...`) or app name families. -Next, if you do not already have a `data/parquet` directory, you must create it. Release 33 does not have this directory by default, but future Releases will. You can create the directory by using the following command: + + TTL rules apply only to rows imported after the rules are loaded. Previously imported rows keep their existing expiration. To apply new rules retroactively, re-import affected Parquet partitions. + -```bash -mkdir -p data/parquet -``` +### Normalization + +- Tag names are lower-cased and trimmed +- Tag values are trimmed but case-preserving +- Owner values use base64url format (as displayed on Arweave explorers) + +## Deployment Topologies -or by starting the gateway ClickHouse container with the following command: +### Single Gateway (Default) + +One gateway runs ClickHouse locally. The auto-import daemon handles continuous data export and import. ```bash -docker compose --profile clickhouse up clickhouse -d +docker compose --profile clickhouse up -d ``` - -Depending on your system configurations, allowing the gateway to create the directory may result in the directory being created with incorrect permissions. If this is the case, you can remove the restrictions by running the following command: +### Shared ClickHouse Cluster + +Multiple gateways share a replicated ClickHouse cluster. Only **one** gateway runs `clickhouse-auto-import` as the writer; others query the same cluster as readers. ```bash -sudo chmod -R 777 data/parquet +# Writer gateway +CLICKHOUSE_URL=http://clickhouse-writer:8123 +# runs clickhouse-auto-import + +# Reader gateways +CLICKHOUSE_URL=http://clickhouse-reader:8123 +# do NOT run clickhouse-auto-import ``` + + Reader gateways still need `START_WRITERS=true` because SQLite indexing is required for data retrieval, chunk fetches, and other non-GraphQL operations. Only GraphQL queries are routed to ClickHouse. -With the directory created, you can now move the extracted parquet files into it. +### Independent ClickHouse per Gateway -```bash -mv 2025-04-23-ardrive-ans104-parquet/* data/parquet -``` +Each gateway runs its own ClickHouse instance, bootstrapped from a shared Parquet snapshot (downloaded from Arweave, S3, or BitTorrent). No shared state between gateways. - +This is the simplest topology for operators who want ClickHouse performance without cluster management. - -### Import Data into ClickHouse +## Configuration Reference -When this is complete, you can run the import script to import the parquet files into ClickHouse. +### Connection -If you haven't done so already, start the ClickHouse container with the following command: +| Variable | Default | Description | +|----------|---------|-------------| +| `CLICKHOUSE_URL` | - | ClickHouse HTTP endpoint (e.g., `http://clickhouse:8123`) | +| `CLICKHOUSE_USER` | `default` | ClickHouse username | +| `CLICKHOUSE_PASSWORD` | - | ClickHouse password | -```bash -docker compose --profile clickhouse up clickhouse -d -``` +### Query Routing -Then run the import script with the following command: +| Variable | Default | Description | +|----------|---------|-------------| +| `CLICKHOUSE_SQLITE_MIN_HEIGHT_ENABLED` | `false` | Enable height-based query split between ClickHouse and SQLite | +| `CLICKHOUSE_SQLITE_MIN_HEIGHT_BUFFER` | `10` | Number of blocks below ClickHouse max height still served by SQLite | +| `CLICKHOUSE_MAX_HEIGHT_CACHE_TTL_SECONDS` | `60` | Cache duration for ClickHouse max-height lookup | +| `CLICKHOUSE_QUERY_TIMEOUT_SECONDS` | `3` | Query timeout (both client and server side) | +| `CLICKHOUSE_GQL_MAX_ROWS_TO_READ` | `10000000` | Safety guardrail: max rows scanned per query | +| `CLICKHOUSE_GQL_DEDUPE_HEADROOM` | `4` | Pagination headroom multiplier for deduplication | -```bash -./scripts/clickhouse-import -``` +### Circuit Breaker -This process will take several minutes, and will output the progress of the import. +The circuit breaker protects against cascading failures when the SQLite leg is unhealthy. When tripped, queries degrade to ClickHouse-only results with a partial result warning. - - +| Variable | Default | Description | +|----------|---------|-------------| +| `CLICKHOUSE_SQLITE_CIRCUIT_BREAKER_TIMEOUT_MS` | `5000` | SQLite query timeout before counting as error | +| `CLICKHOUSE_SQLITE_CIRCUIT_BREAKER_ERROR_THRESHOLD_PERCENTAGE` | `80` | Error rate percentage that trips the breaker | +| `CLICKHOUSE_SQLITE_CIRCUIT_BREAKER_ROLLING_COUNT_TIMEOUT_MS` | `60000` | Rolling window for error rate calculation | +| `CLICKHOUSE_SQLITE_CIRCUIT_BREAKER_RESET_TIMEOUT_MS` | `30000` | Wait time before attempting half-open trial | -## Verifying Successful Import +## Tuning - - -### Verify ClickHouse Import +### Short Pages in GraphQL Pagination -To verify that the import was successful, run the following commands: +If GraphQL queries return fewer results than expected per page, unmerged duplicates in ClickHouse may be consuming pagination slots. Increase the deduplication headroom: ```bash -clickhouse client --password -h localhost -q 'SELECT COUNT(DISTINCT id) FROM transactions' +CLICKHOUSE_GQL_DEDUPE_HEADROOM=8 # default is 4 ``` -Being sure to replace `` with the password you set for the selected ClickHouse user. +This makes the inner query fetch more rows to compensate for duplicates that get collapsed. -This should return a count of the number of unique transactions in the parquet file, which is `32712311`. +### Query Timeouts - +For queries over very large datasets or complex tag filters, increase the timeout: - -### Test GraphQL Endpoint +```bash +CLICKHOUSE_QUERY_TIMEOUT_SECONDS=10 # default is 3 +``` -You can also verify that the data is being served by the gateway's GraphQL endpoint by ensuring the gateway is not proxying its GraphQL queries (Make sure `GRAPHQL_HOST` is not set) and running the following command: +### Max Rows Safety Guardrail -```bash -curl -g -X POST \ - -H "Content-Type: application/json" \ - -d '{"query":"query { transactions(ids: [\"YSNwoYB01EFIzbs6HmkGUjjxHW3xuqh-rckYhi0av4A\"]) { edges { node { block { height } bundledIn { id } } } } }"}' \ - http://localhost:3000/graphql +If queries hit the `max_rows_to_read` limit (returns an error), you can raise it — but do so with caution as it removes the protection against runaway full-table scans: -# Expected output: -# {"data":{"transactions":{"edges":[{"node":{"block":{"height":1461918},"bundledIn":{"id":"ylhb0PqDtG5HwBg00_RYztUl0x2RuKvbNzT6YiNR2JA"}}}]}}} +```bash +CLICKHOUSE_GQL_MAX_ROWS_TO_READ=50000000 # default is 10M ``` - - +## Upgrading -## Starting and Stopping the Gateway with ClickHouse +### Schema Evolution -The gateway ClickHouse container is run as a "profile" in the main docker compose file. That means you must specify the profile when starting or stopping the gateway if you want to include the ClickHouse container in the commands. +Schema changes are applied automatically on each import cycle via idempotent `ALTER TABLE` statements. No manual migration steps are needed for routine upgrades. - - -### Start Gateway with ClickHouse +### Projection Rebuild (One-Time) -To start the gateway with the ClickHouse profile, run the following command: +Gateways upgrading from pre-24.8 ClickHouse deployments may need to rebuild the owner projection: -```bash -docker compose --profile clickhouse up -d +```sql +ALTER TABLE transactions MATERIALIZE PROJECTION owner_projection; ``` -This will start all of the containers normally covered by the `docker compose up` command, but will also start the ClickHouse container. +This runs in the background and may take time on large tables. - +### Rollback - -### Stop Gateway with ClickHouse +If you need to roll back a ClickHouse upgrade: + +1. Drop the `transactions` table +2. Re-import from your Parquet files using `./scripts/clickhouse-import` + +Parquet files are the source of truth — ClickHouse is always rebuildable from them. + +## Troubleshooting -To stop the gateway with the ClickHouse profile, run the following command: +### Import Failures + +Failed imports leave files in `data/etl/staging/` for inspection. Check the auto-import container logs: ```bash -docker compose --profile clickhouse down +docker compose --profile clickhouse logs clickhouse-auto-import -f ``` -This will stop all of the containers normally covered by the `docker compose down` command, but will also stop the ClickHouse container. +Fix the underlying issue (disk space, permissions, ClickHouse connectivity) and restart the container — the next cycle will retry. - +### Disk Usage Growing - -### Manage ClickHouse Container Only +If ClickHouse disk usage is higher than expected: -To start or stop only the ClickHouse container, you can use the following commands: +- Check your TTL rules are configured and expiring data as intended +- Verify background merges are running (rows are only deleted on merge): + ```bash + docker compose exec clickhouse clickhouse-client \ + --password your-password \ + -q 'SELECT partition, count() FROM system.parts WHERE table = '\''transactions'\'' GROUP BY partition' + ``` +- Force a merge on a specific partition if needed (heavyweight operation): + ```bash + OPTIMIZE TABLE transactions PARTITION 14 FINAL + ``` -```bash -docker compose --profile clickhouse up clickhouse -d -``` +### Checking Import Status -and +View the current max imported height: ```bash -docker compose --profile clickhouse down clickhouse +docker compose exec clickhouse clickhouse-client \ + --password your-password \ + -q 'SELECT max(height) FROM transactions' ``` - - - -## Useful Docker Commands +Compare this to the gateway's current height (`/height` endpoint) — the difference indicates how far behind ClickHouse is. -Monitor and manage your gateway with ClickHouse using these commands: +## Docker Commands ```bash -# View all running services -docker ps - -# Start gateway with ClickHouse profile +# Start gateway with ClickHouse docker compose --profile clickhouse up -d -# Stop gateway with ClickHouse profile +# Stop gateway with ClickHouse docker compose --profile clickhouse down -# Pull latest images -docker compose --profile clickhouse pull - -# Start only ClickHouse container -docker compose --profile clickhouse up clickhouse -d - -# Stop only ClickHouse container -docker compose --profile clickhouse down clickhouse +# View ClickHouse logs +docker compose --profile clickhouse logs clickhouse -f -# Follow gateway logs -docker compose logs core -f -n 10 +# View auto-import logs +docker compose --profile clickhouse logs clickhouse-auto-import -f -# Follow ClickHouse logs -docker compose --profile clickhouse logs clickhouse -f -n 10 +# Restart auto-import after config change +docker compose --profile clickhouse restart clickhouse-auto-import -# Check ClickHouse container status -docker compose --profile clickhouse ps clickhouse - -# Restart ClickHouse container -docker compose --profile clickhouse restart clickhouse +# Pull latest images +docker compose --profile clickhouse pull ``` -## Next Steps - -Now that you have ClickHouse set up for improved query performance, continue building your gateway infrastructure: +## Related } - title="Set Up Monitoring" - description="Deploy Grafana to visualize your gateway's performance metrics" - href="/build/extensions/grafana" + icon={} + title="Filters" + description="Configure unbundling and indexing filters that control what data flows into ClickHouse" + href="/build/run-a-gateway/manage/filters" /> } - title="Deploy Bundler" - description="Accept data uploads directly through your gateway" - href="/build/extensions/bundler" + icon={} + title="Grafana Monitoring" + description="Visualize gateway and ClickHouse performance metrics" + href="/build/extensions/grafana" /> } - title="Run Compute Unit" - description="Execute AO processes locally for maximum efficiency" - href="/build/extensions/compute-unit" + title="Environment Variables" + description="Full configuration reference for all gateway services" + href="/build/run-a-gateway/manage/environment-variables" /> } - title="Join the Network" - description="Register your gateway and start serving the permanent web" - href="/build/run-a-gateway/join-the-network" + icon={} + title="Gateway Architecture" + description="Understand the data layer and worker system architecture" + href="/learn/gateways/architecture" /> diff --git a/content/build/extensions/compute-unit.mdx b/content/build/extensions/compute-unit.mdx deleted file mode 100644 index 4fada28e6..000000000 --- a/content/build/extensions/compute-unit.mdx +++ /dev/null @@ -1,364 +0,0 @@ ---- -title: "AO Compute Unit (CU)" -description: "Steps for deploying an AO Compute Unit (CU) sidecar alongside your ar.io Gateway." ---- - -## Overview - -An AO Compute Unit (CU) is a critical component in the AO ecosystem responsible for executing AO processes and maintaining their state. CUs serve as the computational backbone of the AO network by: - -- **Processing Messages**: CUs receive and process messages sent to AO processes -- **Executing WASM Modules**: CUs run the WebAssembly (WASM) code that defines process behavior -- **Maintaining State**: CUs track and update the state of AO processes -- **Creating Checkpoints**: CUs periodically save process state to the Arweave network as checkpoints - -Running a CU alongside your gateway allows you to: - -1. Process AO requests locally rather than relying on external services -2. Improve response times for AO-related queries -3. Contribute computational resources to the AO network -4. Ensure your gateway has reliable access to AO functionality - -For more detailed information about Compute Units, please refer to the [AO Cookbook: Units](https://cookbook_ao.arweave.net/concepts/units.html#summary). - -## System Requirements - -Before deploying a CU, ensure your system meets the following requirements: - -- **Recommended**: At least 16GB RAM for optimal CU operation -- **Minimum**: 4GB RAM is possible with adjusted memory limits (see resource allocation settings) -- At least 100GB disk space dedicated to CU operation -- These requirements are separate from your gateway requirements - - - Running a CU is resource-intensive. Make sure your system has sufficient - resources to handle both the gateway and the CU. While you can run a CU with - less than the recommended RAM, you'll need to adjust the memory limits - accordingly. - - -## Deploying an AO CU - - - -### Navigate to Gateway Directory - -First, navigate to the root directory of your gateway: - -```bash -cd /path/to/your/gateway -``` - - - - -### Configure Environment Variables - -Copy the example environment file: - -```bash -cp .env.ao.example .env.ao -``` - -### Default .env.ao.example Contents - -The default `.env.ao.example` file contains the following settings: - -``` -CU_WALLET='[wallet json here]' -PROCESS_CHECKPOINT_TRUSTED_OWNERS=WjnS-s03HWsDSdMnyTdzB1eHZB2QheUWP_FVRVYxkXk,-HFe6PleLxj1EdFMYMSetT2NIJioDsZIktn-Y0AwP54 -GATEWAY_URL=http://envoy:3000 -UPLOADER_URL=http://envoy:3000/bundler -``` - -These default settings are configured to work with a gateway running on the same machine, but you'll need to modify them as described below. - -Open the `.env.ao` file in your preferred text editor: - -```bash -nano .env.ao -``` - -Configure the following settings: - -1. **CU_WALLET**: Replace `'[wallet json here]'` with the JSON from an Arweave wallet. - - - The entire JSON must be placed on a single line for proper registration. - - -2. **PROCESS_CHECKPOINT_TRUSTED_OWNERS**: This is a comma-separated list of trusted wallet addresses: - - ``` - PROCESS_CHECKPOINT_TRUSTED_OWNERS=WjnS-s03HWsDSdMnyTdzB1eHZB2QheUWP_FVRVYxkXk,-HFe6PleLxj1EdFMYMSetT2NIJioDsZIktn-Y0AwP54 - ``` - - - - If you are uploading your own checkpoints, you should add your own CU wallet address after the default value, separated by a comma: - - ``` - PROCESS_CHECKPOINT_TRUSTED_OWNERS=WjnS-s03HWsDSdMnyTdzB1eHZB2QheUWP_FVRVYxkXk,-HFe6PleLxj1EdFMYMSetT2NIJioDsZIktn-Y0AwP54,YOUR_WALLET_ADDRESS_HERE - ``` - - This allows your CU to trust checkpoints from both the official source and your own wallet. - - - -3. **GATEWAY_URL**: By default, this is set to use your own gateway: - - ``` - GATEWAY_URL=http://envoy:3000 - ``` - - A gateway must be set to index all ANS-104 data items from AO or the CU will not operate properly. Most users will want to set this to: - - ``` - GATEWAY_URL=https://arweave.net - ``` - -4. **UPLOADER_URL**: By default, this is set to use a bundler sidecar run by your gateway: - - ``` - UPLOADER_URL=http://envoy:3000/bundler - ``` - - - Checkpoints are uploaded to Arweave, so the upload must be paid for. You - must ensure your wallet has sufficient funds: - If using - `https://up.arweave.net` (recommended), your CU_WALLET must contain Turbo - Credits - If using your own bundler or another service, you'll need the - appropriate token (AR or other) - Without proper funding, checkpoints will - fail to upload and your CU may not function correctly - - - The simplest option for most users is to use: - - ``` - UPLOADER_URL=https://up.arweave.net - ``` - - This requires your CU_WALLET to contain Turbo Credits. - -5. **Optional: Disable Checkpoint Creation**: If you want to disable checkpoint uploads, add: - ``` - DISABLE_PROCESS_CHECKPOINT_CREATION=true - ``` - -### Example of a Completed .env.ao File - -Here's an example of what your completed `.env.ao` file might look like with common settings: - -``` -CU_WALLET='{"kty":"RSA","e":"AQAB","n":"mYM07..."}' -PROCESS_CHECKPOINT_TRUSTED_OWNERS=WjnS-s03HWsDSdMnyTdzB1eHZB2QheUWP_FVRVYxkXk,-HFe6PleLxj1EdFMYMSetT2NIJioDsZIktn-Y0AwP54 -GATEWAY_URL=https://arweave.net -UPLOADER_URL=https://up.arweave.net -``` - -After making your changes, save and exit the nano editor: - -1. Press `Ctrl+X` to exit -2. Press `Y` to confirm saving changes -3. Press `Enter` to confirm the filename - -### Optional Resource Allocation Settings - -You can fine-tune the CU's resource usage by adding these optional environment variables: - -1. **PROCESS_WASM_MEMORY_MAX_LIMIT**: Sets the maximum memory limit (in bytes) for WASM processes. - - ``` - PROCESS_WASM_MEMORY_MAX_LIMIT=17179869184 # 16GB (16 * 1024^3) - ``` - - - - To work with the ar.io process, `PROCESS_WASM_MEMORY_MAX_LIMIT` must be at least `17179869184` (16GB). - - Note: This doesn't mean your server needs 16GB of RAM. This is the maximum memory limit the CU will support for processes. Most processes don't use their maximum allocated memory. - - You can set this value to 16GB even if your server only has 4GB of RAM. However, if a process requires more memory than your server has available, the CU will fail when evaluating messages that need more memory. - - - -2. **WASM_EVALUATION_MAX_WORKERS**: Sets the maximum number of worker threads for WASM evaluation. - - ``` - WASM_EVALUATION_MAX_WORKERS=4 # Example: Use 4 worker threads - ``` - - - This will default to (available CPUs - 1) if not specified. If you're - running a gateway and unbundling on the same server, consider setting this - to 2 or less to avoid overloading your CPU. - - -3. **PROCESS_WASM_COMPUTE_MAX_LIMIT**: The maximum Compute-Limit, in bytes, supported for ao processes (defaults to 9 billion) - - ``` - PROCESS_WASM_COMPUTE_MAX_LIMIT=9000000000 - ``` - -4. **NODE_OPTIONS**: Sets Node.js memory allocation for the Docker container. - ``` - NODE_OPTIONS=--max-old-space-size=8192 # Example: 8GB for Node.js heap - ``` - - - Start with conservative values and monitor performance. You can adjust these - settings based on your system's capabilities and the CU's performance. - - - - - -### Start the CU Container - -Once your environment file is configured, start the CU container: - -```bash -docker compose --env-file .env.ao -f docker-compose.ao.yaml up -d -``` - -This command uses the following flags: - -- `--env-file .env.ao`: Specifies the environment file to use -- `-f docker-compose.ao.yaml`: Specifies the Docker Compose file to use -- `up`: Creates and starts the containers -- `-d`: Runs containers in detached mode (background) - - - - -### Check the Logs - -To check the logs of your CU container: - -```bash -docker compose -f docker-compose.ao.yaml logs -f --tail=20 -``` - -This command uses the following flags: - -- `-f`: Follows the log output (continuous display) -- `--tail=20`: Shows only the last 20 lines of logs - -Exit the logs by pressing `Ctrl+C`. - - - - -## Connecting Your Gateway to the CU - -To make your gateway use your local CU: - -1. Add the following line to your gateway's `.env` file: - - ``` - AO_CU_URL=http://ao-cu:6363 - ``` - - This assumes the CU is running on the same machine as the gateway. - -2. Restart your gateway: - ```bash - docker compose down - docker compose up -d - ``` - - - A CU won't do anything until requests are being made of it. By connecting your - gateway to the CU, you'll start generating these requests. - - -### Accessing Your CU - -Once properly set up and connected to your gateway, you can access your CU via: - -``` -https:///ao/cu -``` - -This endpoint allows you to interact with your CU directly through your gateway's domain. - -## Important Notes - -- **Initial Processing Time**: A CU will need to process AO history before it can give valid responses. This process can take several hours. - -- **Gateway Fallback**: A gateway on release 27 or above will fallback to arweave.net if its default CU is not responding quickly enough, so gateway operations will not be significantly impacted during the initial processing. - -- **Monitoring Progress**: Check the CU logs after pointing a gateway at it to watch the process of working through AO history: - - ```bash - docker compose -f docker-compose.ao.yaml logs -f --tail=20 - ``` - -- **Resource Usage**: Running a CU is resource-intensive. Monitor your system's performance to ensure it can handle both the gateway and CU workloads. - -## Useful Docker Commands - -Monitor and manage your AO Compute Unit with these commands: - -```bash -# View all running services -docker ps - -# Start CU container with environment file -docker compose --env-file .env.ao -f docker-compose.ao.yaml up -d - -# Stop CU container -docker compose -f docker-compose.ao.yaml down - -# Pull latest CU images -docker compose -f docker-compose.ao.yaml pull - -# Follow CU logs -docker compose -f docker-compose.ao.yaml logs -f --tail=20 - -# Check CU container status -docker compose -f docker-compose.ao.yaml ps - -# Restart CU container -docker compose -f docker-compose.ao.yaml restart - -# View CU logs without following -docker compose -f docker-compose.ao.yaml logs --tail=50 - -# Start CU in foreground (for debugging) -docker compose --env-file .env.ao -f docker-compose.ao.yaml up -``` - -## Next Steps - -Now that you have a Compute Unit running alongside your gateway, continue building your infrastructure: - - - } - title="Set Up Monitoring" - description="Deploy Grafana to visualize your gateway's performance metrics" - href="/build/extensions/grafana" - /> - } - title="Add ClickHouse" - description="Improve query performance with ClickHouse and Parquet integration" - href="/build/extensions/clickhouse" - /> - } - title="Deploy Bundler" - description="Accept data uploads directly through your gateway" - href="/build/extensions/bundler" - /> - } - title="Join the Network" - description="Register your gateway and start serving the permanent web" - href="/build/run-a-gateway/join-the-network" - /> - diff --git a/content/build/extensions/grafana.mdx b/content/build/extensions/grafana.mdx index ff777c8f9..1ba86f4b0 100644 --- a/content/build/extensions/grafana.mdx +++ b/content/build/extensions/grafana.mdx @@ -414,10 +414,4 @@ Now that you have monitoring set up, continue building your gateway infrastructu description="Accept data uploads directly through your gateway" href="/build/extensions/bundler" /> - } - title="Run Compute Unit" - description="Execute AO processes locally for maximum efficiency" - href="/build/extensions/compute-unit" - />
diff --git a/content/build/extensions/index.mdx b/content/build/extensions/index.mdx index 11eb09d4f..7d636491c 100644 --- a/content/build/extensions/index.mdx +++ b/content/build/extensions/index.mdx @@ -40,9 +40,6 @@ The following sidecars are developed and maintained by the ar.io team, designed access control. - }> - Execute AO processes locally with WASM module support and state management. - **Ready to enhance your gateway?** Click any sidecar above to get started with detailed setup guides. diff --git a/content/build/extensions/meta.json b/content/build/extensions/meta.json index 215ce7eba..854d077ed 100644 --- a/content/build/extensions/meta.json +++ b/content/build/extensions/meta.json @@ -2,5 +2,5 @@ "title": "Extensions & Sidecars", "icon": "Plug", "defaultOpen": false, - "pages": ["grafana", "clickhouse", "bundler", "compute-unit"] + "pages": ["grafana", "clickhouse", "bundler"] } diff --git a/content/build/guides/application-distribution.mdx b/content/build/guides/application-distribution.mdx index 01b41d147..4f23526ee 100644 --- a/content/build/guides/application-distribution.mdx +++ b/content/build/guides/application-distribution.mdx @@ -13,7 +13,7 @@ You can publish binaries once and serve them from stable, human-readable URLs th **What you get:** - Permanent storage for all releases -- Stable URLs via ArNS (for example, `releases_yourapp.arweave.net`) +- Stable URLs via ArNS (for example, `releases_yourapp.turbo-gateway.com`) - Simple versioned paths for latest and pinned releases - Automated CI/CD with GitHub Actions and Turbo - No centralized registries or CDNs @@ -161,7 +161,7 @@ await ant.setRecord({ This creates a complete URL API: ``` -https://install_cli_harlequin.arweave.net/ +https://install_cli_harlequin.turbo-gateway.com/ ├── install_cli.sh # Installation script (default) ├── releases # JSON API with version metadata └── releases/ @@ -175,7 +175,7 @@ https://install_cli_harlequin.arweave.net/ ``` **Key Benefits:** -- `install_cli_harlequin.arweave.net` provides a permanent, friendly URL +- `install_cli_harlequin.turbo-gateway.com` provides a permanent, friendly URL - Manifest paths create a logical REST-like API structure - Updating the ArNS record points to new releases while preserving the URL diff --git a/content/build/guides/arns-marketplace.mdx b/content/build/guides/arns-marketplace.mdx index 5c9c68b18..a7f7919dc 100644 --- a/content/build/guides/arns-marketplace.mdx +++ b/content/build/guides/arns-marketplace.mdx @@ -5,44 +5,49 @@ description: "Potential for trading and selling ArNS tokens and digital assets i import { Globe, Code, Book } from "lucide-react"; -**ArNS tokens** have the potential to be traded and sold in decentralized marketplaces. ANTs (Arweave Name Tokens) are both smart contracts and transferable tokens, making them valuable digital assets that could be bought, sold, and traded. However, no established marketplace has yet emerged as the preferred platform for ArNS trading. +**ArNS tokens** can be traded and sold on standard Solana NFT marketplaces. ANTs (Ar.io Name Tokens) are Metaplex Core NFTs, making them tradeable on **Tensor**, **Magic Eden**, and other compatible platforms. -## Current State of ArNS Trading +## Trading ANTs on NFT Marketplaces -**No established marketplace exists yet** - While ANTs are technically transferable tokens, there is currently no widely adopted marketplace specifically for ArNS trading. +ANTs are standard Metaplex Core NFTs on Solana. This means they are natively supported by the Solana NFT ecosystem: -**Direct transfers are possible** - You can transfer ANTs directly between wallets, but this requires technical knowledge and direct coordination between buyer and seller. +- **[Tensor](https://tensor.trade)** — Solana's leading NFT marketplace with advanced trading features +- **[Magic Eden](https://magiceden.io)** — Popular cross-chain NFT marketplace with Solana support +- **Direct transfers** — ANTs can be sent directly between Solana wallets -**Future potential** - As the ArNS ecosystem grows, dedicated marketplaces may emerge to facilitate easier trading of ArNS tokens and domains. +### Lazy Reconciliation + +When an ANT is sold on a marketplace (outside the ar.io protocol), the existing controllers are not immediately cleared. Instead, **lazy reconciliation** occurs on the next write operation — controllers are cleared automatically, ensuring the new owner gets clean control of the name. ## What Are ANTs? -**Arweave Name Tokens (ANTs)** are: +**Ar.io Name Tokens (ANTs)** are: -- **Smart contracts** - Define the rules and functionality of your domain -- **Transferable tokens** - Can be bought, sold, and traded -- **Digital assets** - Represent ownership of ArNS domains -- **Permanent** - Stored on Arweave forever +- **Metaplex Core NFTs** — Standard Solana NFTs with onchain state in PDAs +- **Transferable assets** — Can be bought, sold, and traded on any compatible marketplace +- **Digital assets** — Represent ownership and control of ArNS domains +- **DNS-like routing** — Each ANT stores records pointing to Arweave transaction IDs -## How Trading Could Work +## How Trading Works ### 1. Token Ownership **When you own an ANT:** -- You control the domain name -- You can update where it points -- You can transfer ownership -- You could potentially sell it to others +- You control the domain name and its records +- You can update where it points (Arweave TX IDs or IPFS CIDs) +- You can transfer ownership via direct transfer or marketplace sale +- You can list it on Tensor, Magic Eden, or other Solana NFT marketplaces +- You can add up to 10 controllers who can manage records on your behalf -### 2. Potential Marketplace Dynamics +### 2. Marketplace Trading -**Possible trading mechanisms:** +**Trading mechanisms:** -- **Direct transfers** - Send tokens directly to another wallet -- **Marketplace platforms** - Use dedicated trading platforms (when available) -- **Auction systems** - Bid on available domains (when implemented) -- **Fixed price sales** - Set a price and wait for buyers (when supported) +- **Marketplace listings** — List your ANT on Tensor or Magic Eden with a fixed price or auction +- **Direct transfers** — Send the NFT directly to another Solana wallet +- **Collection offers** — Accept collection-wide offers from buyers +- **Instant sales** — Sell into existing bids on marketplaces ### 3. Name Characteristics diff --git a/content/build/guides/crossmint-nft-minting-app.mdx b/content/build/guides/crossmint-nft-minting-app.mdx deleted file mode 100644 index 5370a42df..000000000 --- a/content/build/guides/crossmint-nft-minting-app.mdx +++ /dev/null @@ -1,548 +0,0 @@ ---- -title: "Crossmint NFT Minting App" -description: "Build a decentralized NFT minting app with Arweave and Crossmint" ---- - -Build a **completely decentralized NFT minting app** that leverages the power of Arweave for permanent storage and Crossmint for simplified NFT creation. Learn how to store NFT content permanently, create and mint NFTs, build a frontend with authentication and payment options, and deploy your application to Arweave. - -## What You'll Learn - -- How to store NFT content permanently on Arweave -- How to create and mint NFTs using Crossmint's API -- How to build a frontend with authentication and payment options -- How to deploy your application to Arweave -- How to configure a human-readable ArNS domain - -## Example Project - -- **Live Demo**: [https://crossmint_zerotoarweave.arweave.net](https://crossmint_zerotoarweave.arweave.net) -- **GitHub Repository**: [https://github.com/ar-io/crossmint-arweave-example](https://github.com/ar-io/crossmint-arweave-example) - -## Prerequisites - -- Node.js environment -- Arweave wallet with AR tokens -- Crossmint developer account -- Basic understanding of React and JavaScript - -## Quick Start - - - - ### Storage Setup - - Store your NFT image permanently on Arweave using [ArDrive.io](http://ArDrive.io): - - - - #### Generate AI Image - - **Create an AI-generated image for your NFT:** - - 1. Visit [ChatGPT](https://chat.openai.com/) or another AI image generation tool - 2. Use a prompt to generate an interesting image for your NFT - 3. Download the generated image to your local machine - 4. Make sure to save it in a common format like PNG or JPG - - - - - #### Upload to Arweave - - **Store the image permanently on Arweave:** - - 1. Visit [ArDrive.io](http://ArDrive.io) and log in to your account - 2. Fund your ArDrive wallet if needed (requires AR tokens) - 3. Create a new folder for your NFT project - 4. Drag and drop your AI-generated image into this folder - 5. Wait for the upload to complete and for the transaction to be processed - - - - - #### Get Transaction ID - - **Retrieve the Arweave Transaction ID:** - - 1. Click on the uploaded image in your ArDrive folder - 2. Look for the "Transaction ID" or "TX ID" in the file details - 3. Copy this Transaction ID - it looks like `Abc123XYZ...` - 4. Save this Transaction ID - you'll need it for creating your NFT metadata - - - **Important:** This Transaction ID is the permanent reference to your image on the Arweave network. - - - - - - - - - ### Collection and Template Creation - - Create an ERC-1155 collection and template using Crossmint's API: - - - - #### Create Account - - **Set up your Crossmint developer account:** - - 1. Visit the [Crossmint Staging Console](https://staging.crossmint.com/console) - 2. Sign in and accept the dialog to continue - 3. Note that Crossmint provides two environments: - - **Staging**: For development and testing (what we'll use first) - - **Production**: For your final, live application - - - - - #### Get API Key - - **Get a server-side API key:** - - 1. After logging in, navigate to the "Integrate" tab - 2. Click on "API Keys" at the top of the page - 3. In the "Server-side keys" section, click "Create new key" - 4. Select the following scopes under "Minting API": - - `collections.create` - Required for creating a new collection - - `nfts.create` - Required for minting NFTs - - `nfts.read` - Needed to read NFT information - 5. Create and save this API key securely - - - - - #### Create Collection - - **Create an ERC-1155 collection:** - - ```javascript - const apiKey = "YOUR_API_KEY"; - const env = "staging"; // Using staging environment for development - const url = `https://${env}.crossmint.com/api/2022-06-09/collections`; - - const options = { - method: "POST", - headers: { - "accept": "application/json", - "content-type": "application/json", - "x-api-key": apiKey, - }, - body: JSON.stringify({ - chain: "ethereum-sepolia", // Using Ethereum testnet for development - fungibility: "semi-fungible", // For ERC-1155 tokens - metadata: { - name: "lil dumdumz SFT Collection", - imageUrl: "https://arweave.net/YOUR_ARWEAVE_TX_ID", // Optional collection image - description: "A collection of semi-fungible tokens with images stored on Arweave" - } - }), - }; - - fetch(url, options) - .then((res) => res.json()) - .then((json) => { - console.log("Collection created! Collection ID:", json.id); - console.log("Save this Collection ID for the next steps"); - }) - .catch((err) => console.error("Error:", err)); - ``` - - - - - #### Create Template - - **Create an SFT template:** - - ```javascript - const apiKey = "YOUR_API_KEY"; - const collectionId = "YOUR_COLLECTION_ID"; - const env = "staging"; - const url = `https://${env}.crossmint.com/apis/2022-06-09/collections/${collectionId}/templates`; - - const options = { - method: "POST", - headers: { - "accept": "application/json", - "content-type": "application/json", - "x-api-key": apiKey, - }, - body: JSON.stringify({ - name: "lil dumdumz SFT", - description: "A semi-fungible token with image stored on Arweave", - imageUrl: "https://arweave.net/YOUR_ARWEAVE_TX_ID", - attributes: [ - { - trait_type: "Rarity", - value: "Common" - }, - { - trait_type: "Storage", - value: "Arweave" - } - ] - }), - }; - - fetch(url, options) - .then((res) => res.json()) - .then((json) => { - console.log("Template created! Template ID:", json.id); - console.log("Save this Template ID for minting NFTs"); - }) - .catch((err) => console.error("Error:", err)); - ``` - - - - - - - - ### Frontend Development - - Clone and set up the Zero-to-Arweave starter kit: - - - - #### Clone Repository - - **Clone the starter kit:** - - ```bash - git clone https://github.com/ar-io/crossmint-arweave-example.git - cd crossmint-arweave-example - ``` - - - - - #### Install Dependencies - - **Install required packages:** - - ```bash - npm install - # or - yarn install - ``` - - - - - #### Configure Environment - - **Set up your environment variables:** - - Create a `.env` file in the root directory: - - ``` - VITE_CROSSMINT_API_KEY=your_api_key_here - VITE_CROSSMINT_ENV=staging - VITE_COLLECTION_ID=your_collection_id_here - VITE_TEMPLATE_ID=your_template_id_here - ``` - - - - - - - - ### Authentication Integration - - Implement Crossmint's client-side authentication: - - ```javascript - import { CrossmintAuth } from "@crossmint/client-sdk-react"; - - function App() { - const { user, login, logout, isLoading } = CrossmintAuth.useAuth(); - - return ( -
- {user ? ( -
-

Welcome, {user.email}!

- -
- ) : ( - - )} -
- ); - } - ``` - -
- - - ### Payment Integration - - Add Crossmint's embedded checkout for NFT purchases: - - ```javascript - import { CrossmintPaymentButton } from "@crossmint/client-sdk-react"; - - function NFTMinting() { - const handlePaymentSuccess = (result) => { - console.log("Payment successful:", result); - // Handle successful payment - }; - - return ( - - ); - } - ``` - - - - - ### Deploy to Arweave - - Deploy your completed application to Arweave: - - - - #### Build Application - - **Build your React application:** - - ```bash - npm run build - # or - yarn build - ``` - - - - - #### Deploy with ArDrive - - **Deploy using ArDrive:** - - 1. Visit [ArDrive.io](http://ArDrive.io) - 2. Create a new folder for your application - 3. Upload the contents of your `dist` folder - 4. Wait for the upload to complete - - - - - #### Get Manifest ID - - **Retrieve the manifest ID:** - - 1. Click on your uploaded application folder - 2. Look for the "Manifest ID" in the folder details - 3. Copy this ID - you'll need it for domain configuration - - - - - - - - ### Domain Configuration - - Connect your application to a human-readable domain name using ArNS: - - - - #### Purchase ARNS Name - - **Get an ARNS name (if needed):** - - 1. Visit [arns.app](https://arns.app/) - 2. Connect your Arweave wallet - 3. Search for an available name - 4. Purchase it with $ARIO tokens - - - - - #### Get Process ID - - **Get your Process ID:** - - 1. Visit [arns.app](https://arns.app/) - 2. Connect your Arweave wallet - 3. Click "Manage Assets" in the top-right - 4. Find your ARNS name and click on the settings icon - 5. Copy the Process ID displayed - - - - - #### Update Configuration - - **Update the configuration:** - - ```javascript - const ant = ANT.init({ - signer: new ArweaveSigner(jwk), - processId: 'YOUR_PROCESS_ID_HERE' // Replace with your Process ID - }); - - const result = await ant.setRecord({ - name: '@', - ttlSeconds: 900, // 15 minutes - dataLink: 'YOUR_MANIFEST_ID' // Replace with the manifest ID - }); - ``` - - - - - #### Set Base Record - - **Set the base record:** - - ```bash - # Using pnpm - pnpm run set-base - - # Using yarn - yarn set-base - ``` - - When successful, you'll see: - ``` - ✅ Base record update successful! - 🔗 Your application is now available at: https://YOUR-NAME.ar.io - ``` - - - - - -
- -## Advanced Features - - - - ### Custom NFT Metadata - - ```javascript - const customMetadata = { - name: "Custom NFT Name", - description: "A unique NFT with custom attributes", - imageUrl: "https://arweave.net/YOUR_TX_ID", - attributes: [ - { - trait_type: "Rarity", - value: "Legendary" - }, - { - trait_type: "Power", - value: 95 - }, - { - trait_type: "Element", - value: "Fire" - } - ] - }; - ``` - - - - - ### Batch Minting NFTs - - ```javascript - const batchMint = async (templateId, quantity) => { - const promises = Array(quantity).fill().map(() => - mintNFT(templateId) - ); - - const results = await Promise.all(promises); - return results; - }; - ``` - - - - - ### Track Sales and Engagement - - ```javascript - const trackMint = (nftId, userEmail) => { - // Send analytics data - analytics.track('nft_minted', { - nftId, - userEmail, - timestamp: Date.now() - }); - }; - ``` - - - - - ### Comprehensive Error Handling - - ```javascript - const mintWithErrorHandling = async (templateId) => { - try { - const result = await mintNFT(templateId); - return { success: true, data: result }; - } catch (error) { - console.error('Minting failed:', error); - return { - success: false, - error: error.message - }; - } - }; - ``` - - - - -## Benefits of This Approach - -- **True Permanence**: NFT images are stored permanently on Arweave -- **Accessibility**: Credit card payments make NFTs accessible to mainstream users -- **Complete Decentralization**: Both application and assets are stored on decentralized networks -- **User-Friendly Experience**: Seamless experience for both creators and collectors -- **No Server Maintenance**: No need to manage servers or renew domains - -## Ready to Build? - - - } - arrow - > - Get started with the complete example project - - -} -> - Learn more about Crossmint's APIs and features - - - } - > - Understand Arweave's file system for advanced storage - - diff --git a/content/build/guides/depin.mdx b/content/build/guides/depin.mdx index f0ef36821..0fdf3cc59 100644 --- a/content/build/guides/depin.mdx +++ b/content/build/guides/depin.mdx @@ -5,7 +5,7 @@ description: "Complete guide to storing and accessing DePIN network data permane import { Database, Globe, Search, Upload, Tag, Code, Book } from "lucide-react"; -DePIN networks require **scalable and cost-effective storage solutions** they can trust. With vast amounts of data generated by decentralized physical infrastructure networks, traditional on-chain storage is prohibitively expensive, yet networks need reliable, long-term access to their device data. +DePIN networks require **scalable and cost-effective storage solutions** they can trust. With vast amounts of data generated by decentralized physical infrastructure networks, traditional onchain storage is prohibitively expensive, yet networks need reliable, long-term access to their device data. Arweave via ar.io provides **chain-agnostic, permanent and immutable storage** for a one-time fee, ensuring networks can access any device data previously stored and verify it has not been tampered with. @@ -229,7 +229,7 @@ Arweave via ar.io provides **chain-agnostic, permanent and immutable storage** f const results = [] for (const txId of transactionIds) { - const response = await fetch(`https://arweave.net/${txId}`) + const response = await fetch(`https://turbo-gateway.com/${txId}`) const data = await response.json() // Process the data @@ -272,7 +272,7 @@ Arweave via ar.io provides **chain-agnostic, permanent and immutable storage** f fallbackStrategy: new FastestPingRoutingStrategy({ timeoutMs: 500 }), }), verificationStrategy: new HashVerificationStrategy({ - trustedGateways: ['https://arweave.net'], + trustedGateways: ['https://turbo-gateway.com'], }), telemetrySettings: { enabled: true, @@ -332,7 +332,7 @@ In production, teams have several options to take this further to provide signif Operate a gateway optimised to index and serve your device data fast. -}> +}> Create mutable data structures for permanent device data and decentralised apps. diff --git a/content/build/guides/encrypted-data-nillion.mdx b/content/build/guides/encrypted-data-nillion.mdx index e04567d09..609a6289b 100644 --- a/content/build/guides/encrypted-data-nillion.mdx +++ b/content/build/guides/encrypted-data-nillion.mdx @@ -300,7 +300,7 @@ Let's build the application step by step, explaining what each part does: import { logger } from './logger'; import { TurboFactory, ArweaveSigner } from "@ardrive/turbo-sdk"; - const ARWEAVE_HOST = "arweave.net"; + const ARWEAVE_HOST = "turbo-gateway.com"; export async function createWallet() { try { @@ -530,7 +530,7 @@ Let's build the application step by step, explaining what each part does: // Download the file and decrypt it const downloadFileName = `./test/encrypted_demo_${Date.now()}.txt`; console.log(`🕒 Downloading file from Arweave`); - await downloadFile(`https://arweave.net/${upload.id}`, downloadFileName); + await downloadFile(`https://turbo-gateway.com/${upload.id}`, downloadFileName); const decrypted = decryptContent(encrypted, retrievedUserKey.data.private_key, `./test/decrypted_demo_${Date.now()}.txt`); ``` diff --git a/content/build/guides/hosting-decentralised-apps/deploy-permanent-dapp.mdx b/content/build/guides/hosting-decentralised-apps/deploy-permanent-dapp.mdx new file mode 100644 index 000000000..8e3d783ae --- /dev/null +++ b/content/build/guides/hosting-decentralised-apps/deploy-permanent-dapp.mdx @@ -0,0 +1,338 @@ +--- +title: "Deploy a Permanent dApp" +description: "End-to-end guide to deploying a permanent web application using a Solana wallet — upload, register a name, and go live" +--- + +import { Rocket, Globe, Upload, Check } from 'lucide-react'; + +Deploy a permanent web application to Arweave with a human-readable ArNS name — using only your Solana wallet. Your app will be served by the network of decentralized gateways with zero ongoing hosting costs. + +## What You'll Build + +By the end of this guide, your app will be: +- **Permanently stored** on Arweave (can never be deleted or modified) +- **Accessible** at `https://yourname.ar.io` (and every other ar.io gateway) +- **Owned by you** as a Metaplex Core NFT (the ArNS name token) + +## Prerequisites + +- **Node.js 18+** +- **A Solana wallet** with SOL and ARIO tokens +- **A built web app** (any framework that outputs a static folder — React, Next.js, Vue, Svelte, etc.) + +```bash +npm install @ar.io/sdk @ardrive/turbo-sdk @solana/kit bs58 +``` + + +Requires `@ar.io/sdk` version 3.23+ for Solana support. + + +## Step-by-Step + + + + ### Build Your App + + Generate a static build of your web application: + + + + ```bash + # next.config.js must have: output: 'export' + npm run build + # Output: ./out/ + ``` + + + ```bash + npm run build + # Output: ./dist/ + ``` + + + ```bash + # Build your app to a static folder + npm run build + # Use whatever output directory your framework creates + ``` + + + + + + ### Set Up Your Signer + + Create a shared setup file that both Turbo (for uploads) and the ar.io SDK (for naming) can use: + + ```typescript + // setup.ts + import { ARIO, ANT } from '@ar.io/sdk'; + import { TurboFactory } from '@ardrive/turbo-sdk'; + import { createKeyPairSignerFromBytes } from '@solana/kit'; + import bs58 from 'bs58'; + import fs from 'fs'; + + // Load your Solana keypair + const keypairBytes = new Uint8Array( + JSON.parse(fs.readFileSync('./solana-keypair.json', 'utf-8')), + ); + + // For ar.io SDK (ArNS names, records) + export const signer = await createKeyPairSignerFromBytes(keypairBytes); + export const ario = ARIO.mainnet({ signer }); + + // For Turbo (file uploads to Arweave) + // Turbo uses its own signer format — pass the secret key as base58 + export const turbo = TurboFactory.authenticated({ + privateKey: bs58.encode(keypairBytes.slice(0, 32)), + token: 'solana', + }); + ``` + + + **Security:** Never commit your keypair file to version control. Use a dedicated deployment wallet with only the SOL and ARIO needed for the operation. + + + + + ### Upload to Arweave via Turbo + + Upload your build folder. Turbo bundles all files into an Arweave manifest — a single transaction ID that maps to all your app's files: + + ```typescript + // deploy.ts + import { turbo, ario, signer } from './setup'; + import { ANT } from '@ar.io/sdk'; + + const ARNS_NAME = 'my-cool-app'; // the ArNS name you want + const BUILD_DIR = './dist'; // your build output folder + + // Step 1: Upload the build folder + console.log('Uploading to Arweave...'); + const uploadResult = await turbo.uploadFolder({ + folderPath: BUILD_DIR, + dataItemOpts: { + tags: [ + { name: 'App-Name', value: ARNS_NAME }, + { name: 'App-Version', value: '1.0.0' }, + ], + }, + }); + + const manifestTxId = uploadResult.manifestResponse.id; + console.log(`Uploaded! Manifest TX: ${manifestTxId}`); + console.log(`Direct access: https://turbo-gateway.com/${manifestTxId}`); + ``` + + + Files under 100KB are free to upload via Turbo. Larger uploads are paid with SOL from your wallet — no pre-funding needed when using just-in-time payments. + + + + + ### Register an ArNS Name (If You Don't Have One) + + Skip this step if you already own the ArNS name. + + ```typescript + // Check if the name is available + try { + const existing = await ario.getArNSRecord({ name: ARNS_NAME }); + console.log(`Name "${ARNS_NAME}" is already registered`); + } catch { + // Name is available — register it + console.log(`Registering "${ARNS_NAME}"...`); + + // Check the cost first + const cost = await ario.getTokenCost({ + intent: 'Buy-Name', + name: ARNS_NAME, + type: 'lease', + years: 1, + }); + console.log(`Cost: ${cost / 1_000_000} ARIO`); + + // Buy it (this mints an ANT as a Metaplex Core NFT) + await ario.buyRecord({ + name: ARNS_NAME, + type: 'lease', + years: 1, + }); + + console.log(`Registered "${ARNS_NAME}"!`); + } + ``` + + + + ### Point Your Name to Your App + + Set the ANT's root (@) record to your uploaded manifest: + + ```typescript + // Get the ANT mint address from the ArNS record + const record = await ario.getArNSRecord({ name: ARNS_NAME }); + + // Initialize the ANT + const ant = ANT.init({ signer, processId: record.processId }); + + // Set the root record to your manifest + console.log('Setting ArNS record...'); + await ant.setRecord({ + undername: '@', + transactionId: manifestTxId, + ttlSeconds: 3600, + }); + + console.log('Done! Your app is live at:'); + console.log(` https://${ARNS_NAME}.ar.io`); + console.log(` https://${ARNS_NAME}.turbo-gateway.com`); + ``` + + + + ### Verify + + Wait a minute for gateways to pick up the new record, then verify: + + ```bash + curl -I https://my-cool-app.ar.io + # Should return 200 OK with your app's index.html + ``` + + Your app is now permanently hosted and accessible through every ar.io gateway in the network. + + + +## Updating Your App + +Since Arweave data is immutable, "updating" means uploading a new version and updating your ArNS record to point to it: + +```typescript +// Upload new version +const newUpload = await turbo.uploadFolder({ + folderPath: './dist', + dataItemOpts: { + tags: [ + { name: 'App-Name', value: 'my-cool-app' }, + { name: 'App-Version', value: '2.0.0' }, + ], + }, +}); + +// Update the record +const ant = ANT.init({ signer, processId: record.processId }); +await ant.setRecord({ + undername: '@', + transactionId: newUpload.manifestResponse.id, + ttlSeconds: 3600, +}); + +// Old version is still on Arweave forever — instant rollback if needed +``` + +## Using Undernames for Staging + +You can use undernames to deploy staging environments alongside production: + +```typescript +// Deploy staging version +await ant.setRecord({ + undername: 'staging', + transactionId: stagingManifestTxId, + ttlSeconds: 300, // short TTL for faster updates +}); +// Access at: https://staging_my-cool-app.ar.io + +// When ready, promote to production +await ant.setRecord({ + undername: '@', + transactionId: stagingManifestTxId, + ttlSeconds: 3600, +}); +``` + +## Full Script + +Here's the complete deployment script you can adapt: + +```typescript +import { ARIO, ANT } from '@ar.io/sdk'; +import { TurboFactory } from '@ardrive/turbo-sdk'; +import { createKeyPairSignerFromBytes } from '@solana/kit'; +import bs58 from 'bs58'; +import fs from 'fs'; + +const ARNS_NAME = process.argv[2] || 'my-app'; +const BUILD_DIR = process.argv[3] || './dist'; + +async function deploy() { + // Load Solana keypair + const keypairBytes = new Uint8Array( + JSON.parse(fs.readFileSync('./solana-keypair.json', 'utf-8')), + ); + + // Init SDKs + const signer = await createKeyPairSignerFromBytes(keypairBytes); + const ario = ARIO.mainnet({ signer }); + const turbo = TurboFactory.authenticated({ + privateKey: bs58.encode(keypairBytes.slice(0, 32)), + token: 'solana', + }); + + // Upload + console.log(`Uploading ${BUILD_DIR}...`); + const upload = await turbo.uploadFolder({ + folderPath: BUILD_DIR, + dataItemOpts: { + tags: [{ name: 'App-Name', value: ARNS_NAME }], + }, + }); + const txId = upload.manifestResponse.id; + console.log(`Uploaded: https://turbo-gateway.com/${txId}`); + + // Get ANT and update record + const arnsRecord = await ario.getArNSRecord({ name: ARNS_NAME }); + const ant = ANT.init({ signer, processId: arnsRecord.processId }); + + console.log('Updating ArNS record...'); + await ant.setRecord({ + undername: '@', + transactionId: txId, + ttlSeconds: 3600, + }); + + console.log(`\nLive at: https://${ARNS_NAME}.ar.io`); +} + +deploy().catch(console.error); +``` + +```bash +# Usage +npx tsx deploy.ts my-cool-app ./dist +``` + +## Next Steps + + + } + /> + } + /> + } + /> + diff --git a/content/build/guides/hosting-decentralised-apps/deploying-with-arlink.mdx b/content/build/guides/hosting-decentralised-apps/deploying-with-arlink.mdx index 1f187c268..088bae729 100644 --- a/content/build/guides/hosting-decentralised-apps/deploying-with-arlink.mdx +++ b/content/build/guides/hosting-decentralised-apps/deploying-with-arlink.mdx @@ -55,10 +55,10 @@ See [Limitations & Considerations](#limitations--considerations) below to help d Before deploying with Arlink, you'll need: -- **Arweave Wallet** - Create one at [Wander](https://www.wander.app/) and add AR tokens or [Turbo credits](https://turbo.ar.io/topup) +- **Arweave Wallet** - Create one at [Wander](https://www.wander.app/) and add AR tokens or [Turbo credits](https://console.ar.io/topup) - **GitHub Repository** - Your application code in a GitHub repository with build scripts - **Static Build Output** - Application must build to static files (HTML, CSS, JS) -- **Optional: ArNS Name** - Purchase at [arns.app](https://arns.app) or use free Arlink undernames +- **Optional: ArNS Name** - Purchase at [arns.ar.io](https://arns.ar.io) or use free Arlink undernames Test your build locally (`npm run build`) before deploying to ensure it produces static output. @@ -79,7 +79,7 @@ The deployment process consists of four main phases. For detailed step-by-step i ### 1. Connect & Authorize -Navigate to the [Arlink Dashboard](https://arlink.arweave.net/) and connect your Arweave wallet (Wander). Then authorize GitHub access to enable repository connections. +Navigate to the [Arlink Dashboard](https://arlink.ar.io/) and connect your Arweave wallet (Wander). Then authorize GitHub access to enable repository connections. ![Arlink login page showing GitHub, Wander, and MetaMask authentication options](/content/arlink-login.png) @@ -114,13 +114,13 @@ Review the auto-detected settings and adjust if needed. Ensure your output direc Select how your application will be accessible: **Free Arlink Undername:** -- Format: `yourname_arlink.arweave.net` +- Format: `yourname_arlink.ar.io` - No ArNS name purchase required - Available immediately **Existing ArNS Name:** -- Use your purchased ArNS name (e.g., `myapp.arweave.net`) -- Optionally add undernames (e.g., `staging_myapp.arweave.net`) +- Use your purchased ArNS name (e.g., `myapp.ar.io`) +- Optionally add undernames (e.g., `staging_myapp.ar.io`) - Arlink automatically updates ArNS records {/* Screenshot: Domain selection interface showing Arlink undername and existing ArNS options */} @@ -142,7 +142,7 @@ Arlink enforces a **10MB max build output** and **10-minute build timeout**. For Once complete, your application is permanently deployed and accessible via: -- Your chosen domain (e.g., `myapp_arlink.arweave.net`) +- Your chosen domain (e.g., `myapp_arlink.ar.io`) - Any ar.io gateway (e.g., `myapp_arlink.g8way.io`) - Direct transaction ID @@ -156,7 +156,7 @@ Arlink offers two domain options for your deployments: Arlink provides free subdomains under the `arlink` ArNS name: -- **Format**: `yourname_arlink.arweave.net` +- **Format**: `yourname_arlink.ar.io` - **Cost**: Free (no ArNS purchase required) - **Availability**: Instant, accessible via all ar.io gateways - **Limitation**: Must be unique across all Arlink deployments @@ -237,7 +237,7 @@ Choose the right tool for your use case: **ArNS Management:** - Cannot create new ArNS names through Arlink -- Must purchase ArNS names separately at [arns.app](https://arns.app) +- Must purchase ArNS names separately at [arns.ar.io](https://arns.ar.io) - Limited undername configuration options diff --git a/content/build/guides/hosting-decentralised-apps/deploying-with-permaweb-deploy.mdx b/content/build/guides/hosting-decentralised-apps/deploying-with-permaweb-deploy.mdx index ccebca784..f0b5e4600 100644 --- a/content/build/guides/hosting-decentralised-apps/deploying-with-permaweb-deploy.mdx +++ b/content/build/guides/hosting-decentralised-apps/deploying-with-permaweb-deploy.mdx @@ -32,7 +32,7 @@ Before starting, ensure you have: - **Node.js 18+** - Download from [nodejs.org](https://nodejs.org/) - **Arweave or EVM Wallet** - You'll need the private key or JWK file -- **ArNS Name** - Register one at [arns.app](https://arns.app) +- **ArNS Name** - Register one at [arns.ar.io](https://arns.ar.io) - **Command Line Familiarity** - Basic terminal/shell knowledge @@ -166,9 +166,9 @@ For this walkthrough, we'll deploy directly from the command line using inline c - Permaweb-deploy uses [Turbo](https://turbo.ar.io) to upload files to Arweave. Before deploying, ensure your wallet has sufficient credits. + Permaweb-deploy uses [Turbo](https://console.ar.io) to upload files to Arweave. Before deploying, ensure your wallet has sufficient credits. - Visit the [Turbo app](https://turbo.ar.io/topup) and connect your deployment wallet to view your current balance. + Visit the [Console app](https://console.ar.io/topup) and connect your deployment wallet to view your current balance. A typical static site (5-10 MB) costs approximately 0.1-0.5 ARIO. Credits are applied instantly and remain in your wallet for future deployments. diff --git a/content/build/guides/hosting-decentralised-apps/hosting-with-ardrive.mdx b/content/build/guides/hosting-decentralised-apps/hosting-with-ardrive.mdx index c09c6c00c..a9324c23d 100644 --- a/content/build/guides/hosting-decentralised-apps/hosting-with-ardrive.mdx +++ b/content/build/guides/hosting-decentralised-apps/hosting-with-ardrive.mdx @@ -89,7 +89,7 @@ Check out the [series introduction](/build/guides/hosting-decentralised-apps) to Your dApp is now live on the permaweb forever! - - Append the Data TX ID to an Arweave gateway URL: `https://arweave.net/YOUR-TX-ID` + - Append the Data TX ID to a gateway URL: `https://ardrive.net/YOUR-TX-ID` - It may take a few minutes for files to propagate through the network - Once propagated, your dApp is accessible to anyone, anywhere, at any time @@ -98,7 +98,7 @@ Check out the [series introduction](/build/guides/hosting-decentralised-apps) to Make your dApp easier to access with an ArNS name: - If you own an ArNS name, you'll be prompted during manifest creation - - If not, purchase one from [arns.app](https://arns.arweave.net) + - If not, purchase one from [arns.ar.io](https://arns.ar.io) - You can also assign an ArNS name later by clicking the three dots next to any file and selecting "Assign ArNS name" diff --git a/content/build/guides/hosting-decentralised-apps/index.mdx b/content/build/guides/hosting-decentralised-apps/index.mdx index 59348983c..15cde5bca 100644 --- a/content/build/guides/hosting-decentralised-apps/index.mdx +++ b/content/build/guides/hosting-decentralised-apps/index.mdx @@ -18,17 +18,17 @@ The goal of decentralised apps (dapps) has always been full stack. However, unti Ar.io solves this via: - **Permanent Storage**: Pay once and store your site forever on [Arweave](https://arweave.org/). -- **Smart Domains**: a sovereign owned domain-space via [Arweave Name System (ArNS)](/learn/arns). -- **Distributed Gateway Network**: access your site as a subdomain of ~600 active [gateways](/learn/gateways) +- **Smart Domains**: a sovereign owned domain-space via [Ar.io Name System (ArNS)](/learn/arns). +- **Distributed Gateway Network**: access your site as a subdomain of active [gateways](/learn/gateways) For example, this docs portal is a decentralised app: - [docs.ar.io](https://docs.ar.io) -- [docs.arweave.net](https://docs.arweave.net) +- [docs.ardrive.net](https://docs.ardrive.net) Same app, different gateway. -You can repeat this by loading the `docs` subdomain for any of the ~600 active gateways in the network. +You can repeat this by loading the `docs` subdomain for any of the active gateways in the network. ## What You'll Learn @@ -37,6 +37,12 @@ In this guide series, you'll learn how to deploy and manage permanent applicatio We'll cover the following: + } + /> } /> + } + /> } /> diff --git a/content/build/guides/hosting-decentralised-apps/meta.json b/content/build/guides/hosting-decentralised-apps/meta.json index 9e9fda8cc..3ca26b6f7 100644 --- a/content/build/guides/hosting-decentralised-apps/meta.json +++ b/content/build/guides/hosting-decentralised-apps/meta.json @@ -2,6 +2,8 @@ "title": "Hosting Decentralised Apps", "defaultOpen": false, "pages": [ + "migrating-your-app-to-new-sdks", + "deploy-permanent-dapp", "deploying-with-permaweb-deploy", "using-undernames-for-versioning", "deploying-with-arlink", diff --git a/content/build/guides/hosting-decentralised-apps/migrating-your-app-to-new-sdks.mdx b/content/build/guides/hosting-decentralised-apps/migrating-your-app-to-new-sdks.mdx new file mode 100644 index 000000000..2997b2252 --- /dev/null +++ b/content/build/guides/hosting-decentralised-apps/migrating-your-app-to-new-sdks.mdx @@ -0,0 +1,32 @@ +--- +title: "Migrating Your App to the New SDKs" +description: "Placeholder guide for updating applications to the current ar.io SDKs and Solana-based protocol interactions." +--- + +## Overview + +This guide will help developers migrate applications that interact with ar.io protocol features to the current SDKs and Solana-based workflows. + +For now, use this page as a placeholder while final SDK examples and migration details are reviewed. + +## What This Guide Will Cover + +- Updating to the latest `@ar.io/sdk` version +- Replacing older protocol interaction patterns with current Solana signer flows +- Updating ArNS name registration and record-management code +- Reviewing ANT handling for Metaplex Core NFT ownership and control +- Separating Arweave data-upload wallets from Solana protocol wallets where needed +- Verifying application behavior against devnet or staging examples + +## Current References + +These pages contain related material that should inform the final migration guide: + +- [ar.io SDK](/sdks/ar-io-sdk) +- [Registering ArNS Names Programmatically](/build/guides/working-with-arns/register-arns-programmatically) +- [Setting ArNS Records Programmatically](/build/guides/working-with-arns/set-arns-records-programmatically) +- [Deploy a Permanent dApp](/build/guides/hosting-decentralised-apps/deploy-permanent-dapp) + +## Review Status + +This page needs technical review before launch. Final content should be checked against the latest `@ar.io/sdk` release, generated SDK docs, Solana signer examples, and current ArNS/ANT behavior. diff --git a/content/build/guides/hosting-decentralised-apps/using-undernames-for-versioning.mdx b/content/build/guides/hosting-decentralised-apps/using-undernames-for-versioning.mdx index ec114dec4..36e91dd10 100644 --- a/content/build/guides/hosting-decentralised-apps/using-undernames-for-versioning.mdx +++ b/content/build/guides/hosting-decentralised-apps/using-undernames-for-versioning.mdx @@ -9,7 +9,7 @@ import { Card, Cards } from 'fumadocs-ui/components/card'; import { Tab, Tabs } from 'fumadocs-ui/components/tabs'; import { GitBranch, Code, Rocket, Network, Settings, RefreshCw, CheckCircle, Zap, Upload } from 'lucide-react'; -In the [previous guide](/build/guides/hosting-decentralised-apps/deploying-with-permaweb-deploy), you deployed your application to your base ArNS name. Now you'll learn how to manage multiple versions and environments using **undernames** - subdomains under your ArNS name.` +In the [previous guide](/build/guides/hosting-decentralised-apps/deploying-with-permaweb-deploy), you deployed your application to your base ArNS name. Now you'll learn how to manage multiple versions and environments using **undernames** - subdomains under your ArNS name. ## What You'll Learn @@ -27,10 +27,10 @@ Undernames let you create multiple versions under one ArNS name - like subdomain ``` your-arns-name (your ArNS name) -├─ @ (base) → your-arns-name.arweave.net -├─ dev → dev_your-arns-name.arweave.net -├─ staging → staging_your-arns-name.arweave.net -└─ v2 → v2_your-arns-name.arweave.net +├─ @ (base) → your-arns-name.ar.io +├─ dev → dev_your-arns-name.ar.io +├─ staging → staging_your-arns-name.ar.io +└─ v2 → v2_your-arns-name.ar.io ``` **Key benefits:** @@ -134,7 +134,7 @@ Create a script that automatically archives based on your `package.json` version { stdio: 'inherit' } ); - console.log(`✓ Archived at: https://v${version}_your-arns-name.arweave.net`); + console.log(`✓ Archived at: https://v${version}_your-arns-name.ar.io`); ``` Then add to `package.json`: @@ -167,7 +167,7 @@ Create a script that automatically archives based on your `package.json` version { stdio: 'inherit' } ); - console.log(`✓ Archived at: https://v${version}_your-arns-name.arweave.net`); + console.log(`✓ Archived at: https://v${version}_your-arns-name.ar.io`); ``` Then add to `package.json`: @@ -185,9 +185,9 @@ Create a script that automatically archives based on your `package.json` version -Now whenever you run `deploy:archive` a version specific immutable deployment of your app we be create and hosted permanently. +Now whenever you run `deploy:archive` a version-specific immutable deployment of your app will be created and hosted permanently. -For example, version 2.1.0 would be accessible forever at `https://v2-1-0_your-arns-name.arweave.net`. +For example, version 2.1.0 would be accessible forever at `https://v2-1-0_your-arns-name.ar.io`. --- @@ -258,7 +258,7 @@ Building on the [GitHub Actions workflow](/build/guides/hosting-decentralised-ap DEPLOY_KEY: ${{ secrets.DEPLOY_KEY }} ``` - **Usage:** Push to `develop` branch → Auto-deploys to `dev_your-arns-name.arweave.net` + **Usage:** Push to `develop` branch → Auto-deploys to `dev_your-arns-name.ar.io` @@ -293,7 +293,7 @@ Building on the [GitHub Actions workflow](/build/guides/hosting-decentralised-ap DEPLOY_KEY: ${{ secrets.DEPLOY_KEY }} ``` - **Usage:** Push to `staging` branch → Auto-deploys to `staging_your-arns-name.arweave.net` + **Usage:** Push to `staging` branch → Auto-deploys to `staging_your-arns-name.ar.io` @@ -334,10 +334,10 @@ Building on the [GitHub Actions workflow](/build/guides/hosting-decentralised-ap DEPLOY_KEY: ${{ secrets.DEPLOY_KEY }} ``` - **Usage:** Push to `main` branch → Auto-deploys to `your-arns-name.arweave.net` and archives to `v{version}_your-arns-name.arweave.net` + **Usage:** Push to `main` branch → Auto-deploys to `your-arns-name.ar.io` and archives to `v{version}_your-arns-name.ar.io` - Every production deployment automatically creates a permanent archive based on your `package.json` version. For example, version 2.1.0 is archived at `v2-1-0_your-arns-name.arweave.net`. + Every production deployment automatically creates a permanent archive based on your `package.json` version. For example, version 2.1.0 is archived at `v2-1-0_your-arns-name.ar.io`. @@ -366,13 +366,13 @@ git push origin main # Auto-deploys to production ## Instant Rollbacks with ArNS App -One of the key benefits of undernames is instant rollbacks. Since all versions are permanently stored on Arweave, you can instantly switch between them using the [ArNS app](https://arns.app). +One of the key benefits of undernames is instant rollbacks. Since all versions are permanently stored on Arweave, you can instantly switch between them using the [ArNS app](https://arns.ar.io). **To rollback to a previous version:** - Navigate to [arns.app](https://arns.app) and find your ArNS name. You'll see all undernames and their transaction IDs. + Navigate to [arns.ar.io](https://arns.ar.io) and find your ArNS name. You'll see all undernames and their transaction IDs. diff --git a/content/build/guides/index.mdx b/content/build/guides/index.mdx index 4c15d1268..a4b55064f 100644 --- a/content/build/guides/index.mdx +++ b/content/build/guides/index.mdx @@ -67,7 +67,7 @@ Explore real-world applications and use cases for **Arweave** and **ar.io** infr **Trade and sell** ArNS tokens and digital assets **Key topics:** - - Arweave Name Token (ANT) trading + - Ar.io Name Token (ANT) trading - Marketplace dynamics - Asset ownership @@ -105,6 +105,17 @@ Explore real-world applications and use cases for **Arweave** and **ar.io** infr - Cost optimization with compression + + + **Move large IPFS pin sets** to permanent Arweave storage + + **Key topics:** + - Custom Turbo-based migration scripts + - CID-to-transaction mapping and provenance tags + - Batch migration at scale (100k+ CIDs) + - NFT metadata rewrite patterns + + ## Why Use Arweave? diff --git a/content/build/guides/meta.json b/content/build/guides/meta.json index 7657a2693..4a396e5db 100644 --- a/content/build/guides/meta.json +++ b/content/build/guides/meta.json @@ -3,13 +3,13 @@ "icon": "Book", "defaultOpen": false, "pages": [ - "application-distribution", - "crossmint-nft-minting-app", - "depin", - "encrypted-data-nillion", "hosting-decentralised-apps", - "storing-nfts", + "working-with-arns", "using-turbo-in-a-browser", - "working-with-arns" + "application-distribution", + "storing-nfts", + "migrate-from-ipfs-to-arweave", + "depin", + "encrypted-data-nillion" ] } diff --git a/content/build/guides/migrate-from-ipfs-to-arweave.mdx b/content/build/guides/migrate-from-ipfs-to-arweave.mdx new file mode 100644 index 000000000..576ab4d9a --- /dev/null +++ b/content/build/guides/migrate-from-ipfs-to-arweave.mdx @@ -0,0 +1,553 @@ +--- +title: "Migrate from IPFS to Arweave" +description: "A practical playbook for migrating large IPFS pin sets to permanent Arweave storage using Turbo" +--- + +import { Callout } from "fumadocs-ui/components/callout"; +import { Steps, Step } from "fumadocs-ui/components/steps"; +import { Card, Cards } from "fumadocs-ui/components/card"; +import { Database, Upload, Tag, Shield, Globe, FileText } from "lucide-react"; + +If you have thousands, or hundreds of thousands, of IPFS pins and want **permanent, pay-once storage**, Arweave via ar.io is a strong fit. Unlike pinning services that require ongoing fees, data uploaded to Arweave is stored permanently with a single upfront payment. + +This guide walks through a **custom programmatic migration** using the [Turbo SDK](/sdks/turbo-sdk). It applies to any file set pinned on IPFS, with NFT collections as a concrete example. + +## Why Use a Custom Migration? + +IPFS migrations are usually project-specific. Your CIDs may represent raw files, directories, NFT assets, metadata JSON, application bundles, or a mix of all of them. You may also need custom tagging, retry behavior, validation rules, metadata rewrites, or contract-specific URL formats. + +For that reason, a custom script is usually the most flexible approach. With modern LLM-assisted development, generating and adapting a migration script for your exact CID inventory, metadata schema, and validation requirements is often faster than forcing a generic tool to fit your project. + +Useful docs to combine for a custom migration: + +- [Advanced Uploading with Turbo](/build/upload/advanced-uploading-with-turbo): authentication, uploads, and payment +- [Tagging](/build/upload/tagging): metadata and discoverability +- [Manifests](/build/upload/manifests): organizing collections under path-based routing +- [Receipts](/build/upload/receipts): audit trail for uploads +- [Storing NFTs on ar.io](/build/guides/storing-nfts): NFT asset and metadata upload patterns +- ArDrive CLI also supports IPFS CID tagging for some workflows, but most large migrations benefit from a custom Turbo SDK script so you can control batching, retries, metadata rewrites, and validation. + +This guide connects those pieces into a migration workflow. + +## When to Use a Custom Migration + +| Scenario | Recommended approach | +| --- | --- | +| A few dozen files | Manual upload via [Turbo SDK](/sdks/turbo-sdk) or [console.ar.io](https://console.ar.io) | +| Hundreds to millions of CIDs | Custom migration script (this guide) | +| NFT collection with linked metadata | Custom script + metadata rewrite (see [NFT example](#nft-collection-example) below) | +| Ongoing ingestion pipeline | Build a reusable worker that fetches from IPFS and uploads to Turbo | + + + For full control over tagging, provenance, and reference rewriting, a custom script is usually the most flexible option. + + +## Migration Architecture + +At a high level, every migration follows the same pipeline: + +```mermaid +flowchart LR + inventory["Inventory CIDs"] --> fetch["Fetch from IPFS gateway"] + fetch --> upload["Upload via Turbo"] + upload --> map["Persist CID to txId mapping"] + map --> rewrite["Rewrite references"] + rewrite --> index["Publish migration index"] + index --> validate["Validate on ar.io"] +``` + + + + ### Inventory your CIDs + + Export your pin list from your pinning provider (Pinata, NFT.Storage, a self-hosted node, etc.) into a simple format: + + ```json + [ + { "cid": "bafybeig...", "path": "images/0.png", "contentType": "image/png" }, + { "cid": "bafkreif...", "path": "metadata/0.json", "contentType": "application/json" } + ] + ``` + + Or use a newline-delimited file of CIDs if you have no path metadata: + + ``` + bafybeig... + bafkreif... + ``` + + Deduplicate CIDs before starting. For file CIDs, the same CID should resolve to the same content bytes, so you only need to upload each unique file CID once. + + + + ### Fetch bytes from IPFS + + Retrieve each CID through a reliable IPFS gateway or your own node: + + ```javascript + const IPFS_GATEWAY = 'https://ipfs.io/ipfs'; + + async function fetchFromIpfs(cid) { + const response = await fetch(`${IPFS_GATEWAY}/${cid}`, { + redirect: 'follow', + }); + + if (!response.ok) { + throw new Error(`Failed to fetch ${cid}: ${response.status}`); + } + + const buffer = Buffer.from(await response.arrayBuffer()); + const contentType = + response.headers.get('content-type') ?? 'application/octet-stream'; + + return { buffer, contentType }; + } + ``` + + + **Gateway reliability:** Public gateways can rate-limit or go offline. For large migrations, use your own IPFS node or a dedicated gateway from your pinning provider. Retry failed fetches with exponential backoff. + + + + + ### Upload to Arweave via Turbo + + Authenticate with the Turbo SDK and upload each file. Tag every upload with provenance metadata so you can trace it back to its IPFS origin. + + ```javascript + import { TurboFactory } from '@ardrive/turbo-sdk'; + import fs from 'fs'; + + const jwk = JSON.parse(fs.readFileSync('./wallet.json', 'utf-8')); + const turbo = TurboFactory.authenticated({ + privateKey: jwk, + token: 'arweave', + }); + + async function uploadToArweave(cid, buffer, contentType, projectTag) { + const result = await turbo.upload({ + data: buffer, + dataItemOpts: { + tags: [ + { name: 'Content-Type', value: contentType }, + { name: 'App-Name', value: projectTag }, + { name: 'Source-Protocol', value: 'ipfs' }, + { name: 'Source-CID', value: cid }, + ], + }, + }); + + return result.id; + } + ``` + + See [Tagging best practices](#tagging-for-provenance) below for recommended tags. + + + + ### Persist the CID-to-txId mapping + + Write results to an append-only state file after every upload. This is your migration ledger and lets you resume after failures. + + ```json + { + "cid": "bafybeig...", + "txId": "Xj9k2Lm8Pq3Rn5Tv7Wz1Yb4Dc6Fg8Hj0Kl2Mn4Pq6Rs8", + "contentType": "image/png", + "bytes": 48291, + "status": "success", + "uploadedAt": "2026-05-27T12:00:00Z" + } + ``` + + Store this mapping durably, on disk at minimum, and ideally backed up. You will need it to rewrite references and to audit the migration. + + + + ### Rewrite references + + Update any documents, metadata, or application configs that point at `ipfs://` URIs to use Arweave references instead. For broad compatibility today, use gateway URLs in fields that external platforms must fetch immediately, and keep `ar://` references where your application, contract, or metadata consumers support them. The mapping file from the previous step drives this rewrite pass. + + + + ### Publish a migration index + + After the main upload pass, create a JSON index that links every original CID to its Arweave transaction ID. This gives your team and downstream users a durable lookup table for audits, support, and future migrations. + + + + ### Validate + + Fetch each uploaded transaction from an ar.io gateway or via [Wayfinder](/build/access/wayfinder) and compare it with what you fetched from IPFS. Retain [Turbo receipts](/build/upload/receipts) for audit purposes. + + + +## Running Migrations at Scale + +For collections in the **tens or hundreds of thousands**, treat the migration as a long-running batch job. + +### Cost estimation + +Before starting, estimate total upload cost: + +1. Sum the byte size of all unique CIDs in your inventory. +2. Use the [Turbo pricing calculator](https://turbo.ardrive.net) or `turbo.getFiatEstimateForBytes()` from the SDK. +3. Purchase sufficient [Turbo Credits](/build/upload/turbo-credits) before beginning. + + + Uploads under **100 KiB are free** and do not require a prior top-up. For large migrations this is negligible, but worth knowing for small metadata files. + + +### Concurrency and rate limits + +- Start with **low concurrency** (3–5 parallel uploads) and increase gradually while monitoring for errors. +- IPFS gateways and Turbo both have rate limits. Separate fetch concurrency from upload concurrency. +- Use exponential backoff on transient failures (HTTP 429, 502, network timeouts). + +### Checkpointing and resume + +- Skip CIDs that already appear in your state file with `status: "success"`. +- Write state **after each successful upload**, not in batches. If the process crashes, you lose at most one item. +- Log failures separately so you can retry them in a second pass. + +### Handling edge cases + +| Issue | What to do | +| --- | --- | +| Duplicate CIDs | Upload once, reuse the same txId in your mapping | +| Missing/unavailable CID | Log as failed, retry later; do not block the entire run | +| Unknown content type | Default to `application/octet-stream`; inspect bytes if needed | +| Very large files | Stream via `turbo.uploadFile` with `fileStreamFactory` instead of buffering entirely in memory | +| DagPB / directory CIDs | Resolve to individual file CIDs first; directory CIDs are not uploadable as a single blob | + +## Tagging for Provenance + +Every migrated upload should include tags that make the data discoverable and traceable: + +| Tag | Purpose | +| --- | --- | +| `Content-Type` | Required: tells gateways how to serve the data | +| `App-Name` | Identifies your project (e.g. `MyCollection-Migration-v1`) | +| `Source-Protocol` | Set to `ipfs` to mark migrated content | +| `Source-CID` | The original IPFS CID, useful for provenance and GraphQL queries | +| `Migration-Date` | ISO timestamp of when the upload occurred | +| `Collection-Name` | Optional: groups uploads from the same project | + + + **Why tag the original CID?** Storing the source CID on each Arweave transaction lets you query your migrated data via [GraphQL](/build/access/find-data), correlate uploads back to IPFS origins, and support provenance checks without relying on an external mapping file alone. + + + + **IPFS-specific tags:** Some ArDrive CLI workflows support adding an `IPFS-Add` tag to public uploads, which may be useful where bridge-aware infrastructure recognizes that tag. For large custom migrations, the Turbo SDK pattern above is usually a better fit because you can choose your own tag names, preserve richer migration state, and adapt the script to your data model. + + +See the full [Tagging guide](/build/upload/tagging) for tag size limits and best practices. + +## Publish a Migration Index + +Your append-only state file is operational state. Once migration is complete, publish a clean migration index that others can use without reading your job logs. + +```json +{ + "type": "ipfs-to-arweave-migration", + "version": "1.0.0", + "project": "MyProject", + "createdAt": "2026-05-27T12:00:00Z", + "items": [ + { + "cid": "bafybeig...", + "txId": "Xj9k2Lm8Pq3Rn5Tv7Wz1Yb4Dc6Fg8Hj0Kl2Mn4Pq6Rs8", + "gatewayUrl": "https://arweave.net/Xj9k2Lm8Pq3Rn5Tv7Wz1Yb4Dc6Fg8Hj0Kl2Mn4Pq6Rs8", + "arUri": "ar://Xj9k2Lm8Pq3Rn5Tv7Wz1Yb4Dc6Fg8Hj0Kl2Mn4Pq6Rs8", + "contentType": "image/png", + "bytes": 48291 + } + ] +} +``` + +Upload the index with Turbo and tag it so it can be discovered later: + +```javascript +const migrationIndex = JSON.parse(fs.readFileSync('./migration-index.json', 'utf-8')); + +const indexUpload = await turbo.upload({ + data: Buffer.from(JSON.stringify(migrationIndex)), + dataItemOpts: { + tags: [ + { name: 'Content-Type', value: 'application/json' }, + { name: 'App-Name', value: PROJECT_TAG }, + { name: 'Data-Type', value: 'IPFS-Migration-Index' }, + { name: 'Source-Protocol', value: 'ipfs' }, + ], + }, +}); + +console.log(`Migration index: https://arweave.net/${indexUpload.id}`); +console.log(`Migration index: ar://${indexUpload.id}`); +``` + +## Complete Script Skeleton + +This skeleton ties the pipeline together. Extend it with your own retry logic, logging, and concurrency controls. + + + **Memory usage:** This example buffers each fetched CID in memory before uploading. That keeps the script compact, but it is best for small or medium-sized files. For large media files, stream to disk first and upload with `turbo.uploadFile` so concurrent workers do not hold many large buffers in memory. + + +```javascript +import { TurboFactory } from '@ardrive/turbo-sdk'; +import fs from 'fs'; +import path from 'path'; + +const IPFS_GATEWAY = process.env.IPFS_GATEWAY ?? 'https://ipfs.io/ipfs'; +const STATE_FILE = './migration-state.jsonl'; +const CONCURRENCY = 5; +const PROJECT_TAG = 'MyProject-Migration-v1'; + +const jwk = JSON.parse(fs.readFileSync('./wallet.json', 'utf-8')); +const turbo = TurboFactory.authenticated({ privateKey: jwk, token: 'arweave' }); + +const inventory = JSON.parse(fs.readFileSync('./cids.json', 'utf-8')); +const completed = new Set( + fs.existsSync(STATE_FILE) + ? fs.readFileSync(STATE_FILE, 'utf-8') + .trim() + .split('\n') + .filter(Boolean) + .map((line) => JSON.parse(line)) + .filter((r) => r.status === 'success') + .map((r) => r.cid) + : [], +); + +function appendState(record) { + fs.appendFileSync(STATE_FILE, JSON.stringify(record) + '\n'); +} + +async function fetchFromIpfs(cid) { + const response = await fetch(`${IPFS_GATEWAY}/${cid}`, { redirect: 'follow' }); + if (!response.ok) throw new Error(`Fetch failed: ${response.status}`); + const buffer = Buffer.from(await response.arrayBuffer()); + const contentType = response.headers.get('content-type') ?? 'application/octet-stream'; + return { buffer, contentType }; +} + +async function migrateOne({ cid, contentType: declaredType }) { + if (completed.has(cid)) { + console.log(`Skipping ${cid} (already migrated)`); + return; + } + + try { + const { buffer, contentType } = await fetchFromIpfs(cid); + const result = await turbo.upload({ + data: buffer, + dataItemOpts: { + tags: [ + { name: 'Content-Type', value: declaredType ?? contentType }, + { name: 'App-Name', value: PROJECT_TAG }, + { name: 'Source-Protocol', value: 'ipfs' }, + { name: 'Source-CID', value: cid }, + { name: 'Migration-Date', value: new Date().toISOString() }, + ], + }, + }); + + appendState({ + cid, + txId: result.id, + contentType: declaredType ?? contentType, + bytes: buffer.length, + status: 'success', + uploadedAt: new Date().toISOString(), + }); + + console.log(`Migrated ${cid} → ar://${result.id}`); + } catch (error) { + appendState({ + cid, + status: 'failed', + error: error.message, + failedAt: new Date().toISOString(), + }); + console.error(`Failed ${cid}:`, error.message); + } +} + +async function runPool(items, fn, concurrency) { + const queue = [...items]; + const workers = Array.from({ length: concurrency }, async () => { + while (queue.length > 0) { + const item = queue.shift(); + if (item) await fn(item); + } + }); + await Promise.all(workers); +} + +await runPool(inventory, migrateOne, CONCURRENCY); +console.log('Migration complete. Review migration-state.jsonl for results.'); +``` + +## NFT Collection Example + +NFT collections are a common migration target because metadata JSON files typically reference images via `ipfs://` URIs. The workflow has an extra rewrite step. + + + + ### Separate assets from metadata + + Split your inventory into two groups: + + - **Assets**: images, animations, and other media files + - **Metadata**: JSON files containing `name`, `description`, `image`, `attributes`, etc. + + Upload assets first so you have transaction IDs to reference in metadata. + + + + ### Upload assets and build the mapping + + Run the migration script on all asset CIDs. Your state file now maps each image CID to an Arweave transaction ID. + + + + ### Rewrite metadata references + + For each metadata JSON, replace `ipfs://` URIs with Arweave references using your mapping. Because `ar://` is not yet universally supported by NFT marketplaces and wallets, the most robust current approach is to use gateway URLs for widely consumed fields like `image`, while also keeping `ar://` values in additional fields for applications that support them. + + ```javascript + const GATEWAY_URL = 'https://arweave.net'; + + function getArweaveReferences(uri, cidToTxId) { + if (!uri.startsWith('ipfs://')) return uri; + + const cid = uri.replace('ipfs://', '').split('/')[0]; + const txId = cidToTxId[cid]; + + if (!txId) { + console.warn(`No mapping for CID: ${cid}`); + return uri; + } + + return { + gatewayUrl: `${GATEWAY_URL}/${txId}`, + arUri: `ar://${txId}`, + }; + } + + function rewriteMetadata(metadata, cidToTxId) { + const rewritten = { ...metadata }; + + for (const field of ['image', 'animation_url', 'external_url']) { + if (rewritten[field]) { + const references = getArweaveReferences(rewritten[field], cidToTxId); + + if (typeof references === 'string') continue; + + rewritten[field] = references.gatewayUrl; + rewritten[`${field}_ar`] = references.arUri; + } + } + + return rewritten; + } + ``` + + + **Use both where practical.** Gateway URLs have the broadest compatibility today. `ar://` URIs are more future-proof and can resolve through [Wayfinder](/build/access/wayfinder), but adoption is still growing. Keeping both gives downstream consumers a stable HTTP URL now and a protocol-native reference for clients that support it. + + + + + ### Upload rewritten metadata + + Upload each rewritten metadata JSON via Turbo. Tag with the same provenance tags plus the token identifier if applicable. + + For large collections (100+ tokens), consider uploading metadata as a [manifest](/build/upload/manifests) so your smart contract can use a single base URI. Use an HTTP gateway base URI when you need maximum marketplace compatibility, and keep the `ar://` manifest URI documented for clients that support it: + + ```solidity + string private constant MANIFEST_ID = "your-manifest-transaction-id"; + string private constant GATEWAY = "https://arweave.net/"; + + function tokenURI(uint256 tokenId) public view returns (string memory) { + return string(abi.encodePacked(GATEWAY, MANIFEST_ID, "/", tokenId.toString(), ".json")); + } + ``` + + See [Storing NFTs on ar.io](/build/guides/storing-nfts) for the full manifest workflow. + + + + ### Update onchain references + + If NFTs are already minted with `ipfs://` token URIs, updating onchain metadata requires a contract-specific approach (owner update functions, redeployment, or a new base URI if your contract supports it). Plan this step before migrating. + + + +## Validation + +After migration, verify a sample of uploads and ideally all failed items from your retry pass. A size check is a useful first pass; for stronger validation, compare hashes or the full buffers: + +```javascript +async function validateUpload(txId, expectedBuffer) { + const response = await fetch(`https://turbo-gateway.com/${txId}`); + if (!response.ok) throw new Error(`Gateway fetch failed: ${response.status}`); + + const actual = Buffer.from(await response.arrayBuffer()); + + if (actual.length !== expectedBuffer.length) { + throw new Error(`Size mismatch: expected ${expectedBuffer.length}, got ${actual.length}`); + } + + if (!actual.equals(expectedBuffer)) { + throw new Error(`Content mismatch for ${txId}`); + } + + console.log(`Validated ar://${txId} (${actual.length} bytes)`); +} +``` + +For production workloads, [Wayfinder](/build/access/wayfinder) can add cryptographic verification for clients that support it. + +## Next Steps + + + } + /> + } + /> + } + /> + } + /> + } + /> + } + /> + diff --git a/content/build/guides/using-turbo-in-a-browser/html.mdx b/content/build/guides/using-turbo-in-a-browser/html.mdx index deb44e74c..aff2873ea 100644 --- a/content/build/guides/using-turbo-in-a-browser/html.mdx +++ b/content/build/guides/using-turbo-in-a-browser/html.mdx @@ -341,7 +341,7 @@ This guide demonstrates how to integrate the `@ardrive/turbo-sdk` directly into statusDiv.innerHTML = `
Wander wallet is not installed!
- Install Wander Wallet + Install Wander Wallet
`; return; @@ -476,9 +476,9 @@ This guide demonstrates how to integrate the `@ardrive/turbo-sdk` directly into Timestamp: ${new Date( result.timestamp ).toLocaleString()}
- View File: arweave.net/${result.id}
+ }" target="_blank">turbo-gateway.com/${result.id}
Explorer: ViewBlock @@ -521,7 +521,7 @@ This guide demonstrates how to integrate the `@ardrive/turbo-sdk` directly into content=" default-src 'self'; script-src 'self' 'unsafe-inline' https://esm.sh https://unpkg.com https://cdn.jsdelivr.net; - connect-src 'self' https://upload.ardrive.io https://payment.ardrive.io https://arweave.net; + connect-src 'self' https://upload.ardrive.io https://payment.ardrive.io https://turbo-gateway.com; img-src 'self' data:; " /> @@ -730,8 +730,8 @@ const result = await turboClient.uploadFile({...}); winc: "0", version: "0.2.0", deadlineHeight: 1708455, - dataCaches: ["arweave.net"], - fastFinalityIndexes: ["arweave.net"], + dataCaches: ["turbo-gateway.com"], + fastFinalityIndexes: ["turbo-gateway.com"], public: "wZgP9Rpfh0nXb5EdzBUg7y2LFMkBp2ADX3gdZhLXHYbz...", signature: "k31SCZwxerhuEojUZ7rnLMr0T6CwgVp1_dKDZc7UdvV...", owner: "cF0H0SKdnaDTqWKY9iJKBktTpdEWgb3GnlndE7ABv0Q" diff --git a/content/build/guides/working-with-arns/arns-primary-names.mdx b/content/build/guides/working-with-arns/arns-primary-names.mdx index e6d70cfe6..a2f724fae 100644 --- a/content/build/guides/working-with-arns/arns-primary-names.mdx +++ b/content/build/guides/working-with-arns/arns-primary-names.mdx @@ -17,6 +17,7 @@ Create **web3 identity** using ArNS names. Primary names allow you to use human- - **Require ownership** - Only the owner of an ArNS name can set it as their primary name - **Enable secure verification** - Ownership requirement ensures identity authenticity - **Work across gateways** - Accessible from any ar.io gateway +- **Are unique** - A name cannot be the Primary Name for more than one wallet ## How It Works @@ -25,10 +26,12 @@ Create **web3 identity** using ArNS names. Primary names allow you to use human- **Register a primary name:** - Choose a unique name (e.g., `jonniesparkles`) -- Pay the registration fee +- Pay the Primary Name fee - Link the name to your wallet address - Use as your web3 identity +The Primary Name fee is equivalent to a single undername purchase on a 51-character name of the same purchase type, adjusted by the current Demand Factor. + ### 2. Bidirectional Resolution **Name to address resolution:** @@ -106,12 +109,19 @@ console.log(nameData.name); // e.g., "jonniesparkles" - Enhance user experience with human-readable identifiers - **Trust identity ownership** - Only name owners can set primary names, ensuring secure verification +**Ownership controls:** + +- A wallet can have one Primary Name at a time +- A name can only be the Primary Name for one wallet +- The base name's ANT owner can remove any Primary Name set on one of its undernames + ## Benefits - **Web3 identity** - Use human-readable names as your identity - **Easy discovery** - Others can find you by name instead of wallet address - **Bidirectional resolution** - Resolve name to address or address to name - **Secure verification** - Only name owners can set primary names, preventing impersonation +- **Network uniqueness** - A name can identify only one wallet as its Primary Name - **Permanent ownership** - Own your identity forever - **App integration** - Works in any app that supports primary names diff --git a/content/build/guides/working-with-arns/index.mdx b/content/build/guides/working-with-arns/index.mdx index 4dc664d06..cb655e996 100644 --- a/content/build/guides/working-with-arns/index.mdx +++ b/content/build/guides/working-with-arns/index.mdx @@ -1,12 +1,12 @@ --- title: "Working with ArNS" -description: "Complete guide to managing Arweave Name System (ArNS) names and records" +description: "Complete guide to managing Ar.io Name System (ArNS) names and records" --- import { Card, Cards } from "fumadocs-ui/components/card"; import { Globe, Code, Book, Settings, ShoppingCart } from "lucide-react"; -The **Arweave Name System (ArNS)** provides human-readable names for Arweave content, making it easy to access and manage your permanent websites and applications. Learn how to register, manage, and use ArNS names effectively. +The **Ar.io Name System (ArNS)** provides human-readable names for Arweave content, making it easy to access and manage your permanent websites and applications. Learn how to register, manage, and use ArNS names effectively. ## What is ArNS? @@ -16,7 +16,7 @@ ArNS is a decentralized naming system that allows you to: - **Point names to content** stored on Arweave for permanent access - **Create subdomains** (undernames) for organizing different versions and components - **Establish web3 identity** with primary names linked to wallet addresses -- **Transfer ownership** of names through the ANT (Arweave Name Token) system +- **Transfer ownership** of names through the ANT (Ar.io Name Token) system — ANTs are Metaplex Core NFTs tradeable on Solana marketplaces ## Getting Started @@ -24,7 +24,7 @@ ArNS is a decentralized naming system that allows you to: } title="Purchase an ArNS Name" - description="Learn how to register and purchase ArNS names using the arns.app interface" + description="Learn how to register and purchase ArNS names using the arns.ar.io interface" href="/build/guides/working-with-arns/purchase-arns-ui" /> - Visit [arns.app](https://arns.app) in your browser and connect your wallet using the "Connect" button at the top right of the screen. + Visit [arns.ar.io](https://arns.ar.io) in your browser and connect your wallet using the "Connect" button at the top right of the screen. Connect Wallet diff --git a/content/build/guides/working-with-arns/purchase-arns-ui.mdx b/content/build/guides/working-with-arns/purchase-arns-ui.mdx index 121b113a3..825940aac 100644 --- a/content/build/guides/working-with-arns/purchase-arns-ui.mdx +++ b/content/build/guides/working-with-arns/purchase-arns-ui.mdx @@ -1,18 +1,18 @@ --- -title: "Purchasing an ArNS name using arns.app" -description: "Purchase an ArNS name using the arns.app UI" +title: "Purchasing an ArNS name using arns.ar.io" +description: "Purchase an ArNS name using the arns.ar.io UI" --- import { Card, Cards } from "fumadocs-ui/components/card"; import { Settings, Book, Code } from "lucide-react"; -## Arns.app +## arns.ar.io -The simplest way to register or manage an ArNS name is by using the user interface maintained by the ar.io team at [arns.app](https://arns.app). This site can also be accessed as an ArNS name itself on any ar.io gateway via ar://arns. +The simplest way to register or manage an ArNS name is by using the user interface maintained by the ar.io team at [arns.ar.io](https://arns.ar.io). This site can also be accessed as an ArNS name itself on any ar.io gateway via ar://arns. - Visit [arns.app](https://arns.app) in your browser and connect your wallet using the "Connect" button at the top right of the screen. + Visit [arns.ar.io](https://arns.ar.io) in your browser and connect your wallet using the "Connect" button at the top right of the screen. Connect Wallet diff --git a/content/build/guides/working-with-arns/register-arns-programmatically.mdx b/content/build/guides/working-with-arns/register-arns-programmatically.mdx index 20d96e78b..38a8462a0 100644 --- a/content/build/guides/working-with-arns/register-arns-programmatically.mdx +++ b/content/build/guides/working-with-arns/register-arns-programmatically.mdx @@ -16,27 +16,20 @@ Use the **ar.io SDK** to programmatically register and purchase ArNS names. This npm install @ar.io/sdk ``` -**Required files:** +**Required:** -```bash -# Your Arweave wallet JSON file -wallet.json -``` +- A Solana wallet with ARIO tokens and SOL for transaction fees ## Basic Setup -**Initialize the SDK with your wallet:** +**Initialize the SDK with your Solana wallet:** ```javascript const { ARIO } = require("@ar.io/sdk"); -const { ArweaveSigner } = require("@ardrive/turbo-sdk"); -const fs = require("fs"); - -// Load your wallet JSON file -const jwk = JSON.parse(fs.readFileSync("./wallet.json", "utf8")); -// Initialize ARIO with signer for transactions -const ario = ARIO.mainnet({ signer: new ArweaveSigner(jwk) }); +// Initialize ARIO with a Solana signer for transactions +// See the SDK Configuration page for how to create a signer from @solana/kit +const ario = ARIO.mainnet({ signer }); ``` ## Complete Registration Process diff --git a/content/build/guides/working-with-arns/set-arns-records-programmatically.mdx b/content/build/guides/working-with-arns/set-arns-records-programmatically.mdx index c1a2f349f..2815dc420 100644 --- a/content/build/guides/working-with-arns/set-arns-records-programmatically.mdx +++ b/content/build/guides/working-with-arns/set-arns-records-programmatically.mdx @@ -12,15 +12,12 @@ Use the **ar.io SDK** to programmatically set and manage ArNS records. This appr **Install the required packages:** ```bash -npm install @ar.io/sdk @ardrive/turbo-sdk +npm install @ar.io/sdk ``` -**Required files:** +**Required:** -```bash -# Your Arweave wallet JSON file -wallet.json -``` +- A Solana wallet with ARIO tokens and SOL for transaction fees ## Basic Setup @@ -28,15 +25,12 @@ wallet.json ```javascript const { ARIO, ANT } = require("@ar.io/sdk"); -const { ArweaveSigner } = require("@ardrive/turbo-sdk"); -const fs = require("fs"); // Initialize ARIO for mainnet const ario = ARIO.mainnet(); -// Load your wallet JSON file -const jwk = JSON.parse(fs.readFileSync("./wallet.json", "utf8")); -const signer = new ArweaveSigner(jwk); +// See the Configuration page for how to create a signer from @solana/kit +// const signer = await createKeyPairSignerFromBytes(keypairBytes); ``` ## Setting Base Name Records @@ -152,7 +146,7 @@ async function deployAndSetRecord(arnsName, undername, distFolderPath) { console.log(`Deployment complete!`); console.log( - `Access at: https://${arnsName}.ar-io.dev${undername === "@" ? "" : `/${undername}`}` + `Access at: https://${arnsName}.turbo-gateway.com${undername === "@" ? "" : `/${undername}`}` ); } catch (error) { console.error("Deployment failed:", error); diff --git a/content/build/index.mdx b/content/build/index.mdx index 401a7fd8d..168625517 100644 --- a/content/build/index.mdx +++ b/content/build/index.mdx @@ -4,7 +4,7 @@ description: "Developer documentation and resources for building on the ar.io ec icon: Wrench --- -import { Code, Zap, BookOpen, Rocket, Network, Database } from "lucide-react"; +import { Code, Zap, BookOpen, Rocket, Network, Database, ShieldCheck } from "lucide-react"; import Image from "next/image"; Welcome to ar.io's developer documentation. @@ -94,6 +94,12 @@ If you're unfamiliar with Arweave's permanent storage and ar.io we recommend rea description="Explore the ArNS marketplace for name trading and management" href="/build/guides/arns-marketplace" /> + } + title="Verifiable AI" + description="Anchor MLflow lifecycle proofs and verify them through ar.io gateways" + href="/build/verifiable-ai" + /> ## Get Help diff --git a/content/build/meta.json b/content/build/meta.json index c1c77a6bb..f14fdd433 100644 --- a/content/build/meta.json +++ b/content/build/meta.json @@ -11,6 +11,7 @@ "run-a-gateway", "run-wayfinder-router", "extensions", + "verifiable-ai", "guides", "advanced" ] diff --git a/content/build/run-a-gateway/join-the-network.mdx b/content/build/run-a-gateway/join-the-network.mdx index ca12fff68..f1e97552f 100644 --- a/content/build/run-a-gateway/join-the-network.mdx +++ b/content/build/run-a-gateway/join-the-network.mdx @@ -27,15 +27,19 @@ Take control of the permanent web by running your own **ar.io Gateway**. Join th ### Minimum Stake Requirement - To join the network as a gateway operator, you need **10,000 ARIO tokens** as the minimum stake requirement. + To join the network as a gateway operator, you need **20,000 ARIO tokens** as the minimum stake requirement. **Need to acquire ARIO tokens?** Visit our [Get the Token guide](/learn/token/get-the-token) to learn about all available methods including exchanges, DEXs, and network participation. **Acquisition Options:** - Purchase on centralized exchanges like Gate.io - - Trade on decentralized exchanges (Dexi, Botega, Vento) - - Use Wander wallet for easy exchange and swap functionality + - Trade on Solana DEXs (Jupiter, Raydium) + - Use a Solana wallet (Phantom, Solflare, Backpack) to manage tokens - Earn through network participation and community programs + + + You will also need SOL in your wallet for Solana transaction fees when joining the network. + @@ -67,7 +71,7 @@ Choose your preferred method to register your gateway: Connect Wallet - Choose your preferred wallet (Wander, Metamask, or Beacon) to connect. + Choose your preferred Solana wallet (Phantom, Solflare, or Backpack) to connect. Use the same wallet address that you configured in your gateway's `AR_IO_WALLET` environment variable. This wallet will be the owner of your gateway registration. @@ -88,7 +92,7 @@ Choose your preferred method to register your gateway: - **Address**: Your gateway's domain with port (e.g., `https://fastandfurious.io:443`) - **Observer Wallet**: The public address of your observer wallet - **Properties ID**: Transaction ID of your gateway properties - - **Stake (ARIO)**: Minimum stake required (typically 10,000 ARIO) + - **Stake (ARIO)**: Minimum stake required (20,000 ARIO) - **Delegated Staking**: Enable to allow others to delegate stake to your gateway - **Minimum Delegated Stake**: Set minimum delegation amount (e.g., 100 ARIO) - **Reward Share Ratio**: Percentage of rewards shared with delegators (e.g., 50%) @@ -135,51 +139,82 @@ Choose your preferred method to register your gateway: ### Run the Join Network Command - Use the `ar.io join-network` command with your gateway configuration: + Use the `ar.io join-network` command with your gateway configuration. The operator key can be supplied either as a JSON keypair file (`--wallet-file`) or as a base58-encoded 64-byte secret key string (`--private-key`). The latter is the format Phantom and similar browser wallets export, so most operators can paste it directly. ```bash ar.io join-network \ - --wallet-file ./path/to/wallet.json \ - --qty 10000000000 \ - --auto-stake true \ - --allow-delegated-staking true \ + --wallet-file ./path/to/solana-keypair.json \ + -t solana \ + --mainnet \ + --rpc-url https://api.mainnet-beta.solana.com \ + --core-program-id 73YoECm6NKXpVRoe5f1Q9BcP5DJGPFUjnFy6AxBE5Nvh \ + --gar-program-id 89fNiiwgpFSPHKuqfNUkgYTYjtAJAhyqHjXmgXeppGpf \ + --arns-program-id 2yCUx5edFvUrkibYaUa2ZXWyx9kuJkS8CwyzsgHPWdZZ \ + --ant-program-id 2MWexMHfMhGJwMHv9Qm9YAVCqjUFUJwDJAysW4oCUGk5 \ + --operator-stake 20000 \ + --auto-stake \ + --allow-delegated-staking \ --min-delegated-stake 100000000 \ --delegate-reward-share-ratio 10 \ - --label "My Test Gateway" \ - --note "Test gateway for development" \ - --observer-wallet 0VE0wIhDy90WiQoV3U2PeY44FH1aVetOoulPGqgYukj \ + --label "My Gateway" \ + --note "Production ar.io gateway" \ + --observer-address 7xKXtR2qpZm8FjvKNsG3kL9p5yMnYhVdEbxQ4oWc2Rn \ --fqdn my-gateway.example.com \ --port 443 \ - --protocol https \ - --mainnet + --protocol https ``` - - The wallet file used in the `--wallet-file` parameter must be the same wallet configured in your ar.io Gateway's `AR_IO_WALLET` environment variable. This ensures your gateway registration is properly linked to your running gateway instance. + + To use a Phantom-exported secret key directly, replace `--wallet-file ./path/to/solana-keypair.json` with `--private-key ''`. The SDK decodes the 64-byte secret, derives the public key, and signs in-memory — no JSON file required. + + + + The public key derived from your `--wallet-file` / `--private-key` becomes your `AR_IO_WALLET` (operator). The `--observer-address` must be a **unique** Solana address; no two gateways can share an observer. If you don't have a separate observer key, set `--observer-address` to your operator address. The gateway accepts this single-key setup. **Parameter explanations:** - - `--qty 10000000000` - 10,000 ARIO in mARIO (multiply by 1,000,000) - - `--min-delegated-stake 100000000` - 100 ARIO in mARIO - - `--delegate-reward-share-ratio 10` - 10% shared with delegators - - `--observer-wallet` - Must match your gateway's OBSERVER_WALLET env var - - `--fqdn` - Your gateway's domain name + - `--operator-stake 20000` — operator stake in whole ARIO units (CLI converts to mARIO internally). Minimum is 20,000 ARIO on mainnet. + - `--auto-stake` — flag-only (no value). When set, operator rewards auto-restake to your gateway. + - `--allow-delegated-staking` — flag-only. Permits other wallets to delegate stake to you. + - `--min-delegated-stake 100000000` — minimum delegation in mARIO (100 ARIO × 1,000,000). Unlike `--operator-stake`, this argument is in mARIO directly. + - `--delegate-reward-share-ratio 10` — percentage of rewards shared with delegators (0-100). + - `--observer-address` — Solana address of the observer key (often the same as the operator address). + - `--fqdn` — your gateway's public domain name. + - `-t solana` — required to switch the CLI into Solana mode (default is Arweave). + - `--mainnet` — selects Solana mainnet behavior. The explicit `--core-program-id`, `--gar-program-id`, `--arns-program-id`, `--ant-program-id`, and `--rpc-url` flags pin the command to the canonical mainnet deployment. ### Verify Registration - After running the command, verify your gateway registration using the CLI: + Verify the on-chain registration via the CLI: + + ```bash + ar.io get-gateway \ + --mainnet \ + -t solana \ + --rpc-url https://api.mainnet-beta.solana.com \ + --core-program-id 73YoECm6NKXpVRoe5f1Q9BcP5DJGPFUjnFy6AxBE5Nvh \ + --gar-program-id 89fNiiwgpFSPHKuqfNUkgYTYjtAJAhyqHjXmgXeppGpf \ + --arns-program-id 2yCUx5edFvUrkibYaUa2ZXWyx9kuJkS8CwyzsgHPWdZZ \ + --ant-program-id 2MWexMHfMhGJwMHv9Qm9YAVCqjUFUJwDJAysW4oCUGk5 \ + --address + ``` + + The result should show `"status": "joined"` and the settings (FQDN, stake, allow-delegated, etc.) you passed at join time. + + Once your gateway is running and pointed at the same network, cross-check from the gateway side too: ```bash - ar.io get-gateway 0VE0wIhDy90WiQoV3U2PeY44FH1aVetOoulPGqgYukj + curl https:///ar-io/info | jq '{wallet, programIds}' ``` - The status should show `joined` when your gateway is successfully registered. + - `wallet` should match the operator pubkey you joined with. + - `programIds.{core,gar,arns,ant}` should match the network you joined (mainnet program IDs, or whichever staging/devnet set you're using). - You can also verify by visiting: `gateways.ar.io/#/` + If those don't line up, your gateway is querying a different network than the one you joined. That's the usual reason a freshly-joined gateway doesn't show up as observing. - The CLI will output transaction details and your gateway should appear in the network portal within a few minutes. + You can also verify in the portal: `gateways.ar.io/#/`. @@ -203,7 +238,7 @@ Your gateway is now part of ar.io! Here are some next steps to maximize your par } /> +The default public Solana RPC is rate-limited and may block `getProgramAccounts` queries needed for full registry enumeration. For production gateways, use a dedicated RPC provider such as [Helius](https://helius.dev), [Triton](https://triton.one), or [QuickNode](https://quicknode.com). + ### Circuit Breaker @@ -316,14 +336,14 @@ For detailed configuration and usage, see [CDB64 Root TX Index](/build/run-a-gat | `PORT` | number | `5050` | Observer service port | | `LOG_LEVEL` | string | - | Observer log level | | `OBSERVER_WALLET` | string | - | Observer wallet | -| `IO_PROCESS_ID` | string | - | ar.io process ID | +| `IO_PROCESS_ID` | string | - | ar.io Solana program address | | `AR_IO_NODE_RELEASE` | string | `33` | ar.io node release version | ### Observer Operation | Variable | Type | Default | Description | | ------------------------------------- | ------- | ------- | ------------------------------------------ | -| `SUBMIT_CONTRACT_INTERACTIONS` | boolean | `true` | Submit contract interactions | +| `SUBMIT_CONTRACT_INTERACTIONS` | boolean | `true` | Submit observations to Solana programs | | `NUM_ARNS_NAMES_TO_OBSERVE_PER_GROUP` | number | `8` | Number of ArNS names per observation group | | `REPORT_GENERATION_INTERVAL_MS` | string | - | Report generation interval | | `REPORT_DATA_SINK` | string | - | Report data sink | @@ -350,11 +370,11 @@ For detailed configuration and usage, see [CDB64 Root TX Index](/build/run-a-gat | ------------------------------------- | ------ | ------- | ------------------------------------------ | | `NUM_ARNS_NAMES_TO_OBSERVE_PER_GROUP` | number | `8` | Number of ArNS names per observation group | -### Contract Interaction +### On-Chain Observations -| Variable | Type | Default | Description | -| ------------------------------ | ------- | ------- | ---------------------------- | -| `SUBMIT_CONTRACT_INTERACTIONS` | boolean | `true` | Submit contract interactions | +| Variable | Type | Default | Description | +| ------------------------------ | ------- | ------- | ------------------------------------- | +| `SUBMIT_CONTRACT_INTERACTIONS` | boolean | `true` | Submit observations to Solana programs | ### Offset Observation @@ -362,16 +382,6 @@ For detailed configuration and usage, see [CDB64 Root TX Index](/build/run-a-gat | ------------------------------- | ------ | ------- | -------------------------- | | `REPORT_GENERATION_INTERVAL_MS` | string | - | Report generation interval | -### AO (Autonomous Objects) - -| Variable | Type | Default | Description | -| ------------------- | ------ | ------- | ----------------- | -| `AO_CU_URL` | string | - | AO CU URL | -| `NETWORK_AO_CU_URL` | string | - | Network AO CU URL | -| `AO_MU_URL` | string | - | AO MU URL | -| `AO_GATEWAY_URL` | string | - | AO Gateway URL | -| `AO_GRAPHQL_URL` | string | - | AO GraphQL URL | - ### Data Paths | Variable | Type | Default | Description | @@ -391,7 +401,7 @@ For detailed configuration and usage, see [CDB64 Root TX Index](/build/run-a-gat | `TVAL_AR_IO_PORT` | number | `4000` | ar.io port | | `TVAL_OBSERVER_HOST` | string | `observer` | Observer host | | `TVAL_OBSERVER_PORT` | number | `5050` | Observer port | -| `TVAL_GATEWAY_HOST` | string | `arweave.net` | Gateway host | +| `TVAL_GATEWAY_HOST` | string | `turbo-gateway.com` | Gateway host | | `TVAL_GRAPHQL_HOST` | string | `core` | GraphQL host | | `TVAL_GRAPHQL_PORT` | number | `4000` | GraphQL port | | `TVAL_ARNS_ROOT_HOST` | string | - | ArNS root host | @@ -550,8 +560,8 @@ PORT=4000 ADMIN_API_KEY=your-admin-key-here # Network configuration -TRUSTED_NODE_URL=https://arweave.net -TRUSTED_GATEWAY_URL=https://arweave.net +TRUSTED_NODE_URL=https://turbo-gateway.com +TRUSTED_GATEWAY_URL=https://turbo-gateway.com # Data paths CHUNKS_DATA_PATH=/data/chunks diff --git a/content/build/run-a-gateway/manage/filters.mdx b/content/build/run-a-gateway/manage/filters.mdx index cac849ecf..b287ae9d1 100644 --- a/content/build/run-a-gateway/manage/filters.mdx +++ b/content/build/run-a-gateway/manage/filters.mdx @@ -53,8 +53,8 @@ ENABLE_BACKGROUND_DATA_VERIFICATION=true Choose between local-only or proxied queries: ```bash -# For new gateways - proxy to arweave.net for complete index -GRAPHQL_HOST=arweave.net +# For new gateways - proxy to turbo-gateway.com for complete index +GRAPHQL_HOST=turbo-gateway.com GRAPHQL_PORT=443 # For local-only queries (uncomment to use) @@ -762,10 +762,4 @@ Now that you understand gateway filtering, continue building your infrastructure description="Accept data uploads directly through your gateway" href="/build/extensions/bundler" /> - } - title="Run Compute Unit" - description="Execute AO processes locally for maximum efficiency" - href="/build/extensions/compute-unit" - /> diff --git a/content/build/run-a-gateway/manage/index.mdx b/content/build/run-a-gateway/manage/index.mdx index 498ff607d..d3eef234e 100644 --- a/content/build/run-a-gateway/manage/index.mdx +++ b/content/build/run-a-gateway/manage/index.mdx @@ -14,6 +14,7 @@ import { Wrench, CreditCard, Search, + Bot, } from "lucide-react"; Master the advanced features and configurations of your ar.io Gateway. These comprehensive guides cover everything from performance optimization to content moderation, helping you run a professional-grade gateway infrastructure. @@ -94,6 +95,23 @@ Master the advanced features and configurations of your ar.io Gateway. These com +## AI-Assisted Operations + + + } + > + The `ar-io-node` repo ships a Claude Code skill at + `.claude/skills/ar-io-gateway-operator/` with a one-screen + `scripts/health-check` snapshot and an operational runbook covering the + ANS-104 pipeline, ClickHouse, ArNS resolution, observer/cranker, and + common pitfalls. Claude Code auto-loads it when run from the repo root. + Readable as plain Markdown if you don't use Claude Code. + + + ## Support & Troubleshooting diff --git a/content/build/run-a-gateway/manage/meta.json b/content/build/run-a-gateway/manage/meta.json index 1977b27e1..3806437b5 100644 --- a/content/build/run-a-gateway/manage/meta.json +++ b/content/build/run-a-gateway/manage/meta.json @@ -2,9 +2,12 @@ "title": "Manage your Gateway", "defaultOpen": false, "pages": [ + "solana-migration", "upgrading-a-gateway", "ssl-certs", "environment-variables", + "verification-headers", + "nginx-caching", "filters", "cdb64", "content-moderation", diff --git a/content/build/run-a-gateway/manage/nginx-caching.mdx b/content/build/run-a-gateway/manage/nginx-caching.mdx new file mode 100644 index 000000000..92f3997e8 --- /dev/null +++ b/content/build/run-a-gateway/manage/nginx-caching.mdx @@ -0,0 +1,798 @@ +--- +title: "Advanced NGINX Caching" +description: "Production-tested NGINX caching configuration for high-traffic ar.io gateways with tiered cache zones, thundering herd protection, and route-specific TTLs" +--- + +import { Callout } from "fumadocs-ui/components/callout"; +import { Tabs, Tab } from "fumadocs-ui/components/tabs"; +import { Card, Cards } from "fumadocs-ui/components/card"; +import { Database, Shield, Zap, Network } from "lucide-react"; + +The [quick-start guide](/build/run-a-gateway/quick-start) covers basic NGINX reverse proxy setup for SSL termination and header forwarding. This guide covers adding a caching layer in front of your ar.io node for high-traffic gateways, based on production configurations running at scale. + + + NGINX caching is optional. The ar.io node has its own internal caching and serves data correctly without it. Add NGINX caching when you need to reduce load on the node process or serve high volumes of traffic for popular content. + + +## Prerequisites + +- A running ar.io gateway with NGINX already configured as a reverse proxy (see the [quick-start guide](/build/run-a-gateway/quick-start)) +- Root or sudo access on the host running NGINX + + + If you are running the default Docker Compose setup, NGINX runs on the host and proxies to **envoy** on port 3000, which in turn proxies to the ar.io node core on port 4000. The `proxy_pass http://127.0.0.1:3000` directives in this guide target envoy, which is the correct entry point. The cache directories described below are on the host filesystem. + + +If you already have an NGINX configuration (e.g., `/etc/nginx/sites-available/default` from the quick-start), you will be adding cache directives to it. The `http`-block directives (cache zones, maps) go outside your existing `server` block, and the location blocks replace or extend the ones in your existing `server` block. + +## Why Cache at NGINX + +Arweave data is immutable - once a transaction is confirmed, its content never changes. This makes it an ideal candidate for aggressive edge caching: + +- **Reduce node load** - Serve repeated requests for the same content directly from NGINX's disk cache without hitting the node process. +- **Thundering herd protection** - When many clients request the same uncached content simultaneously, NGINX ensures only one request reaches the node while others wait for the cached result. +- **Stale serving during failures** - If the node is temporarily unavailable, NGINX can serve stale cached content rather than returning errors. +- **Faster response times** - Cached responses skip the node entirely, reducing latency. + + + **Rate limiter interaction**: Once NGINX caches a significant portion of traffic, fewer requests reach the ar.io node's rate limiter. This effectively increases per-IP limits for cached content. This is generally beneficial but means abusive clients can hammer cached endpoints without triggering rate limits. Consider this when tuning rate limit values. + + +## http Block Configuration + +All `proxy_cache_path` and `map` directives must be placed in the `http` block of your NGINX config, **outside** the `server` block. On Debian/Ubuntu, files in `/etc/nginx/sites-enabled/` are included inside the `http` block via `nginx.conf`, so you can place these directives at the top of your site config file (before the `server` block). + +### Create Cache Directories + +Create the directories and set ownership before reloading NGINX. NGINX will fail to start if these directories don't exist. + + + +```bash +sudo mkdir -p /var/lib/nginx/cache/api +sudo mkdir -p /var/lib/nginx/cache/block +sudo mkdir -p /var/lib/nginx/cache/tx-and-chunk +sudo mkdir -p /var/lib/nginx/cache/data +sudo mkdir -p /var/lib/nginx/cache/arns +sudo chown -R www-data:www-data /var/lib/nginx/cache +``` + + +```bash +sudo mkdir -p /var/lib/nginx/cache/api +sudo mkdir -p /var/lib/nginx/cache/block +sudo mkdir -p /var/lib/nginx/cache/tx-and-chunk +sudo mkdir -p /var/lib/nginx/cache/data +sudo mkdir -p /var/lib/nginx/cache/arns +sudo chown -R nginx:nginx /var/lib/nginx/cache +``` + + + + + If your cache directories are on a different filesystem than NGINX's temp path, cached file writes will use slow cross-filesystem copies instead of fast renames. For best performance, keep cache directories on the same filesystem as NGINX's default temp path. Also avoid placing the cache on the same disk as your ar.io node's SQLite/ClickHouse databases if I/O is constrained. + + +### Tiered Cache Zones + +Different types of gateway content have different caching characteristics. A tiered approach uses separate cache zones sized and configured for each content type: + +```nginx +# --- Tiered cache zones (http block, outside server) --- + +# High churn - small short-TTL API responses +proxy_cache_path /var/lib/nginx/cache/api + levels=1:2 + keys_zone=api_cache:10m + max_size=1g + inactive=10m + use_temp_path=off; + +# Block metadata - highly cacheable, rarely changes +proxy_cache_path /var/lib/nginx/cache/block + levels=1:2 + keys_zone=block_cache:10m + max_size=10g + inactive=365d + use_temp_path=off; + +# TX + chunk metadata - tens of millions of items +proxy_cache_path /var/lib/nginx/cache/tx-and-chunk + levels=1:2 + keys_zone=tx_and_chunk_cache:30m + max_size=30g + inactive=90d + use_temp_path=off + manager_threshold=500ms + loader_files=1000 + loader_threshold=500ms; + +# Data - immutable Arweave content, bulk of disk usage +# Set max_size based on your available disk space +proxy_cache_path /var/lib/nginx/cache/data + levels=1:2 + keys_zone=data_cache:200m + max_size=500g + inactive=90d + use_temp_path=off + manager_threshold=500ms + loader_files=1000 + loader_threshold=500ms; + +# ArNS - web app content served via ArNS subdomains +proxy_cache_path /var/lib/nginx/cache/arns + levels=1:2 + keys_zone=arns_cache:10m + max_size=50g + inactive=30d + use_temp_path=off; +``` + + + Size the `data_cache` zone based on your actual available disk space. The data cache will be the largest zone by far. Leave headroom for the node's own data storage, databases, and OS needs. The `keys_zone` memory size determines how many entries can be tracked — 1MB holds approximately 8,000 keys. + + +| Parameter | Purpose | +|-----------|---------| +| `levels=1:2` | Two-level directory structure for cache files. Prevents any single directory from containing too many files. | +| `keys_zone=name:size` | Shared memory zone for cache keys. Size based on expected number of cached items. | +| `max_size` | Maximum disk space for this cache zone. NGINX evicts least-recently-used entries when exceeded. | +| `inactive` | Remove entries not accessed within this period, even if not expired. | +| `use_temp_path=off` | Write cache files directly to the cache directory (avoids cross-filesystem copies). | +| `manager_threshold` | Maximum time the cache manager spends per cleanup iteration. Prevents disk I/O spikes on large caches. | +| `loader_files` / `loader_threshold` | Controls how NGINX loads cache metadata on startup. Important for large caches to avoid slow restarts. | + +### Map Directives + +All `map` directives must also be in the `http` block. These must be defined **before** the `server` block because the server references the variables they create. + +```nginx +# --- Map directives (http block, outside server) --- + +# WebSocket upgrade support (used by /graphql and catch-all) +map $http_upgrade $connection_upgrade { + default upgrade; + '' close; +} + +# Route ArNS subdomains to arns_cache, everything else to api_cache. +# Excludes sandbox subdomains (52-char base32) which are not ArNS. +map $host $catch_all_cache { + "~^[a-z2-7]{52}\." api_cache; + "~^[^.]+\..+\..+" arns_cache; + default api_cache; +} + +# Never cache 429 responses — prevent rate-limit errors from +# overwriting valid cache entries. +map $upstream_status $no_cache_429 { + 429 1; + default 0; +} + +# Never cache responses with Cache-Control: no-store — prevents +# caching 402 Payment Required and other uncacheable responses. +map $upstream_http_cache_control $no_cache_no_store { + "~no-store" 1; + default 0; +} +``` + +## Cache Key Design + +For Arweave content, query strings are not meaningful - the content is identified by its transaction ID in the URL path. The cache key excludes query strings to prevent cache fragmentation: + +```nginx +proxy_cache_key "$scheme://$host$uri"; +``` + +The `$host` component is important because ArNS subdomains serve different content at the same path. + +## Server Block Configuration + +The following directives go inside your `server` block. If you have an existing server block from the quick-start guide, merge these directives into it. + +### Shared Defaults + +These directives apply as defaults across all locations. Individual locations override them as needed: + +```nginx +server { + # ... your existing SSL, server_name, listen directives ... + + # Default cache zone (overridden per-location) + proxy_cache api_cache; + proxy_cache_key "$scheme://$host$uri"; + + # Thundering herd protection: first request fetches from origin, + # others wait up to lock_timeout, then go to origin if lock_age exceeded + proxy_cache_lock on; + proxy_cache_lock_age 60s; + proxy_cache_lock_timeout 60s; + + # Serve stale entries during errors, timeouts, or background updates + proxy_cache_use_stale error timeout updating http_500 http_502 http_503 http_504; + proxy_cache_revalidate on; + proxy_cache_background_update on; + + # Never cache 429 or no-store responses + proxy_no_cache $no_cache_429 $no_cache_no_store; + proxy_cache_bypass $no_cache_429 $no_cache_no_store; + + # Error TTLs shared across all locations + proxy_cache_valid 400 60s; + proxy_cache_valid 403 10s; + proxy_cache_valid 451 30d; + proxy_cache_valid 500 502 503 504 10s; + + # Prevent cache fragmentation from Vary headers. + # Trade-off: this drops ALL Vary values, not just Vary: Origin. + # For Arweave gateways this is safe because the node does not use + # Vary: Accept-Encoding for content negotiation. + proxy_ignore_headers Vary; + + # Proxy buffer settings + proxy_buffering on; + proxy_buffer_size 32k; + proxy_buffers 64 32k; + proxy_max_temp_file_size 8192m; + + # Default timeouts + proxy_read_timeout 120s; + proxy_send_timeout 30s; + + # ... location blocks follow ... +} +``` + +### Route-Specific Caching + +#### Immutable Data (30-day TTL) + +Transaction data and raw content are immutable on Arweave. Cache aggressively with extended timeouts for large file downloads: + +```nginx +# Raw data endpoint +location ^~ /raw/ { + proxy_pass http://127.0.0.1:3000; + proxy_http_version 1.1; + proxy_set_header Connection ""; + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + proxy_set_header X-Forwarded-Host $host; + proxy_set_header X-AR-IO-Origin $http_x_ar_io_origin; + proxy_set_header X-AR-IO-Origin-Node-Release $http_x_ar_io_origin_node_release; + proxy_set_header X-AR-IO-Hops $http_x_ar_io_hops; + + add_header X-Cache-Status $upstream_cache_status always; + + proxy_read_timeout 600s; + proxy_send_timeout 600s; + # Lock holder gets 10 minutes for large file downloads; + # waiters give up after 5s and go to origin directly + proxy_cache_lock_age 600s; + proxy_cache_lock_timeout 5s; + proxy_cache data_cache; + # Disable background refresh — avoids expensive multi-GB origin fetches + proxy_cache_background_update off; + proxy_ignore_headers Set-Cookie Vary; + proxy_hide_header Set-Cookie; + proxy_cache_valid 200 30d; + proxy_cache_valid 404 60s; +} + +# Transaction data (43-char base64url IDs) +location ~ "^/[a-zA-Z0-9_-]{43}(/|$)" { + proxy_pass http://127.0.0.1:3000; + proxy_http_version 1.1; + proxy_set_header Connection ""; + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + proxy_set_header X-Forwarded-Host $host; + proxy_set_header X-AR-IO-Origin $http_x_ar_io_origin; + proxy_set_header X-AR-IO-Origin-Node-Release $http_x_ar_io_origin_node_release; + proxy_set_header X-AR-IO-Hops $http_x_ar_io_hops; + + add_header X-Cache-Status $upstream_cache_status always; + + proxy_read_timeout 600s; + proxy_send_timeout 600s; + proxy_cache_lock_age 600s; + proxy_cache_lock_timeout 5s; + proxy_cache data_cache; + proxy_cache_background_update off; + proxy_ignore_headers Set-Cookie Vary; + proxy_hide_header Set-Cookie; + proxy_cache_valid 200 30d; + proxy_cache_valid 404 60s; +} +``` + + + Note that `/raw/TX_ID` and `/TX_ID` serve the same content but are cached as separate entries because the cache key includes the URI path. For most gateways this duplication is acceptable. If disk space is tight, you can normalize the cache key with a `map` — see the [Troubleshooting](#troubleshooting) section. + + +#### Chunks (24-hour TTL) + +Arweave chunks are immutable but accessed less frequently than full transactions: + +```nginx +location ^~ /chunk/ { + proxy_pass http://127.0.0.1:3000; + proxy_http_version 1.1; + proxy_set_header Connection ""; + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + proxy_set_header X-Forwarded-Host $host; + proxy_set_header X-AR-IO-Origin $http_x_ar_io_origin; + proxy_set_header X-AR-IO-Origin-Node-Release $http_x_ar_io_origin_node_release; + proxy_set_header X-AR-IO-Hops $http_x_ar_io_hops; + + add_header X-Cache-Status $upstream_cache_status always; + + proxy_cache tx_and_chunk_cache; + proxy_cache_background_update off; + proxy_ignore_headers Cache-Control Expires Set-Cookie Vary; + proxy_hide_header Set-Cookie; + proxy_cache_valid 200 24h; + proxy_cache_valid 404 30s; +} +``` + +#### Volatile Metadata (Short TTLs) + +API endpoints and metadata change frequently and need short TTLs: + +```nginx +# Health, info, metrics, tx_anchor (2-minute TTL) +location ~ ^/(health|info|metrics|tx_anchor)$ { + proxy_pass http://127.0.0.1:3000; + proxy_http_version 1.1; + proxy_set_header Connection ""; + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + proxy_set_header X-Forwarded-Host $host; + proxy_set_header X-AR-IO-Origin $http_x_ar_io_origin; + proxy_set_header X-AR-IO-Origin-Node-Release $http_x_ar_io_origin_node_release; + proxy_set_header X-AR-IO-Hops $http_x_ar_io_hops; + + add_header X-Cache-Status $upstream_cache_status always; + proxy_cache_valid 200 120s; + proxy_cache_valid 404 30s; +} + +# Height and time (20-second TTL - changes every block) +location ~ ^/(height|time)$ { + proxy_pass http://127.0.0.1:3000; + proxy_http_version 1.1; + proxy_set_header Connection ""; + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + proxy_set_header X-Forwarded-Host $host; + proxy_set_header X-AR-IO-Origin $http_x_ar_io_origin; + proxy_set_header X-AR-IO-Origin-Node-Release $http_x_ar_io_origin_node_release; + proxy_set_header X-AR-IO-Hops $http_x_ar_io_hops; + + add_header X-Cache-Status $upstream_cache_status always; + proxy_cache_valid 200 20s; + proxy_cache_valid 404 30s; +} + +# Peers, current_block (30-second TTL) +location ~ ^/(peers|current_block)$ { + proxy_pass http://127.0.0.1:3000; + proxy_http_version 1.1; + proxy_set_header Connection ""; + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + proxy_set_header X-Forwarded-Host $host; + proxy_set_header X-AR-IO-Origin $http_x_ar_io_origin; + proxy_set_header X-AR-IO-Origin-Node-Release $http_x_ar_io_origin_node_release; + proxy_set_header X-AR-IO-Hops $http_x_ar_io_hops; + + add_header X-Cache-Status $upstream_cache_status always; + proxy_cache_valid 200 30s; + proxy_cache_valid 404 15s; +} + +# Wallet, price, unconfirmed TX lookups (30-second TTL) +location ~ ^/(wallet|price|unconfirmed_tx)/ { + proxy_pass http://127.0.0.1:3000; + proxy_http_version 1.1; + proxy_set_header Connection ""; + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + proxy_set_header X-Forwarded-Host $host; + proxy_set_header X-AR-IO-Origin $http_x_ar_io_origin; + proxy_set_header X-AR-IO-Origin-Node-Release $http_x_ar_io_origin_node_release; + proxy_set_header X-AR-IO-Hops $http_x_ar_io_hops; + + add_header X-Cache-Status $upstream_cache_status always; + proxy_cache_valid 200 30s; + proxy_cache_valid 404 15s; +} + +# AR.IO API (30-second TTL) +location ^~ /ar-io/ { + proxy_pass http://127.0.0.1:3000; + proxy_http_version 1.1; + proxy_set_header Connection ""; + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + proxy_set_header X-Forwarded-Host $host; + proxy_set_header X-AR-IO-Origin $http_x_ar_io_origin; + proxy_set_header X-AR-IO-Origin-Node-Release $http_x_ar_io_origin_node_release; + proxy_set_header X-AR-IO-Hops $http_x_ar_io_hops; + + add_header X-Cache-Status $upstream_cache_status always; + proxy_cache_valid 200 30s; + proxy_cache_valid 404 5s; +} + +# Block metadata +location ^~ /block/ { + proxy_pass http://127.0.0.1:3000; + proxy_http_version 1.1; + proxy_set_header Connection ""; + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + proxy_set_header X-Forwarded-Host $host; + proxy_set_header X-AR-IO-Origin $http_x_ar_io_origin; + proxy_set_header X-AR-IO-Origin-Node-Release $http_x_ar_io_origin_node_release; + proxy_set_header X-AR-IO-Hops $http_x_ar_io_hops; + + add_header X-Cache-Status $upstream_cache_status always; + proxy_cache block_cache; + proxy_cache_valid 404 30s; +} + +# TX metadata +location /tx/ { + proxy_pass http://127.0.0.1:3000; + proxy_http_version 1.1; + proxy_set_header Connection ""; + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + proxy_set_header X-Forwarded-Host $host; + proxy_set_header X-AR-IO-Origin $http_x_ar_io_origin; + proxy_set_header X-AR-IO-Origin-Node-Release $http_x_ar_io_origin_node_release; + proxy_set_header X-AR-IO-Hops $http_x_ar_io_hops; + + add_header X-Cache-Status $upstream_cache_status always; + proxy_cache tx_and_chunk_cache; + proxy_cache_valid 404 30s; +} + +# TX status (30-second TTL) +location ~ ^/tx/[A-Za-z0-9_-]+/status$ { + proxy_pass http://127.0.0.1:3000; + proxy_http_version 1.1; + proxy_set_header Connection ""; + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + proxy_set_header X-Forwarded-Host $host; + proxy_set_header X-AR-IO-Origin $http_x_ar_io_origin; + proxy_set_header X-AR-IO-Origin-Node-Release $http_x_ar_io_origin_node_release; + proxy_set_header X-AR-IO-Hops $http_x_ar_io_hops; + + add_header X-Cache-Status $upstream_cache_status always; + proxy_cache tx_and_chunk_cache; + proxy_cache_valid 200 30s; + proxy_cache_valid 404 30s; +} + +# Pending transactions (20-second TTL) +location = /tx/pending { + proxy_pass http://127.0.0.1:3000; + proxy_http_version 1.1; + proxy_set_header Connection ""; + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + proxy_set_header X-Forwarded-Host $host; + proxy_set_header X-AR-IO-Origin $http_x_ar_io_origin; + proxy_set_header X-AR-IO-Origin-Node-Release $http_x_ar_io_origin_node_release; + proxy_set_header X-AR-IO-Hops $http_x_ar_io_hops; + + add_header X-Cache-Status $upstream_cache_status always; + proxy_cache tx_and_chunk_cache; + proxy_cache_valid 200 20s; + proxy_cache_valid 404 30s; +} + +# Current block (2-minute TTL) +location = /block/current { + proxy_pass http://127.0.0.1:3000; + proxy_http_version 1.1; + proxy_set_header Connection ""; + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + proxy_set_header X-Forwarded-Host $host; + proxy_set_header X-AR-IO-Origin $http_x_ar_io_origin; + proxy_set_header X-AR-IO-Origin-Node-Release $http_x_ar_io_origin_node_release; + proxy_set_header X-AR-IO-Hops $http_x_ar_io_hops; + + add_header X-Cache-Status $upstream_cache_status always; + proxy_cache_valid 200 120s; + proxy_cache_valid 404 30s; +} +``` + +#### Endpoints to Never Cache + +Some endpoints must never be cached because they handle writes, WebSocket connections, or state-changing operations. These locations use `proxy_cache off` and explicitly set `Cache-Control: no-store` (this is a response header injected by NGINX, distinct from the upstream `no-store` bypass in the map directives): + +```nginx +# GraphQL - supports WebSocket upgrades and POST mutations +location = /graphql { + proxy_pass http://127.0.0.1:3000; + proxy_http_version 1.1; + proxy_set_header Connection ""; + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + proxy_set_header X-Forwarded-Host $host; + proxy_set_header X-AR-IO-Origin $http_x_ar_io_origin; + proxy_set_header X-AR-IO-Origin-Node-Release $http_x_ar_io_origin_node_release; + proxy_set_header X-AR-IO-Hops $http_x_ar_io_hops; + + proxy_set_header Upgrade $http_upgrade; + proxy_set_header Connection $connection_upgrade; + proxy_cache off; + proxy_hide_header Cache-Control; + add_header Cache-Control "no-store" always; +} + +# Transaction submission (POST endpoint) +location = /tx { + proxy_pass http://127.0.0.1:3000; + proxy_http_version 1.1; + proxy_set_header Connection ""; + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + proxy_set_header X-Forwarded-Host $host; + + proxy_cache off; + proxy_hide_header Cache-Control; + add_header Cache-Control "no-store" always; +} + +# Chunk upload (POST endpoint) +location = /chunk { + proxy_pass http://127.0.0.1:3000; + proxy_http_version 1.1; + proxy_set_header Connection ""; + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + proxy_set_header X-Forwarded-Host $host; + + proxy_cache off; + proxy_hide_header Cache-Control; + add_header Cache-Control "no-store" always; +} +``` + +#### Catch-All with ArNS Routing + +The catch-all location handles remaining requests including ArNS subdomain content: + +```nginx +location / { + proxy_pass http://127.0.0.1:3000; + proxy_http_version 1.1; + proxy_set_header Connection ""; + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + proxy_set_header X-Forwarded-Host $host; + proxy_set_header X-AR-IO-Origin $http_x_ar_io_origin; + proxy_set_header X-AR-IO-Origin-Node-Release $http_x_ar_io_origin_node_release; + proxy_set_header X-AR-IO-Hops $http_x_ar_io_hops; + + proxy_set_header Upgrade $http_upgrade; + proxy_set_header Connection $connection_upgrade; + + add_header X-Cache-Status $upstream_cache_status always; + proxy_cache $catch_all_cache; + proxy_cache_valid 200 300s; + proxy_cache_valid 404 30s; +} +``` + +## TTL Summary + +| Route | Cache Zone | 200 TTL | 404 TTL | Notes | +|-------|-----------|---------|---------|-------| +| `/raw/`, `/[TX_ID]` | data_cache | 30d | 60s | Immutable content, extended timeouts | +| `/chunk/` | tx_and_chunk_cache | 24h | 30s | Immutable chunks | +| `/block/` | block_cache | origin | 30s | Origin-controlled TTL | +| `/block/current` | api_cache | 120s | 30s | Changes every block | +| `/tx/` | tx_and_chunk_cache | origin | 30s | Origin-controlled TTL | +| `/tx/[ID]/status` | tx_and_chunk_cache | 30s | 30s | Status can change | +| `/tx/pending` | tx_and_chunk_cache | 20s | 30s | Changes frequently | +| `/ar-io/` | api_cache | 30s | 5s | Short-lived API data | +| `/health`, `/info` | api_cache | 120s | 30s | Metadata endpoints | +| `/height`, `/time` | api_cache | 20s | 30s | Changes every block | +| `/peers`, `/current_block` | api_cache | 30s | 15s | Network state | +| `/wallet/`, `/price/` | api_cache | 30s | 15s | Volatile data | +| `/` (catch-all) | dynamic | 300s | 30s | ArNS or API cache | +| `/graphql`, `/tx` POST, `/chunk` POST | none | - | - | Never cached | + +## Content Moderation & Cache Purging + +Caching immutable content for 30 days creates a compliance risk: if the ar.io node blocks a transaction via [content moderation](/build/run-a-gateway/manage/content-moderation), NGINX may continue serving the cached copy for the remainder of the TTL. + + + This is a compliance concern for production gateways. If you use content moderation filters, you need a way to force-refresh cached entries after blocking. + + +### Cache Bypass Header + +Add a `map` that checks a secret bypass header value, and use it with `proxy_cache_bypass`. When triggered, NGINX skips the cache and fetches fresh from the ar.io node — which now returns the blocked response — and overwrites the stale cache entry. + +In the `http` block (with your other maps), add: + +```nginx +# Cache bypass for content moderation purging. +# Only bypass when the header matches your secret — prevents abuse. +map $http_x_cache_bypass $purge_allowed { + "your-secret-here" 1; + default 0; +} +``` + +In each data-serving location (`/raw/`, `/[TX_ID]`, catch-all), add: + +```nginx +proxy_cache_bypass $purge_allowed; +``` + +Then force-refresh a blocked transaction: + +```bash +# Replace "your-secret-here" and domain with your actual values +curl -s -o /dev/null -H "X-Cache-Bypass: your-secret-here" \ + https://your-domain.example/TX_ID_HERE +curl -s -o /dev/null -H "X-Cache-Bypass: your-secret-here" \ + https://your-domain.example/raw/TX_ID_HERE +``` + +The node returns its blocked/404 response, NGINX caches that instead, and subsequent requests get the blocked response. + +If you run a content scanning sidecar, fire these bypass requests automatically after each block event to complete the purge without any extra infrastructure. + +## Monitoring Cache Performance + +Add the `X-Cache-Status` header to expose cache behavior on every response: + +```nginx +add_header X-Cache-Status $upstream_cache_status always; +``` + +Check cache performance: + +```bash +curl -s -D - -o /dev/null https://your-gateway.example/TX_ID 2>&1 | grep -i x-cache +``` + +| Status | Meaning | +|--------|---------| +| `HIT` | Served from cache | +| `MISS` | Fetched from origin, now cached | +| `EXPIRED` | Cache entry expired, fetched fresh from origin | +| `UPDATING` | Stale entry served while background update runs | +| `STALE` | Stale entry served due to origin error | +| `BYPASS` | Cache was bypassed (e.g., 429 or no-store response) | +| (empty) | Response status not covered by any `proxy_cache_valid` directive for this location | + +## Validate and Reload + +Always test your configuration before reloading NGINX: + +```bash +sudo nginx -t && sudo systemctl reload nginx +``` + +If `nginx -t` reports errors, fix them before reloading. A bad reload with `systemctl reload` is safe (NGINX keeps the old config running), but `systemctl restart` with a broken config will take NGINX offline. + +## Additional Configuration + +### Load Balancer Real IP + +If your gateway is behind a load balancer, configure NGINX to trust the `X-Forwarded-For` header from the load balancer's IP: + +```nginx +# Replace with your load balancer's actual IP +set_real_ip_from 10.0.0.1/32; +real_ip_header X-Forwarded-For; +real_ip_recursive on; +``` + +Without this, all requests appear to come from the load balancer's IP. + +## Troubleshooting + +### Duplicate Cache Entries for /raw/ and / + +`/raw/TX_ID` and `/TX_ID` serve the same bytes but are cached as separate entries because the cache key includes the full URI path. If disk space is a concern, normalize the key with a map in the `http` block: + +```nginx +map $uri $normalized_cache_uri { + "~^/raw/(.+)$" "/$1"; + default $uri; +} +``` + +Then use `proxy_cache_key "$scheme://$host$normalized_cache_uri";` in both the `/raw/` and TX data locations. This causes both endpoints to share one cache entry. + +### Cache Directories Not Writable + +If NGINX logs show write errors, check ownership matches the NGINX worker user (`www-data` on Debian/Ubuntu, `nginx` on RHEL/CentOS): + +```bash +ls -la /var/lib/nginx/cache/ +``` + +### Shared Memory Exhaustion + +If `keys_zone` is too small for the number of cached items, NGINX silently evicts entries. Monitor with `stub_status` and increase `keys_zone` size if your hit rate drops unexpectedly despite having disk space available. + +## Related + + + } + /> + } + /> + } + /> + } + /> + diff --git a/content/build/run-a-gateway/manage/setting-apex-domain.mdx b/content/build/run-a-gateway/manage/setting-apex-domain.mdx index cec21b4db..e6b1b0d13 100644 --- a/content/build/run-a-gateway/manage/setting-apex-domain.mdx +++ b/content/build/run-a-gateway/manage/setting-apex-domain.mdx @@ -180,7 +180,7 @@ ardrive upload-file --file-path ./my-dapp.html Register an ArNS name pointing to your content: -1. Visit [ArNS App](https://arns.app) +1. Visit [ArNS App](https://arns.ar.io) 2. Connect your wallet 3. Choose your desired name (e.g., `my-gateway-content`) 4. Set the transaction ID: `xyz789...abc123` @@ -447,7 +447,7 @@ Several gateway operators have implemented this feature: - Serves personalized portfolio/link tree information - Personal branding and contact information -**permagate.io** +**ardrive.net** - Serves personalized link tree information - Professional operator presence @@ -531,10 +531,10 @@ Test if your content is accessible: ```bash # Test transaction ID directly -curl -I https://arweave.net/your-transaction-id +curl -I https://turbo-gateway.com/your-transaction-id # Test ArNS name resolution -curl -I https://your-arns-name.arweave.net +curl -I https://your-arns-name.ar.io ``` @@ -577,10 +577,10 @@ Check if your ArNS name resolves correctly: ```bash # Test ArNS resolution -nslookup your-arns-name.arweave.net +nslookup your-arns-name.ar.io # Check if it points to the correct transaction -curl -s https://your-arns-name.arweave.net | head -10 +curl -s https://your-arns-name.ar.io | head -10 ``` @@ -590,7 +590,7 @@ curl -s https://your-arns-name.arweave.net | head -10 If ArNS name points to wrong content: -1. Go to [ArNS App](https://arns.app) +1. Go to [ArNS App](https://arns.ar.io) 2. Find your ArNS name 3. Update the transaction ID 4. Wait for propagation (usually immediate) diff --git a/content/build/run-a-gateway/manage/solana-migration.mdx b/content/build/run-a-gateway/manage/solana-migration.mdx new file mode 100644 index 000000000..31cc8c022 --- /dev/null +++ b/content/build/run-a-gateway/manage/solana-migration.mdx @@ -0,0 +1,280 @@ +--- +title: "Solana Migration for Operators" +description: "What existing ar.io gateway operators need to do to transition from AO to Solana" +--- + +import { Shield, Key, Server, Coins } from 'lucide-react'; + +## Overview + +The ar.io network has migrated protocol execution from AO to Solana. If you're an existing gateway operator, this guide covers everything you need to change. **Your gateway will continue serving Arweave data uninterrupted** — the changes affect how your node interacts with the protocol layer (staking, observations, rewards). + + +Complete these steps before the cutover date to ensure uninterrupted reward eligibility. Gateways that fail 30 consecutive epochs will be [pruned](/learn/oip/pruning) with 100% of minimum stake slashed. + + +## What Changed + +| Before (AO) | After (Solana) | +|-------------|----------------| +| Arweave wallet (RSA JWK) | Solana keypair (Ed25519) | +| AO Compute Unit for state reads | Solana RPC for state reads | +| `AO_CU_URL`, `AO_MU_URL`, etc. | `SOLANA_RPC_URL` | +| Observations submitted via AO messages | Observations submitted as Solana transactions | +| Observer wallet pays in AR/Turbo credits | Observer wallet pays in SOL | +| Rewards distributed atomically | Rewards distributed via [cranker pipeline](/learn/oip/epoch-pipeline) | +| Observer address not unique | Observer address must be unique per gateway | +| Minimum stake = 10,000 ARIO | Minimum stake = 20,000 ARIO | + +## What Didn't Change + +- Your gateway still serves Arweave data (caching, indexing, ArNS resolution) +- Docker-based deployment workflow is the same +- Epoch duration remains 24 hours +- Staking, delegation, and reward mechanics are functionally equivalent + +## Step-by-Step Migration + + + + ### Map Your Address + + Before the cutover, register your Solana wallet address using the ar.io claim app. This maps your existing Arweave address to your new Solana address so your stake, delegation, and gateway registration carry over automatically. + + + + ### Generate a Solana Keypair + + Create a new Solana keypair for your gateway and observer: + + ```bash + # Install Solana CLI tools + sh -c "$(curl -sSfL https://release.anza.xyz/stable/install)" + + # Generate gateway operator keypair + solana-keygen new --outfile operator-keypair.json + + # Generate observer keypair (must be unique — not shared with any other gateway) + solana-keygen new --outfile observer-keypair.json + ``` + + + Your observer address must be **unique across all gateways**. If another gateway is already using the same observer address, your registration will fail. This is enforced onchain by the ObserverLookup PDA. + + + + + ### Fund Your Wallets with SOL + + Your observer wallet needs SOL for submitting observation transactions. Typical costs are less than 0.01 SOL per transaction. + + ```bash + # Check your observer address + solana-keygen pubkey observer-keypair.json + + # Send SOL to it from an exchange or existing wallet + ``` + + We recommend keeping at least **0.5 SOL** in the observer wallet to cover several months of observation submissions. + + + + ### Update Environment Variables + + Edit your `.env` file with the new Solana configuration: + + ```bash + # Operator + observer addresses (Solana pubkeys, base58) + AR_IO_WALLET= + OBSERVER_WALLET= + + # Solana RPC endpoint (see Callout below — use a premium provider in production) + SOLANA_RPC_URL=https://your-rpc-provider.com + + # Keypair file paths inside the container (mounted from ./wallets/) + SOLANA_KEYPAIR_PATH=/app/wallets/operator-keypair.json + OBSERVER_KEYPAIR_PATH=/app/wallets/observer-keypair.json + + # The four ar.io Solana programs (canonical mainnet) + ARIO_CORE_PROGRAM_ID=73YoECm6NKXpVRoe5f1Q9BcP5DJGPFUjnFy6AxBE5Nvh + ARIO_GAR_PROGRAM_ID=89fNiiwgpFSPHKuqfNUkgYTYjtAJAhyqHjXmgXeppGpf + ARIO_ARNS_PROGRAM_ID=2yCUx5edFvUrkibYaUa2ZXWyx9kuJkS8CwyzsgHPWdZZ + ARIO_ANT_PROGRAM_ID=2MWexMHfMhGJwMHv9Qm9YAVCqjUFUJwDJAysW4oCUGk5 + ``` + + **Remove the old AO variables** (they are no longer used): + ```bash + # DELETE THESE from your .env: + # AO_CU_URL=... + # NETWORK_AO_CU_URL=... + # ANT_AO_CU_URL=... + # AO_MU_URL=... + # AO_GATEWAY_URL=... + # AO_GRAPHQL_URL=... + # HTTPSIG_UPLOAD_ATTESTATION=... + # WALLETS_PATH=... + ``` + + + The default public Solana RPC (`api.mainnet-beta.solana.com`) is rate-limited. Under any real load your observer and cranker will start dropping transactions, and the resulting timeouts cause most observation cycles to miss their submission window. Use a dedicated RPC provider in production: [Helius](https://helius.dev), [Triton](https://triton.one), or [QuickNode](https://quicknode.com). + + + + If you already have a Phantom-exported base58 secret string for the observer or operator, you can skip the JSON keypair file entirely: set `OBSERVER_PRIVATE_KEY=` (or `SOLANA_PRIVATE_KEY=`) instead of the `*_KEYPAIR_PATH` env. Setting both forms for the same role is rejected at startup. + + + + + ### Place Keypair Files + + Copy your operator and observer Solana keypair JSON files into the `wallets/` directory in your ar-io-node checkout. Docker Compose mounts this directory into `/app/wallets/` inside the container, which is where the `*_KEYPAIR_PATH` env vars point. + + ```bash + cp operator-keypair.json wallets/operator-keypair.json + cp observer-keypair.json wallets/observer-keypair.json + chmod 600 wallets/*.json + ``` + + Skip this step entirely if you set `OBSERVER_PRIVATE_KEY` / `SOLANA_PRIVATE_KEY` env vars (base58 strings) instead. + + + + ### Update ar-io-node + + Pull the latest Solana-compatible release and recreate the containers: + + ```bash + cd ar-io-node + git pull + docker compose pull + docker compose up -d --force-recreate core envoy observer + ``` + + + Avoid `docker compose down -v` here — the `-v` flag deletes named volumes, which on most setups wipes the sqlite index and chunk data the gateway has spent days/weeks building. Use `docker compose up -d --force-recreate ` instead, which restarts containers in place without touching volumes. + + + Check your release version at `https:///ar-io/info`. + + + + ### Optional: Enable Epoch Cranking + + Your observer can optionally help drive the epoch reward pipeline. This is completely permissionless and costs minimal SOL (~0.000155 SOL per epoch): + + ```bash + # Add to .env + ENABLE_EPOCH_CRANKING=true + ``` + + See the [epoch pipeline](/learn/oip/epoch-pipeline) docs for details. Running the cranker creates redundancy so the network is never dependent on a single bot. + + + + ### Verify + + After restarting, verify your gateway is operating correctly: + + 1. **Release + program IDs (cross-check config is what you intended):** + ```bash + curl -s https:///ar-io/info | jq '{release, wallet, programIds}' + ``` + `programIds.core/gar/arns/ant` should match the network you're targeting. If they don't, the gateway is still pointed at the old program set, which is usually why a migrated gateway looks inert. + + 2. **Gateway registration is live on the new network:** + ```bash + ar.io get-gateway -t solana \ + --rpc-url https://api.mainnet-beta.solana.com \ + --core-program-id 73YoECm6NKXpVRoe5f1Q9BcP5DJGPFUjnFy6AxBE5Nvh \ + --gar-program-id 89fNiiwgpFSPHKuqfNUkgYTYjtAJAhyqHjXmgXeppGpf \ + --arns-program-id 2yCUx5edFvUrkibYaUa2ZXWyx9kuJkS8CwyzsgHPWdZZ \ + --ant-program-id 2MWexMHfMhGJwMHv9Qm9YAVCqjUFUJwDJAysW4oCUGk5 \ + --address + ``` + Should return `"status": "joined"` with the FQDN/stake/settings you set at join time. + + 3. **ArNS resolution works end-to-end:** + ```bash + curl -I -H "Host: ardrive." https:/// + ``` + Expect HTTP 200 (or a manifest redirect). 404 here means the cache hasn't hydrated yet — give it a minute and re-check. + + 4. **Observer reports are flowing:** `https:///ar-io/observer/reports/current` + + 5. **Gateway listed in the portal:** [gateways.ar.io](https://gateways.ar.io) + + 6. **No accumulating failed epochs** in the portal's gateway view. + + + +## Troubleshooting + +### "Observer is restart-looping with 'Epoch 0 PDA not found'" + +If the observer logs repeatedly: + +``` +error: Continuous observer start() rejected — attempting auto-restart + ...error: "Epoch 0 PDA not found at — has prescribe_epoch run yet?" +``` + +…the network you're targeting hasn't had its first epoch initialized. The observer needs entropy from `epoch[N].prescribed_observers` to bootstrap, which doesn't exist until someone calls `create_epoch` (typically a cranker). + +Verify the epoch state directly: + +```bash +ar.io get-current-epoch -t solana \ + --rpc-url https://api.mainnet-beta.solana.com \ + --core-program-id 73YoECm6NKXpVRoe5f1Q9BcP5DJGPFUjnFy6AxBE5Nvh \ + --gar-program-id 89fNiiwgpFSPHKuqfNUkgYTYjtAJAhyqHjXmgXeppGpf \ + --arns-program-id 2yCUx5edFvUrkibYaUa2ZXWyx9kuJkS8CwyzsgHPWdZZ \ + --ant-program-id 2MWexMHfMhGJwMHv9Qm9YAVCqjUFUJwDJAysW4oCUGk5 +``` + +For staging-devnet or local devnet, replace the RPC URL and program IDs with that environment's values. + +If this returns `"Epoch 0 not found"`, the network is configured but inactive. This is a network-operations state, not a gateway misconfiguration. Wait for an active cranker (yours or another operator's) to bootstrap epoch 0. + +### "Cranker started but never submits any instructions" + +The cranker silently bails out if `EpochSettings.enabled` is `false`. This is intentional: operators shouldn't pay SOL fees attempting cranker instructions against a paused network. Confirm with `ar.io get-epoch-settings` and check whether epochs are enabled on your target network. If `enabled` is false, no cranker activity is expected. + +### "ArNS names return 404 but the on-chain record exists" + +The gateway hydrates an ArNS names cache at boot, paginating through the on-chain registry. If your SDK pin is significantly older than the deployed `ario-arns` program, the paginated response shape may not match and the cache hydrates incompletely. Symptoms: + +- Cache hydration logs `Successfully hydrated ArNS names cache` quickly (e.g. ~4s for thousands of records) +- Resolver logs report `Base name not found in ArNS names cache` for names that demonstrably exist via the SDK CLI's `get-arns-record --name ` + +Fix: bump `@ar.io/sdk` in your gateway's image (`package.json`) to the latest `^4.0.0-solana.*` and rebuild. + +## New Risks to Be Aware Of + +### Gateway Pruning + +Gateways that fail **30 consecutive epochs** are automatically pruned from the network. When pruned: +- 100% of minimum stake (20,000 ARIO) is slashed to the protocol balance +- Excess operator stake enters the standard 30-day withdrawal queue +- Delegated stakes enter the standard 30-day withdrawal queue (delegators are not slashed) + +See [Gateway Pruning](/learn/oip/pruning) for full details. + +### SOL Balance Monitoring + +Keep your observer wallet funded with SOL. If it runs out, your observer cannot submit observations, which leads to failed epochs and eventually pruning. Set up monitoring/alerts for your observer wallet balance. + +## FAQ + +**Do I need to re-register my gateway?** +No. If you mapped your address before cutover, your gateway registration, stake, and delegations are migrated automatically. + +**What happens to my delegators?** +Delegations are migrated as-is. Delegators who mapped their addresses will see their stakes in their Solana wallet. Unmapped delegations are held in escrow for claiming. + +**Can I still use my Arweave wallet for data uploads?** +Yes. Arweave wallets are still used for uploading data to Arweave via Turbo. The Solana wallet is only for protocol interactions (staking, observations, ArNS). + +**How do I check my observer's SOL balance?** +```bash +solana balance --url mainnet-beta +``` diff --git a/content/build/run-a-gateway/manage/ssl-certs.mdx b/content/build/run-a-gateway/manage/ssl-certs.mdx index 6c6863ef3..4800efbd4 100644 --- a/content/build/run-a-gateway/manage/ssl-certs.mdx +++ b/content/build/run-a-gateway/manage/ssl-certs.mdx @@ -41,7 +41,7 @@ Configure the token with these permissions: - **Zone → Zone → Read** - **Zone → DNS → Edit** -![Cloudflare API Token Configuration](https://arweave.net/GMzqNXNCQMSLqyt7SV7FrGOgCuGBeaO5qjRWibFkVBE) +![Cloudflare API Token Configuration](https://turbo-gateway.com/GMzqNXNCQMSLqyt7SV7FrGOgCuGBeaO5qjRWibFkVBE) diff --git a/content/build/run-a-gateway/manage/troubleshooting.mdx b/content/build/run-a-gateway/manage/troubleshooting.mdx index 8317d9851..b2e5d040e 100644 --- a/content/build/run-a-gateway/manage/troubleshooting.mdx +++ b/content/build/run-a-gateway/manage/troubleshooting.mdx @@ -91,7 +91,7 @@ When observing a gateway, there are two main pass/fail tests. "Ownership" and "A -Once you edit your .env file, you need to "rebuild" your gateway for the changes to take effect. As of release 3, every time you start your gateway with `docker-compose` it is automatically rebuilt. So all you need to do is shut your gateway down and restart it. +Once you edit your .env file, you need to "rebuild" your gateway for the changes to take effect. As of release 3, every time you start your gateway with `docker compose` it is automatically rebuilt. So all you need to do is shut your gateway down and restart it. @@ -111,7 +111,7 @@ This was a common issue prior to release #3, when Redis caching was introduced t -The first thing you should check if your gateway is not resolving ArNS names is that you have `ARNS_ROOT_HOST` set in your .env file. If not, set it to your domain name used for the gateway. For example, `ARNS_ROOT_HOST=arweave.dev`. +The first thing you should check if your gateway is not resolving ArNS names is that you have `ARNS_ROOT_HOST` set in your .env file. If not, set it to your domain name used for the gateway. For example, `ARNS_ROOT_HOST=turbo-gateway.com`. Once this value is set, restart your gateway for the changes to take effect. @@ -210,7 +210,7 @@ If you continue to have issues, you can check the [official certbot instructions -- This is normal. If a gateway fails to resolve an arns name within 3 seconds, it will fall back to a trusted gateway (arweave.net by default) to help resolve the name. +- This is normal. If a gateway fails to resolve an arns name within 3 seconds, it will fall back to a trusted gateway (turbo-gateway.com by default) to help resolve the name. @@ -244,17 +244,17 @@ Manual observations may be run on a gateway at any time buy using the [Network P 2. Select the gateway you are interested in from the list of gateways 3. Click on the "Observe" button in the top right corner of the page. -![Diagram](https://arweave.net/0G52dTWe65abQ6qDGvI99ERAaGU7DHR9srimJXnYRGA) +![Diagram](https://turbo-gateway.com/0G52dTWe65abQ6qDGvI99ERAaGU7DHR9srimJXnYRGA) 4. Click on the "Run Observation" button in the bottom right corner of the page. -![Diagram](https://arweave.net/A_B_58rufQ0Pj4ri0AKuC0DJn61u5ayO5ONWpkMerQw) +![Diagram](https://turbo-gateway.com/A_B_58rufQ0Pj4ri0AKuC0DJn61u5ayO5ONWpkMerQw) Two randomly selected ArNS names will be entered automatically in the "ArNS names" field to the left of the "Run Observation" button. These can be changed, or additional ArNS names can be added to the list before running the observation. The Manual observation will run the same checks as the observer, and will display the results on the right side of the page. -![Diagram](https://arweave.net/vgRXfbx4fa47qGDpjndq128VCHl1wajKaq464KeA0Qg) +![Diagram](https://turbo-gateway.com/vgRXfbx4fa47qGDpjndq128VCHl1wajKaq464KeA0Qg) ### Accessing the Observer Report @@ -265,7 +265,7 @@ The simplest way to access an observer report is via the [Network Portal](https: 3. In the Observation window, select the epoch you are interested in. This will display a list of the observers that failed the gateway for that epoch. 4. Click on the "View Report" button to the right any observer on that list. This will display the entire report that observer generated. -![Diagram](https://arweave.net/ynbxYU_8xLRaU1D6a_LMoUq00roWwsMKgr-xrsDE0Sk) +![Diagram](https://turbo-gateway.com/ynbxYU_8xLRaU1D6a_LMoUq00roWwsMKgr-xrsDE0Sk) 5. Locate the gateway you are interested in in the report, and click on that row. This will display the report for that gateway. @@ -275,9 +275,9 @@ The observer report will display a list of checked ArNS names, and a reason if t #### Timeout awaiting 'socket', or Timeout awaiting 'connect' -![Diagram](https://arweave.net/_GupbMa-EW_wiCD201MuOkDQrLT0MXTfxDXhLSDmyh4) +![Diagram](https://turbo-gateway.com/_GupbMa-EW_wiCD201MuOkDQrLT0MXTfxDXhLSDmyh4) -![Diagram](https://arweave.net/0WkzxdyN-9hJfv0pSiTs0Ozg_wqFvE3-OWlgzYPimtU) +![Diagram](https://turbo-gateway.com/0WkzxdyN-9hJfv0pSiTs0Ozg_wqFvE3-OWlgzYPimtU) This failure means that the observer was unable to connect to the gateway when it tried to check the ArNS name. There are lots of reasons why this might happen, many of them unrelated to the gateway itself. If an observer report has a small number of these failures, among a larger number of successful checks, it is unlikely to be an issue with the gateway. @@ -303,7 +303,7 @@ This failure means that the gateway's SSL certificate has expired. Obtaining a n #### dataHashDigest mismatch -![Diagram](https://arweave.net/xXe0bHne--0JJv-HRf5HT9R1V1UbzaOh2AxvAdQZhjg) +![Diagram](https://turbo-gateway.com/xXe0bHne--0JJv-HRf5HT9R1V1UbzaOh2AxvAdQZhjg) This failure means that the gateway did respond to a resolution request, but the data it returned did not match the data that was expected. This could be due to a number of reasons, including: @@ -315,7 +315,7 @@ A gateway will not return fraudulent data unless that operator intentionally rew #### Response code 502 (Bad Gateway) -![Diagram](https://arweave.net/NBQsYUKP6IZt_rYg77QXgzwUUPvimFGXCQqtesbW1_I) +![Diagram](https://turbo-gateway.com/NBQsYUKP6IZt_rYg77QXgzwUUPvimFGXCQqtesbW1_I) This failure means that the observer was able to connect to the gateway's network, but the reverse proxy returned a 502 error. This is almost always a reverse proxy issue. Ensure that the gateway's reverse proxy is running, and that it is configured to forward requests to the gateway. @@ -325,7 +325,7 @@ It is also possible that the gateway itself is not running at all. Check Docker #### Response code 503 (Service Unavailable) -![Diagram](https://arweave.net/7eFKSm-cs81-aJ_H4xkolR2nSlxl5tYWXJdFTei8Dbs) +![Diagram](https://turbo-gateway.com/7eFKSm-cs81-aJ_H4xkolR2nSlxl5tYWXJdFTei8Dbs) This failure means that the observer was able to connect to the gateway's network, but the reverse proxy was unable to forward the request to the gateway. It differs from the 502 error in that the reverse proxy is likely able to see that the gateway is running, but is unable to communicate with it. This is often a temporary issue, caused by the gateway not being able to handle a heavy load of requests, or the gateway being in the process of restarting. If this failure occurs once or twice in a report, it is likely a temporary issue and should not be considered an issue with the gateway. However, when this failure occurs persistently, particularly for every ArNS name checked on the report, it is likely that the gateway may have crashed. @@ -333,7 +333,7 @@ Manually restarting the gateway can likely resolve the issue. #### connect EHOSTUNREACH -![Diagram](https://arweave.net/O-uG-yGm5bNxjw2ADH_yBjOcGo-ZEiFym8GeFZZNueY) +![Diagram](https://turbo-gateway.com/O-uG-yGm5bNxjw2ADH_yBjOcGo-ZEiFym8GeFZZNueY) This failure means that the observer was unable to connect to the gateway at all. The connection was either refused, or the gateway was not able to find a target based on the domain name's DNS records. @@ -341,19 +341,19 @@ This is almost always an issue with DNS records or local network configuration. #### getaddrinfo ENOTFOUND -![Diagram](https://arweave.net/WJDwW0NM29uKC-9puvhXK_n75vgFXLpa6VKFVMhRsLQ) +![Diagram](https://turbo-gateway.com/WJDwW0NM29uKC-9puvhXK_n75vgFXLpa6VKFVMhRsLQ) This is another DNS related issue. Likely, the gateway does not have a valid DNS record either for the top level domain or the required wildcard subdomain. Having this failure occur once or twice in a report could mean that the DNS server being used by the observer is having temporary issues and should not be considered an issue with the gateway. However, when this failure occurs persistently, particularly for every ArNS name checked on the report, it is likely that the gateway's DNS records are not set, or are misconfigured. #### Hostname/IP does not match certificate's altnames: Host: \. is not in the cert's altnames: DNS:\ -![Diagram](https://arweave.net/HfOfpAYm811dWFPNQC7bANEvjGVK4ch3kO7K7qMN9qs) +![Diagram](https://turbo-gateway.com/HfOfpAYm811dWFPNQC7bANEvjGVK4ch3kO7K7qMN9qs) This failure means that the observer's SSL certificate does not match the gateway's domain name. This is almost always an issue with the gateway's SSL certificate. This most likely occurred because the gateway's operator did not update the gateway's SSL certificate when the gateway's domain name was changed. Obtaining a new SSL certificate and updating the gateway's reverse proxy configuration to use the new certificate is the only solution to this issue. #### write EPROTO \:error:\:SSL routines:ssl3_read_bytes:tlsv1 unrecognized name:\:SSL alert number 112 -![Diagram](https://arweave.net/Hbip_ZmqmFN8-uXijw1aylyYp1YllwgyZTNAcsPCxSg) +![Diagram](https://turbo-gateway.com/Hbip_ZmqmFN8-uXijw1aylyYp1YllwgyZTNAcsPCxSg) This failure almost always means that the gateway operator did not properly obtain SSL certificates for the gateway's wildcard subdomain. Obtaining a new SSL certificate and updating the gateway's reverse proxy configuration to use the new certificate is the only solution to this issue. @@ -385,13 +385,13 @@ Up to 50 gateways are chosen as observers per epoch. If there are more than 50 g -- There is a 90 day locking period when withdrawing stake, either from delegated stake or operator stake on your gateway. This locking period can be skipped, for a fee. The fee starts at 50% of the withdrawal amount, and goes down over time. If you selected instant withdrawal, you paid the fee to skip the locking period. +- There is a 30 day locking period when withdrawing delegated stake or excess operator stake from your gateway. This locking period can be skipped, for a fee. The fee starts at 50% of the withdrawal amount, and goes down over time. If you selected instant withdrawal, you paid the fee to skip the locking period. The required network-join stake follows the separate network leave process. -- The minimum operator stake for gateways (10,000 ARIO) cannot be instantly withdrawn, it is subject to the full 90 day locking period, and withdrawal can only be started by removing your gateway from the network. +- The minimum operator stake for gateways (20,000 ARIO) cannot be withdrawn while the gateway is active. It can only be returned through the network leave process, and it is unrecoverable if the gateway is pruned for sustained poor performance. diff --git a/content/build/run-a-gateway/manage/verification-headers.mdx b/content/build/run-a-gateway/manage/verification-headers.mdx new file mode 100644 index 000000000..38f34cb1a --- /dev/null +++ b/content/build/run-a-gateway/manage/verification-headers.mdx @@ -0,0 +1,207 @@ +--- +title: "Verification & Trust Headers" +description: "Understanding the HTTP response headers that ar.io gateways use to communicate data verification status, integrity, and trust to clients" +--- + +import { Callout } from "fumadocs-ui/components/callout"; +import { Card, Cards } from "fumadocs-ui/components/card"; +import { Shield, Globe, Search, Cpu } from "lucide-react"; + +Ar.io gateways attach HTTP response headers to every data response that communicate whether the data has been verified against the Arweave base layer, whether it has reached finality, and whether the source is trusted. These headers enable clients to make informed trust decisions without relying solely on the gateway operator's reputation. + +## Trust & Verification Headers + +Every data response from an ar.io gateway includes the following headers: + +| Header | Values | Description | +|--------|--------|-------------| +| `X-AR-IO-Verified` | `true` / `false` | `true` only when data is served from local cache **and** has been verified against the Arweave base layer. Data streamed from network sources is marked `false` even if the gateway's database indicates prior verification, because the hash cannot be confirmed in-flight during streaming. | +| `X-AR-IO-Stable` | `true` / `false` | `true` when the data exists beyond Arweave's maximum fork depth, indicating finality. Once stable, the data cannot be reorganized out of the chain. | +| `X-AR-IO-Trusted` | `true` / `false` | `true` when the data was retrieved from a source the operator has configured as trusted (via `TRUSTED_GATEWAY_URL` or `TRUSTED_GATEWAYS_URLS`). | +| `Content-Digest` | `sha-256=:base64:` | SHA-256 hash of the response body in [RFC 9530](https://www.rfc-editor.org/rfc/rfc9530.html) format. Only set on cache hits and HEAD requests, since the hash cannot be computed during network streaming. Enables direct body integrity verification by clients. | +| `X-AR-IO-Digest` | base64url string | The raw SHA-256 hash of the content in base64url encoding. Set alongside `Content-Digest`. | +| `ETag` | `"hash"` | Content hash for HTTP conditional requests (`If-None-Match`). Enables 304 Not Modified responses for efficient caching. | +| `X-AR-IO-Data-Id` | base64url string | The Arweave transaction ID or data item ID of the content being served. | +| `X-AR-IO-Hops` | number | The number of inter-gateway hops the request traversed before reaching this gateway. | +| `X-Cache` | `HIT` / `MISS` | Whether the data was served from the gateway's local cache or fetched from the network. | + +### Why X-AR-IO-Verified Can Be False for Verified Data + +A common source of confusion: data may be verified in the gateway's database but still return `X-AR-IO-Verified: false`. This happens when the data is streamed from a network source rather than served from local cache. During streaming, the gateway cannot compute a hash of the data in-flight to confirm it matches the verified hash. Only when the data is served from local cache can the gateway guarantee that the bytes being sent match what was previously verified. + +``` +# Cache hit - verified data served from local storage +X-AR-IO-Verified: true +X-Cache: HIT +Content-Digest: sha-256=:4ROTs2lTPAKbr8Y41WrjHu+2q+7S+m+yTuO7fAUzZI4=: + +# Network fetch - same data, but hash can't be confirmed in-flight +X-AR-IO-Verified: false +X-Cache: MISS +# Content-Digest is NOT set (can't hash during streaming) +``` + +## ANS-104 Bundle Headers + +For data items served from [ANS-104 bundles](https://specs.turbo-gateway.com/?tx=fFJEsPOLDJKRfNgaeaRPlfSvcNtHp0Y3JVxD-HHzmwA), gateways include additional headers in two categories. + +### Position Headers + +These headers describe the data item's location within its parent bundle, allowing clients to independently verify inclusion: + +| Header | Description | +|--------|-------------| +| `X-AR-IO-Root-Transaction-Id` | The root Arweave transaction containing the bundle | +| `X-AR-IO-Data-Item-Offset` | Byte offset of this data item within the bundle | +| `X-AR-IO-Data-Item-Size` | Size of the data item in bytes | +| `X-AR-IO-Data-Item-Data-Offset` | Byte offset of the data payload within the data item | +| `X-AR-IO-Root-Item-Offset` | Offset of the root item in the bundle | +| `X-AR-IO-Root-Item-Size` | Size of the root item | + +### Metadata Headers + +ANS-104 fields are exposed at the HTTP layer so clients can read data item tags and owner information directly from response headers without parsing the binary format: + +| Header | Description | +|--------|-------------| +| `X-Arweave-Owner` | The raw owner public key of the data item | +| `X-Arweave-Owner-Address` | The data item signer's wallet address (derived from the owner key) | +| `X-Arweave-Tag-*` | One header per tag, with the tag name as the suffix (e.g., `X-Arweave-Tag-Content-Type`) | +| `X-Arweave-Tags-Truncated` | Set when the tag set exceeds a configurable byte budget, indicating partial tag exposure | + +## Client-Side Verification + +Clients can verify gateway responses at multiple levels, from lightweight header checks to full cryptographic verification: + +### Body Integrity + +Compare the `Content-Digest` header (when present) against a locally computed SHA-256 hash of the response body. For Arweave content, the `X-AR-IO-Data-Id` is itself a content hash - clients can walk the Arweave or ANS-104 signature chain from this ID to verify the content. + +```javascript +// Verify Content-Digest in a browser +const response = await fetch('https://gateway.example/raw/TX_ID'); +const body = await response.arrayBuffer(); +const hash = await crypto.subtle.digest('SHA-256', body); +const base64Hash = btoa(String.fromCharCode(...new Uint8Array(hash))); + +const digestHeader = response.headers.get('Content-Digest'); +// Header format: sha-256=:base64hash=: +// Extract the hash between the colons +const match = digestHeader?.match(/sha-256=:(.+?):/); +const verified = match?.[1] === base64Hash; +console.log('Integrity check:', verified); +``` + +### Cross-Gateway Comparison + +Query the same data from multiple gateways and compare their response headers. If `X-AR-IO-Verified: true` and `Content-Digest` values match across independent gateways, the data is authentic with high confidence. + +### Wayfinder Verification + +The [Wayfinder](/learn/wayfinder) protocol provides automated client-side routing and verification across the gateway network. It supports configurable strategies including balanced (random), fastest ping, and static gateway selection, with built-in response verification. + +## HTTP Message Signatures (RFC 9421) + +Gateways can opt in to signing response headers using [RFC 9421 HTTP Message Signatures](https://www.rfc-editor.org/rfc/rfc9421.html). When enabled, an Ed25519 sub-key signs all trust-relevant headers at response time, producing a `Signature` and `Signature-Input` header pair. This provides cryptographic proof that a specific gateway operator produced a given response. + +### Enabling HTTP Signatures + +Add the following to your gateway's `.env` file: + +```bash +# Enable response signing +HTTPSIG_ENABLED=true +``` + +On startup, the gateway automatically generates an Ed25519 key at `data/keys/httpsig.pem` if one doesn't exist. The key also functions as a valid Solana address. + +#### Optional: Attestation + +To link your signing key to your on-chain identity, configure an observer wallet. The gateway will create an RSA-signed attestation linking the Ed25519 key to your Arweave wallet and upload it to Arweave: + +```bash +# Observer wallet for attestation (Arweave address) +OBSERVER_WALLET=your-arweave-address +WALLETS_PATH=wallets + +# Automatically upload attestation to Arweave (default: true) +HTTPSIG_UPLOAD_ATTESTATION=true + +# Optional: include staked gateway address in attestation +AR_IO_WALLET=your-gateway-address +``` + +Place your Arweave wallet JWK file in the `WALLETS_PATH` directory. The attestation is cached to `data/keys/httpsig-attestation.json` and only regenerated if the key or wallet changes. + +### Configuration Reference + +| Variable | Default | Description | +|----------|---------|-------------| +| `HTTPSIG_ENABLED` | `false` | Enable RFC 9421 response signing | +| `HTTPSIG_KEY_FILE` | `data/keys/httpsig.pem` | Path to Ed25519 private key (auto-generated if missing) | +| `HTTPSIG_BIND_REQUEST` | `true` | Include request method and path in signature (prevents replay) | +| `OBSERVER_WALLET` | - | Arweave wallet address for attestation signing | +| `WALLETS_PATH` | `wallets` | Directory containing Arweave wallet JWK files | +| `HTTPSIG_UPLOAD_ATTESTATION` | `true` | Upload attestation to Arweave automatically | + +### Verifying It's Working + +Once enabled, check the `/ar-io/info` endpoint to confirm: + +```bash +curl -s https://your-gateway.example/ar-io/info | jq '.httpsig' +``` + +The response includes the public key, key ID, Solana address, and attestation details (if configured). Signed responses will include `Signature` and `Signature-Input` headers on all trust-relevant data responses. + +### What Gets Signed + +The gateway signs all trust-relevant response headers but **not the response body**. Body integrity is achieved through the signed `X-AR-IO-Data-Id` (which is a content hash) and the `Content-Digest` header. Header-only signing preserves streaming performance for large responses. + +**Signed headers** (when present on the response): +- Trust headers: `X-AR-IO-Data-Id`, `X-AR-IO-Verified`, `X-AR-IO-Stable`, `X-AR-IO-Trusted` +- Bundle headers: `X-AR-IO-Root-Transaction-Id`, `X-Arweave-Owner-Address`, `X-Arweave-Tags-Truncated` +- ArNS headers: `X-ArNS-Name`, `X-ArNS-Resolved-Id`, `X-ArNS-TTL-Seconds`, `X-ArNS-Process-Id` +- Chunk headers: `X-Arweave-Chunk-Data-Root`, `X-Arweave-Chunk-Tx-Id`, `X-AR-IO-Chunk-Source-Type` +- Content headers: `Content-Type`, `Content-Digest` +- All `X-Arweave-Tag-*` headers (dynamically matched) +- Request components (when `HTTPSIG_BIND_REQUEST=true`): `@method`, `@path`, `@status` + +Only responses containing at least one trust-relevant header are signed. Non-data responses (health checks, info endpoint, errors) are not signed. + +### Verification Chain + +Clients can verify signed responses through the following chain: + +1. **Signature verification** - The public key is embedded in the `Signature-Input` header's `keyid` parameter and is verifiable via the Web Crypto API in modern browsers. +2. **Identity verification** - If attestation is configured, follow the attestation chain from the Ed25519 signing key to the operator's Arweave RSA wallet to confirm the signer is a registered gateway. The attestation is available at `/ar-io/info` and on Arweave. +3. **Body integrity** - Compare the `Content-Digest` header against a locally computed hash, or walk the Arweave signature chain from the signed `X-AR-IO-Data-Id`. + +## Related + + + } + /> + } + /> + } + /> + } + /> + diff --git a/content/build/run-a-gateway/quick-start.mdx b/content/build/run-a-gateway/quick-start.mdx index dd2066308..a2fa70d03 100644 --- a/content/build/run-a-gateway/quick-start.mdx +++ b/content/build/run-a-gateway/quick-start.mdx @@ -107,17 +107,21 @@ Ready to run a gateway with your own domain name and SSL certificates? Follow th **Add configuration:** ```bash - GRAPHQL_HOST=arweave.net + GRAPHQL_HOST=turbo-gateway.com GRAPHQL_PORT=443 START_HEIGHT=1000000 RUN_OBSERVER=true ARNS_ROOT_HOST= - AR_IO_WALLET= - OBSERVER_WALLET= + AR_IO_WALLET= + OBSERVER_WALLET= ``` + + `AR_IO_WALLET` and `OBSERVER_WALLET` must be Solana public keys (base58 Ed25519). The observer address must be unique — no other gateway can use the same observer. **Both wallets need SOL** — the observer pays for `save_observations` instructions, and the operator pays for cranker instructions when `ENABLE_EPOCH_CRANKING=true` (`tally_weights`, `prescribe_epoch`, `distribute_epoch`, `close_epoch`). Budget ~0.5 SOL each to cover several months of activity. + + **Supply Observer Wallet Keyfile:** - Save your wallet keyfile as `.json` in the `wallets` directory. + Save your Solana keypair JSON file as `.json` in the `wallets` directory. By default, the Observer will use [Turbo Credits](https://docs.ardrive.io/docs/turbo/credits) to pay for uploading reports to Arweave. This allows reports under 100kb to be uploaded for free, but larger reports will fail if the Observer wallet does not contain Credits. Including `REPORT_DATA_SINK=arweave` in your `.env` file will configure the Observer to use AR tokens instead of Turbo Credits, without any free limit. @@ -295,7 +299,7 @@ Ready to run a gateway with your own domain name and SSL certificates? Follow th **Create .env file with this content:** ```bash - GRAPHQL_HOST=arweave.net + GRAPHQL_HOST=turbo-gateway.com GRAPHQL_PORT=443 START_HEIGHT=0 RUN_OBSERVER=true @@ -435,7 +439,7 @@ Your gateway is running! Now you can: } /> } - arrow > Buy Turbo Credits with fiat or crypto diff --git a/content/build/upload/encryption.mdx b/content/build/upload/encryption.mdx index fa6a4f266..f3aafd379 100644 --- a/content/build/upload/encryption.mdx +++ b/content/build/upload/encryption.mdx @@ -160,7 +160,6 @@ The [Arweave File System (ArFS)](/build/advanced/arfs) protocol provides optiona href="https://ardrive.io" title="Try ArDrive Web App" icon={} - arrow > Use the ArDrive web app for easy encrypted file storage using ArFS. diff --git a/content/build/upload/index.mdx b/content/build/upload/index.mdx index 4280a72f9..0560aea37 100644 --- a/content/build/upload/index.mdx +++ b/content/build/upload/index.mdx @@ -31,60 +31,19 @@ There are multiple ways to upload data to Arweave. Each has its own attributes a } - description={ - <> -

- Production-ready bundling service with enterprise features -

-
-
- Credit cards, AR, ETH, SOL, MATIC -
-
- Free uploads under 100 KiB -
-
- Automatic retry & confirmation -
-
- - } + description="Production-ready bundling service with enterprise features • Pay with credit cards, AR, ETH, SOL, MATIC • Free uploads under 100 KiB • Automatic retry & confirmation" href="/build/upload/bundling-services" icon={Turbo} /> -

No-code solution for personal and business files

-
-
- Drag-and-drop interface -
-
- End-to-end encryption -
-
- Powered by Turbo -
-
- - } + description="No-code solution for personal and business files • Drag-and-drop interface • End-to-end encryption • Powered by Turbo" href="https://app.ardrive.io" icon={ArDrive} /> -

Raw protocol access for advanced use cases

-
-
• AR tokens required
-
• Manual transaction handling
-
• Complex setup needed
-
- - } + description="Raw protocol access for advanced use cases • AR tokens required • Manual transaction handling • Complex setup needed" href="https://arweave.org" icon={Arweave} /> diff --git a/content/build/upload/manifests.mdx b/content/build/upload/manifests.mdx index 14ad5a364..fe8d829ec 100644 --- a/content/build/upload/manifests.mdx +++ b/content/build/upload/manifests.mdx @@ -77,14 +77,14 @@ A resolver, typically an ar.io gateway, resolves URLs requesting content based o ### Example Usage -Assume the manifest above is uploaded to Arweave with the transaction ID `UyC5P5qKPZaltMmmZAWdakhlDXsBF6qmyrbWYFchRTk`. The below table shows https requests to the ar.io gateway `arweave.net`: +Assume the manifest above is uploaded to Arweave with the transaction ID `UyC5P5qKPZaltMmmZAWdakhlDXsBF6qmyrbWYFchRTk`. The below table shows https requests to the ar.io gateway `turbo-gateway.com`: | Request Path | Manifest Path | Data served from txID | | ---------------------------------------------------------------------------- | ------------- | ------------------------------------------- | -| https://arweave.net/UyC5P5qKPZaltMmmZAWdakhlDXsBF6qmyrbWYFchRTk | index | cG7Hdi_iTQPoEYgQJFqJ8NMpN4KoZ-vH_j7pG4iP7NI | -| https://arweave.net/UyC5P5qKPZaltMmmZAWdakhlDXsBF6qmyrbWYFchRTk/index.html | index.html | cG7Hdi_iTQPoEYgQJFqJ8NMpN4KoZ-vH_j7pG4iP7NI | -| https://arweave.net/UyC5P5qKPZaltMmmZAWdakhlDXsBF6qmyrbWYFchRTk/js/style.css | js/style.css | 3zFsd7bkCAUtXUKBQ4XiPiQvpLVKfZ6kiLNt2XVSfoV | -| https://arweave.net/UyC5P5qKPZaltMmmZAWdakhlDXsBF6qmyrbWYFchRTk/foobar | fallback | iXo3LSfVKVtXUKBzfZ4d7bkCAp6kiLNt2XVUFsPiQvQ | +| https://turbo-gateway.com/UyC5P5qKPZaltMmmZAWdakhlDXsBF6qmyrbWYFchRTk | index | cG7Hdi_iTQPoEYgQJFqJ8NMpN4KoZ-vH_j7pG4iP7NI | +| https://turbo-gateway.com/UyC5P5qKPZaltMmmZAWdakhlDXsBF6qmyrbWYFchRTk/index.html | index.html | cG7Hdi_iTQPoEYgQJFqJ8NMpN4KoZ-vH_j7pG4iP7NI | +| https://turbo-gateway.com/UyC5P5qKPZaltMmmZAWdakhlDXsBF6qmyrbWYFchRTk/js/style.css | js/style.css | 3zFsd7bkCAUtXUKBQ4XiPiQvpLVKfZ6kiLNt2XVSfoV | +| https://turbo-gateway.com/UyC5P5qKPZaltMmmZAWdakhlDXsBF6qmyrbWYFchRTk/foobar | fallback | iXo3LSfVKVtXUKBzfZ4d7bkCAp6kiLNt2XVUFsPiQvQ | ## Creating Manifests with Turbo @@ -223,7 +223,7 @@ Defines the URL paths that a manifest can resolve to. Each path maps to a specif Ar.io gateways support relative path routing, making it easy to develop and maintain websites hosted on Arweave. Instead of using fully qualified URLs: ```html - + ``` You can use relative paths: diff --git a/content/build/upload/turbo-credits.mdx b/content/build/upload/turbo-credits.mdx index 1a53e22af..ae85cf053 100644 --- a/content/build/upload/turbo-credits.mdx +++ b/content/build/upload/turbo-credits.mdx @@ -51,11 +51,10 @@ Top up any address of a [supported wallet type](#supported-wallet-types) by payi | Network | Tokens | | ----------- | -------------- | | Arweave | [AR](https://www.coingecko.com/en/coins/arweave) | -| AO | [ARIO](https://www.coingecko.com/en/coins/ar-io-network) | -| Base | [ARIO](https://basescan.org/token/0x138746adfA52909E5920def027f5a8dc1C7EfFb6), [ETH](https://www.coingecko.com/en/coins/ethereum), [USDC](https://basescan.org/token/0x833589fcd6edb6e08f4c7c32d4f71b54bda02913), [ARIO](https://basescan.org/token/0x138746adfA52909E5920def027f5a8dc1C7EfFb6)| +| Solana | [ARIO](https://www.coingecko.com/en/coins/ar-io-network), [SOL](https://www.coingecko.com/en/coins/solana) | +| Base | [ARIO](https://basescan.org/token/0x138746adfA52909E5920def027f5a8dc1C7EfFb6), [ETH](https://www.coingecko.com/en/coins/ethereum), [USDC](https://basescan.org/token/0x833589fcd6edb6e08f4c7c32d4f71b54bda02913) | | Ethereum | [ETH](https://www.coingecko.com/en/coins/ethereum), [USDC](https://etherscan.io/token/0xa0b86991c6218b36c1d19d4a2e9eb0ce3606eb48) | | Polygon | [POL](https://www.coingecko.com/en/coins/polygon), [USDC](https://polygonscan.com/address/0x3c499c542cef5e3811e1192ce70d8cc03d5c3359) | -| Solana | [SOL](https://www.coingecko.com/en/coins/solana) | | Cosmos | [KYVE](https://www.coingecko.com/en/coins/kyve-network) | ### Supported Payment Wallets @@ -63,17 +62,15 @@ Top up any address of a [supported wallet type](#supported-wallet-types) by payi | Network Type | Signature Types | Supported Wallets | | ------------ | -------------- | ------------------------------------------------------ | | Arweave | RSA | JWK Keyfile, [Wander](https://www.wander.app/), [Beacon](https://getbeaconapp.com/) | -| AO | RSA | JWK Keyfile, [Wander](https://www.wander.app/), [Beacon](https://getbeaconapp.com/) | -| AO | ECDSA (secp256k1) | JSON Keystore File, [MetaMask](https://metamask.io/), [Rainbow](https://rainbow.me/), [Brave](https://brave.com/wallet/), [WalletConnect](https://walletconnect.com/), [Privy](https://www.privy.io/) | | EVM | ECDSA (secp256k1) | JSON Keystore File, [MetaMask](https://metamask.io/), [Rainbow](https://rainbow.me/), [Brave](https://brave.com/wallet/), [WalletConnect](https://walletconnect.com/), [Privy](https://www.privy.io/) | | Solana | ED25519 | JSON Keypair File, [Phantom](https://phantom.com/), [Solflare](https://www.solflare.com/) | | Cosmos | secp256k1 | Keyfile | ### Purchasing Interfaces -#### Turbo App +#### Console App -Visit [turbo.ar.io](https://turbo.ar.io) to purchase credits with fiat or crypto using a connected wallet. +Visit [console.ar.io](https://console.ar.io) to purchase credits with fiat or crypto using a connected wallet. #### ArDrive App @@ -116,7 +113,7 @@ Direct API integration is available at [payment.ardrive.io/api-docs](https://pay #### Payment Interfaces By Token -| Payment Method | Turbo SDK | Turbo CLI | Turbo API | Turbo App | ArDrive App | x402 | +| Payment Method | Turbo SDK | Turbo CLI | Turbo API | Console App | ArDrive App | x402 | | ---------------------------- | --------- | --------- | --------- | --------- | ----------- | ---- | | **Fiat (credit/debit card)** | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | | **AR** | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | @@ -124,7 +121,7 @@ Direct API integration is available at [payment.ardrive.io/api-docs](https://pay | **ETH (Base)** | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | | **SOL** | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | | **POL** | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | -| **ARIO (AO)** | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | +| **ARIO (Solana)** | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | | **ARIO (Base)** | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | | **USDC (L1)** | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | | **USDC (Base)** | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | @@ -139,7 +136,7 @@ JIT payments are available via the Turbo SDK, CLI, or direct HTTP API integratio | Token / Currency | Network | JIT Supported | | ---------------------- | -------- | ------------- | -| ARIO | AO | ✅ | +| ARIO | Solana | ✅ | | ARIO | Base | ✅ | | SOL | Solana | ✅ | | ETH | Base | ✅ | @@ -182,7 +179,7 @@ See the [Turbo SDK documentation](/sdks/turbo-sdk) and [HTTP API docs](https://p Turbo Credits can be shared with other users to fund their uploads while maintaining total control over who may spend them, how many they may spend, and for how long they can do so. This is perfect for organizations, collaborations, and teams. -Credits can be shared via the Turbo App at [turbo.ar.io](https://turbo.ar.io) or programmatically via the [Turbo CLI](https://github.com/ardriveapp/turbo-sdk/?tab=readme-ov-file#cli), [SDK](/sdks/turbo-sdk), or [API](https://payment.ardrive.io/api-docs). +Credits can be shared via the Console App at [console.ar.io](https://console.ar.io) or programmatically via the [Turbo CLI](https://github.com/ardriveapp/turbo-sdk/?tab=readme-ov-file#cli), [SDK](/sdks/turbo-sdk), or [API](https://payment.ardrive.io/api-docs). ### How Credit Sharing Works @@ -208,9 +205,9 @@ Credits can be shared via the Turbo App at [turbo.ar.io](https://turbo.ar.io) or | Currency | Top Up Fee Percentage | | ----------------- | --------------------- | -| ARIO (AO or Base) | NO FEE | +| ARIO (Solana or Base) | NO FEE | | KYVE | 50% | -| All others | 23.4% | +| All others | 35% | ## Getting Started @@ -218,27 +215,25 @@ Credits can be shared via the Turbo App at [turbo.ar.io](https://turbo.ar.io) or Ready to start using Turbo Credits? Choose your path: - } - arrow > Buy credits instantly with credit cards or crypto -} -> - Learn how to upload data with your new credits - + } + > + Learn how to upload data with your new credits + -{" "} -}> - Integrate credit sharing and advanced features - + }> + Integrate credit sharing and advanced features + +The plugin is currently alpha. Its proof format, verification flow, and API surface are stable enough for evaluation, but production deployments should use dedicated wallets, explicit signing keys, monitoring, and ar.io Verify attestations. +
+ + +Verifiable AI builds on ar.io's broader verification model for gateway claims, client-side verification, and network accountability. For the network-level concepts, see [Verification and Accountability](/learn/verification). + + +## Architecture + +The lifecycle is: + +1. Train a model in MLflow and anchor the run. +2. Register the model and anchor the registry event. +3. Load the model through an integrity-checking wrapper before inference. +4. Anchor prediction proofs without publishing raw inputs or outputs. +5. Verify and attest the Arweave transactions through an ar.io gateway. + +Each lifecycle event produces a small signed envelope on Arweave. The envelope contains event metadata, a SHA-256 hash of canonical payload bytes, the previous proof hash for that event chain, the public key, and an Ed25519 signature. + +The canonical payload stays in MLflow as an `ario/payload.json` artifact, or as `ario/predictions//payload.json` for inference. A verifier fetches the envelope from an ar.io gateway, re-hashes the MLflow payload, re-derives the live MLflow record, and verifies the signature. + +Nothing sensitive needs to be written to Arweave. Source data, model artifacts, inputs, outputs, metrics, and traces remain in your MLflow store. Arweave stores the commitment that lets an auditor prove whether those records changed after anchoring. + +## What you will build + + + } + /> + } + /> + } + /> + + +## Verification checks + +All verification paths reduce to the same core checks: + +1. **Proof found**: the signed envelope is retrievable from an ar.io gateway for the recorded transaction ID. +2. **Record matches**: MLflow's stored payload re-hashes to the envelope's `payload_hash`, and a separate live MLflow surface re-derives the same canonical bytes. +3. **Signature confirmed**: the envelope signature verifies against the embedded Ed25519 public key. +4. **ar.io attestation**: optional gateway-operator verification that can be signed independently by the operator. + +This gives teams a tamper-evident record for training, registration, promotion, and inference without making Arweave the place where private AI data lives. diff --git a/content/build/verifiable-ai/meta.json b/content/build/verifiable-ai/meta.json new file mode 100644 index 000000000..beecc91d7 --- /dev/null +++ b/content/build/verifiable-ai/meta.json @@ -0,0 +1,10 @@ +{ + "title": "Verifiable AI", + "icon": "ShieldCheck", + "defaultOpen": false, + "pages": [ + "training-provenance", + "model-decision-proofs", + "operator-attestations" + ] +} diff --git a/content/build/verifiable-ai/model-decision-proofs.mdx b/content/build/verifiable-ai/model-decision-proofs.mdx new file mode 100644 index 000000000..2f62b05ff --- /dev/null +++ b/content/build/verifiable-ai/model-decision-proofs.mdx @@ -0,0 +1,159 @@ +--- +title: "Model and Decision Proofs" +description: "Anchor MLflow model registration, promotion, and prediction proofs with ar-io-mlflow" +--- + +import { Callout } from "fumadocs-ui/components/callout"; +import { Steps, Step } from "fumadocs-ui/components/steps"; + +The model registry is where training provenance becomes runtime protection. `ar-io-mlflow` provides a drop-in MLflow client that anchors registration and promotion events, plus a `VerifiedModel` wrapper that checks model artifact integrity before loading the underlying pyfunc model. + +## Prerequisites + +- Completed [Training Provenance](/build/verifiable-ai/training-provenance) +- A model logged to MLflow with `ario_mlflow.anchor()` +- MLflow model registry access +- Optional `ARIO_MLFLOW_ARIO_VERIFY_URL` for ar.io gateway attestations + +## Anchor model registration + + + +### Register with ArioMlflowClient + +Use `ArioMlflowClient` where you would normally use `mlflow.tracking.MlflowClient`. Registration returns immediately; anchoring runs in a background thread. + +```python +from ario_mlflow import ArioMlflowClient + +client = ArioMlflowClient() + +model_version = client.create_model_version( + name="credit-risk-scorer", + source=f"runs:/{run_id}/model", +) + +client.wait_for_anchor( + "registration", + "credit-risk-scorer", + str(model_version.version), + timeout=30, +) + +status = client.anchor_status( + "registration", + "credit-risk-scorer", + str(model_version.version), +) +print(status) +``` + +The status is one of `anchoring`, `anchored`, `signed`, `failed`, or `unknown`. + + + +### Verify the model version + +```bash +MLFLOW_TRACKING_URI=file:///absolute/path/to/mlruns \ +ar-io-mlflow verify model credit-risk-scorer/1 +``` + +Model verification re-hashes the registered model artifact, checks the registration proof, and confirms that the proof chains back to the training event where possible. + + + +### Audit lineage + +Use the audit command when you want a registry-oriented view of the proof chain. + +```bash +MLFLOW_TRACKING_URI=file:///absolute/path/to/mlruns \ +ar-io-mlflow audit credit-risk-scorer/1 +``` + + + +## Guard inference with VerifiedModel + +`VerifiedModel` verifies artifact integrity before loading the model. If the model bytes in MLflow were swapped after registration, the constructor raises `IntegrityError` before user model code can execute. + +```python +from ario_mlflow import IntegrityError, VerifiedModel + +try: + model = VerifiedModel("models:/credit-risk-scorer/1") +except IntegrityError as exc: + # Treat this as a security incident in production. + raise RuntimeError("Registered model artifact failed verification") from exc + +result = model.predict([78000, 0.18, 0.22, 72, 745]) + +print("decision_id:", result.decision_id) +print("proof_status:", result.proof_status) +print("tx_id:", result.tx_id) +``` + +Prediction anchoring is asynchronous. The prediction returns as soon as the model has produced an output, and the proof upload completes in the background. + +When you need the transaction ID before responding to a caller, wait explicitly: + +```python +result.wait_for_anchor(timeout=10) +print(result.tx_id, result.anchor_error) +``` + +## Verify an inference proof + +Each `VerifiedModel.predict()` call writes prediction proof data to MLflow trace tags and `ario/predictions//payload.json`. + +```bash +MLFLOW_TRACKING_URI=file:///absolute/path/to/mlruns \ +ar-io-mlflow verify trace +``` + +The verification flow checks: + +1. The prediction envelope exists on ar.io for the recorded transaction ID. +2. The prediction payload artifact hashes to the envelope's `payload_hash`. +3. The trace tag mirror, `ario.payload_json`, re-derives the same canonical payload. +4. The envelope signature is valid. +5. Optional ar.io Verify attestation passes if configured. + +## What gets written + +On model versions, the client writes tags such as: + +- `ario.artifact_verified` +- `ario.registration_tx` +- `ario.promotion_tx` +- `ario.arweave_url` + +On prediction traces, `VerifiedModel` writes tags such as: + +- `ario.decision_id` +- `ario.model_name` +- `ario.model_version` +- `ario.input_hash` +- `ario.output_hash` +- `ario.payload_hash` +- `ario.proof_status` +- `ario.prediction_tx` +- `ario.payload_json` + +Raw inference inputs and outputs are not written to Arweave by the plugin. The proof stores hashes and lineage metadata. + + +Verification proves that the model artifact and recorded decision metadata match what was anchored. It does not prove that the model is accurate, unbiased, or semantically correct for the task. + + +## Failure behavior + +- Registration and promotion still succeed if the background anchor fails. Inspect `anchor_status()` and alert on `signed` or `failed`. +- `VerifiedModel(...)` fails closed on artifact hash mismatch by raising `IntegrityError`. +- `predict()` returns even if proof anchoring later fails. Inspect `result.proof_status` and `result.anchor_error`. +- If registration anchoring has not completed when a model is loaded, early predictions may chain to `GENESIS` instead of the model version registration transaction. + +## Next steps + +Continue to [Operator Attestations](/build/verifiable-ai/operator-attestations) to add independent gateway verification, PDFs, attestations, and batch audits. diff --git a/content/build/verifiable-ai/operator-attestations.mdx b/content/build/verifiable-ai/operator-attestations.mdx new file mode 100644 index 000000000..6cfd4c243 --- /dev/null +++ b/content/build/verifiable-ai/operator-attestations.mdx @@ -0,0 +1,182 @@ +--- +title: "Operator Attestations" +description: "Run the ar.io Verify sidecar to independently verify Arweave data and issue operator-signed attestations" +--- + +import { Callout } from "fumadocs-ui/components/callout"; +import { Steps, Step } from "fumadocs-ui/components/steps"; + +[`ar-io-verify`](https://github.com/ar-io/ar-io-verify) is a verification sidecar for ar.io gateways. It fetches data through a gateway, reconstructs the cryptographic proof, reports the strongest verification level reached, and can sign the result as an operator attestation. + +For verifiable AI, this gives an independent party a way to verify the envelopes written by `ar-io-mlflow` without trusting the training or inference service. + +For the broader network model behind gateway claims, client-side checks, and economic accountability, see [Verification and Accountability](/learn/verification). + +## Verification levels + +| Level | Name | What it proves | +| --- | --- | --- | +| 1 | Existence confirmed | The transaction exists on Arweave and is associated with a confirmed block. | +| 2 | Partially verified | The sidecar downloaded the bytes and computed a matching SHA-256 fingerprint. | +| 3 | Verified | The sidecar verified the original Arweave or ANS-104 signature against the signed data. | + +When the operator configures a wallet, the sidecar signs the verification result with RSA-PSS SHA-256. That attestation can be checked later using the operator public key. + +## Deploy the sidecar + + + +### Start from the reference repo + +```bash +git clone https://github.com/ar-io/ar-io-verify.git +cd ar-io-verify +pnpm install +``` + + + +### Run locally for development + +In one terminal, run the server: + +```bash +pnpm run dev +``` + +In another terminal, run the standalone web UI: + +```bash +pnpm --filter verify-web run dev +``` + + + +### Run beside an ar.io gateway + +The Docker deployment expects an existing gateway on the `ar-io-network` Docker network. + +```bash +cd deploy +cp .env.example .env +``` + +Edit `.env`: + +```bash +VERIFY_IMAGE=ar-io-verify:local +GATEWAY_URL=http://ar-io-node-envoy-1:3000 +GATEWAY_HOST=example.com +PUBLIC_GATEWAY_URL=https://example.com +VERIFY_PORT=4001 +NODE_ENV=production +WALLET_FILE=/absolute/path/to/operator-wallet.json +``` + +Start the sidecar and nginx proxy: + +```bash +bash start.sh +``` + + + +## Connect ar-io-mlflow to ar.io Verify + +Set the plugin's Verify endpoint to the sidecar base URL exposed by your operator. + +```bash +export ARIO_MLFLOW_ARIO_VERIFY_URL=https://example.com +``` + +After this is configured, `ar-io-mlflow verify run`, `verify model`, and `verify trace` include the ar.io attestation row and write attestation metadata back to MLflow tags when available: + +- `ario.attestation_level` +- `ario.report_url` +- `ario.attested_by` +- `ario.attested_at` + + +The configured base URL must expose `GET /health` and `POST /api/v1/verify`. If you mount the sidecar under a path prefix, make sure those routes are still available under that same prefix. + + +## Single-transaction API + +Submit a transaction ID for verification: + +```bash +curl -X POST http://localhost:4001/api/v1/verify \ + -H "Content-Type: application/json" \ + -d '{"txId":""}' +``` + +Fetch the cached result: + +```bash +curl http://localhost:4001/api/v1/verify/ +``` + +Download the PDF certificate: + +```bash +curl -L http://localhost:4001/api/v1/verify//pdf \ + -o verification.pdf +``` + +Fetch the programmatic attestation: + +```bash +curl http://localhost:4001/api/v1/verify//attestation +``` + +The same API is used by the standalone `/verify/` UI and by ar.io Console integrations. + +## Batch verification jobs + +Use jobs for periodic audits, upload reconciliation, or verifier workflows that need to check many transaction IDs. + +```bash +curl -X POST http://localhost:4001/api/v1/jobs \ + -H "Content-Type: application/json" \ + -H "X-Tenant-Id: audit-team" \ + -H "Idempotency-Key: run-2026-05-14" \ + -d '{"txIds":["",""]}' +``` + +Check status: + +```bash +curl -H "X-Tenant-Id: audit-team" \ + http://localhost:4001/api/v1/jobs/ +``` + +Fetch the signed verification bundle: + +```bash +curl -H "X-Tenant-Id: audit-team" \ + http://localhost:4001/api/v1/jobs//report +``` + +The bundle is canonical JSON. To verify it offline, re-serialize the bundle without `signature` and `payloadHash`, compute SHA-256, compare that to `payloadHash`, then verify the RSA-PSS SHA-256 signature against `operatorPublicKey`. + +## Operations + +The sidecar exposes: + +- `GET /health` for liveness +- `GET /ready` for DB and gateway readiness +- `GET /metrics` for Prometheus metrics +- `GET /api-docs/` for Swagger UI + +Production deployments should restrict `/metrics` and administrative routes at the proxy layer, inject `X-Tenant-Id` from authenticated infrastructure for batch jobs, and treat the attestation wallet as a production signing secret. + +## Production checklist + +- Configure `GATEWAY_URL` to a gateway reachable from the sidecar container. +- Set `GATEWAY_HOST` to the public operator hostname included in attestations. +- Set `PUBLIC_GATEWAY_URL` when browser previews need a public gateway URL. +- Mount `WALLET_FILE` read-only if operator-signed attestations are required. +- Monitor readiness failures, gateway request failures, job duration, and verification outcomes. +- Configure `ARIO_MLFLOW_ARIO_VERIFY_URL` in MLflow verification environments. + +With these pieces in place, training, model, and inference proofs anchored by `ar-io-mlflow` can be independently verified and attested by an ar.io gateway operator. diff --git a/content/build/verifiable-ai/training-provenance.mdx b/content/build/verifiable-ai/training-provenance.mdx new file mode 100644 index 000000000..1d74891db --- /dev/null +++ b/content/build/verifiable-ai/training-provenance.mdx @@ -0,0 +1,178 @@ +--- +title: "Training Provenance" +description: "Anchor MLflow datasets, model artifacts, params, metrics, and training proofs to Arweave" +--- + +import { Callout } from "fumadocs-ui/components/callout"; +import { Steps, Step } from "fumadocs-ui/components/steps"; + +Training provenance starts with a simple question: can someone prove that a model's current MLflow record matches what was recorded when training finished? + +`ar-io-mlflow` answers that by writing a canonical payload to MLflow, signing a compact commitment envelope, and anchoring only that envelope to Arweave. The plugin can also anchor MLflow dataset records so downstream trainers can refer to an immutable dataset proof without publishing rows or raw files. + +## Prerequisites + +- Python 3.10 or newer +- MLflow 2.14 or newer +- A local or remote MLflow tracking store +- Optional Arweave JWK wallet for production identity + + +If no wallet is configured, the plugin generates one at `~/.ario-mlflow/wallet.json` and reuses it. That is convenient for evaluation. In production, set `ARIO_MLFLOW_ARWEAVE_WALLET` to a dedicated wallet file from your secrets manager. + + +## Install + +Install [`ar-io-mlflow`](https://github.com/ar-io/ar-io-mlflow) from source. + +```bash +git clone https://github.com/ar-io/ar-io-mlflow.git +cd ar-io-mlflow +pip install -e . +``` + +For the quickstart example, install scikit-learn as well: + +```bash +pip install scikit-learn +``` + +## Anchor a training run + + + +### Configure MLflow + +Point MLflow at the tracking store you want to use. This can be a local folder for development or your normal remote tracking URI. + +```python +from pathlib import Path + +import mlflow + +tracking_dir = Path("./mlruns").resolve() +mlflow.set_tracking_uri(f"file://{tracking_dir}") +mlflow.set_experiment("verifiable-ai") +``` + + + +### Log a dataset and model + +Log the dataset through MLflow so the plugin can include dataset provenance in the training proof. The dataset proof commits to the dataset name, source, digest, and schema hash, not to the row contents. + +```python +import mlflow.data +from sklearn.datasets import load_iris +from sklearn.linear_model import LogisticRegression +from sklearn.model_selection import train_test_split + +X, y = load_iris(return_X_y=True) +X_train, X_test, y_train, y_test = train_test_split( + X, + y, + test_size=0.2, + random_state=42, +) + +with mlflow.start_run() as run: + dataset = mlflow.data.from_numpy( + X_train, + targets=y_train, + source="https://archive.ics.uci.edu/dataset/53/iris", + name="iris-train", + ) + mlflow.log_input(dataset, context="training") + + model = LogisticRegression(max_iter=200).fit(X_train, y_train) + mlflow.log_params({"max_iter": 200, "random_state": 42}) + mlflow.log_metric("accuracy", model.score(X_test, y_test)) + mlflow.sklearn.log_model(model, name="model") +``` + + + +### Anchor the proof + +Call `ario_mlflow.anchor()` before the run exits. The call hashes model artifacts, writes `ario/payload.json`, signs the envelope, uploads it through Turbo, and writes `ario.*` tags back to MLflow. + +```python +import ario_mlflow + +with mlflow.start_run() as run: + # Fit model, log params, log metrics, log dataset, log model... + + result = ario_mlflow.anchor( + metadata={"service_name": "credit-risk-training"}, + ) + + print("run_id:", run.info.run_id) + print("payload_hash:", result["payload_hash"]) + print("training_tx:", result["tags"].get("ario.training_tx")) + print("verify_status:", result["tags"]["ario.verify_status"]) +``` + +If the Arweave upload fails, the run still succeeds and the envelope remains signed locally. In that case `ario.verify_status` is `signed` and `ario.training_tx` is absent. + + + +### Verify the run later + +Run the CLI against the same MLflow tracking store. + +```bash +MLFLOW_TRACKING_URI=file:///absolute/path/to/mlruns \ +ar-io-mlflow verify run +``` + +Verification checks that the envelope exists on ar.io, the MLflow payload still hashes to the anchored commitment, the live MLflow run still re-derives the same canonical bytes, and the Ed25519 signature is valid. + + + +## Standalone dataset proofs + +Dataset publishers can anchor a dataset proof without an active training run and hand the transaction ID to downstream teams. + +```python +import mlflow +import ario_mlflow + +dataset = mlflow.data.from_pandas( + df, + source="s3://example-bucket/training/q1.parquet", + name="credit-risk-q1", +) + +result = ario_mlflow.anchor(dataset=dataset) +print(result["tx_id"]) +``` + +This pattern is useful when a data platform team publishes approved datasets and model teams consume them later. The proof records an immutable commitment to the dataset descriptor, while the source data remains in S3, a lakehouse, or another controlled system. + +## What gets written + +On the MLflow run, the plugin writes tags such as: + +- `ario.enabled` +- `ario.version` +- `ario.public_key` +- `ario.verify_status` +- `ario.artifact_hash` +- `ario.payload_hash` +- `ario.training_tx` +- `ario.arweave_url` +- `ario.wallet_mode` + +It also writes `ario/payload.json` as the canonical payload artifact. Arweave receives only the compact signed envelope, usually hundreds of bytes rather than the source data or model artifact. + +## Production notes + +- Use a dedicated wallet per environment through `ARIO_MLFLOW_ARWEAVE_WALLET`. +- Set `ARIO_MLFLOW_SIGNING_KEY` explicitly if you need controlled key rotation. +- Configure `ARIO_MLFLOW_GATEWAYS` with at least two gateways for fetch fallback. +- Monitor runs where `ario.verify_status = signed`, because those were signed but not anchored. +- Keep MLflow artifacts backed up independently. Arweave preserves the envelope, but MLflow still holds the canonical payload used for full record matching. + +## Next steps + +After training proofs are anchored, continue to [Model and Decision Proofs](/build/verifiable-ai/model-decision-proofs) to anchor registry events and enforce artifact integrity before inference. diff --git a/content/glossary/index.mdx b/content/glossary/index.mdx index ca3fd569e..0b9e2a271 100644 --- a/content/glossary/index.mdx +++ b/content/glossary/index.mdx @@ -3,30 +3,82 @@ title: "Glossary" description: "Key terms and definitions for the ar.io ecosystem" --- -## AO Computer +## Ar.io Name System (ArNS) -AO (Actor Oriented) is a hyper-parallel computing platform built on Arweave that enables decentralized applications to run with unlimited computational capacity. AO provides the compute layer for ar.io's smart contracts and token operations. +A decentralized naming system in the ar.io network that maps human-readable names to Arweave transaction IDs and application/content routes. -## Winston +## Ar.io Name Token (ANT) -Winston is the smallest unit of Arweave's native AR token. One AR equals 10^12 Winston. +An NFT on Solana associated with a registered ArNS name. ANT ownership controls name-level configuration such as records, undernames, and transfer operations. -## Winston Credits (winc) +## CPI (Cross-Program Invocation) -Winston Credits (`winc`) are the unit used by Turbo to represent upload purchasing power. In Turbo contexts, receipt and balance fields often return values in `winc`. +The mechanism by which Solana programs call into each other. The ar.io architecture uses CPI for token operations — ario-gar and ario-arns call into ario-core for SPL Token transfers. -## Public Key +## Cranker + +A permissionless actor (bot or user) that submits transactions to drive lazy protocol operations, including the epoch pipeline and other state advancement tasks. Any participant can crank if protocol conditions are met. + +## Epoch Pipeline + +The 6-step process for observation, reward calculation, and distribution each epoch: create_epoch → tally_weights → prescribe_epoch → save_observations → distribute_epoch → close_epoch. + +## Gateway Address Registry (GAR) -A cryptographic key that can be shared publicly and is used to verify digital signatures or encrypt data. In the ar.io context, public keys are used to identify wallet addresses and verify transactions. +The protocol-level registry of joined gateways and their metadata. GAR powers gateway discovery and is a core input to observer selection, rewards, and network operations. + +## Metaplex Core + +The NFT standard used for Ar.io Name Tokens (ANTs) on Solana. Metaplex Core assets are tradeable on standard NFT marketplaces like Tensor and Magic Eden. ## Native Address -An address format that uses the raw public key bytes directly, without additional encoding or transformation. This is the most basic form of an Arweave address. +An address format that uses the raw public key bytes directly, without additional encoding or transformation. ## Normalized Address A standardized address format that ensures consistent representation across different systems and contexts. Normalized addresses help prevent issues with address matching and lookup operations. +## Observer + +A gateway selected to evaluate peer gateways during an epoch. Observers test assigned ArNS resolutions and submit reports used by the protocol for performance evaluation and rewards. + ## Optimistic Indexing A data indexing strategy where new data is immediately made available for queries while verification processes continue in the background. This approach improves performance while maintaining data integrity through eventual consistency. + +## PDA (Program Derived Address) + +Deterministic Solana account addresses derived from a program ID and a set of seeds. The ar.io protocol uses PDAs extensively — for gateways, delegations, withdrawals, vaults, ArNS records, ANT configs, and more. + +## Program ID + +The unique onchain address of a deployed Solana program. Ar.io protocol interactions target specific program IDs for instructions and state access. + +## Protocol Balance + +A protocol-controlled ARIO token pool that accumulates relevant inflows (for example, protocol fees) and serves as a source for incentive distributions according to protocol rules. + +## Public Key + +A cryptographic key that can be shared publicly and is used to verify digital signatures or encrypt data. In the ar.io context, Solana public keys (Ed25519, base58-encoded) are used to identify wallet addresses and verify transactions. + +## SPL Token + +Solana Program Library token standard. ARIO is implemented as an SPL Token on Solana, making it compatible with the full Solana wallet and DeFi ecosystem. + +## Transaction (Solana) + +A signed unit of interaction with Solana containing one or more instructions. Instructions in a transaction are processed atomically: all succeed, or the transaction fails without applying state changes. + +## Vault + +A protocol-level token lock with a start time, optional end time, and token balance. Vaults are used to hold ARIO over time and may be revocable depending on how they were created. + +## Winston + +Winston is the smallest unit of Arweave's native AR token. One AR equals 10^12 Winston. + +## Winston Credits (winc) + +Winston Credits (`winc`) are the unit used by Turbo to represent upload purchasing power. In Turbo contexts, receipt and balance fields often return values in `winc`. diff --git a/content/learn/(introduction)/index.mdx b/content/learn/(introduction)/index.mdx index c7bc1c3ca..39eb70295 100644 --- a/content/learn/(introduction)/index.mdx +++ b/content/learn/(introduction)/index.mdx @@ -85,7 +85,7 @@ Ar.io is a data infrastructure solution for long-term access. We help enterprise icon={} title="ArNS Registry" description="Register and manage permanent names for your applications" - href="https://arns.app" + href="https://arns.ar.io" /> } diff --git a/content/learn/(introduction)/meta.json b/content/learn/(introduction)/meta.json index 8057a8ab0..e1f4a61f9 100644 --- a/content/learn/(introduction)/meta.json +++ b/content/learn/(introduction)/meta.json @@ -1,5 +1,6 @@ { "title": "Introduction", + "icon": "Info", "defaultOpen": false, - "pages": ["what-is-arweave", "what-is-ario", "permaweb", "ans-104-bundles"] + "pages": ["what-is-arweave", "what-is-ario", "protocol-architecture", "solana-developers", "ans-104-bundles"] } diff --git a/content/learn/(introduction)/permaweb.mdx b/content/learn/(introduction)/permaweb.mdx deleted file mode 100644 index 5fe75aec7..000000000 --- a/content/learn/(introduction)/permaweb.mdx +++ /dev/null @@ -1,182 +0,0 @@ ---- -title: "What is the Permaweb?" -description: "Understand how the permaweb works through its four-layer architecture: storage, compute, gateways, and applications" -collapsible: false ---- - -import { Server, Database, Globe, Rocket } from 'lucide-react'; - -The **permaweb** is a decentralized, permanent layer of the internet where data, applications, and websites are stored forever and remain accessible through a global network of gateways. - -## How the Permaweb Works - -Unlike the traditional web where data can disappear when servers go offline, the permaweb creates a **permanent archive of human knowledge** through a multi-layer architecture: - -**Foundation**: Arweave blockchain provides immutable storage -**Computation**: AO and other platforms enable smart contracts and processing -**Access**: ar.io gateway network makes everything accessible globally -**Users**: Developers and users interact through familiar web interfaces - -This architecture ensures that once something is published to the permaweb, it remains accessible forever, creating a truly permanent internet. - -## Permaweb Network Architecture - -The permaweb operates through a layered system architecture where each component provides specialized services to create a permanent, decentralized internet. - -
-
- Applications & Users -
- Web Apps, dApps, Developers, End Users -
-
- -
-
- AO - AO -
- Decentralized Compute -
-
- -
- ar.io - ar.io -
- Decentralized Access -
-
-
- -
- Arweave - Arweave -
- Permanent Storage Foundation -
-
-
- -## Permaweb Architecture: How It All Connects - -The permaweb operates through a **four-layer architecture** where each layer serves a specific purpose in creating permanent, accessible internet infrastructure: - -### Layer 1: Permanent Storage (Arweave) -**Core Responsibility: Forever Data Preservation** - -Arweave's primary job is to **store data permanently** with mathematical guarantees: -- **Immutable storage** - Once written, data cannot be changed or deleted -- **Economic sustainability** - Endowment model ensures miners are paid to store data forever -- **Cryptographic verification** - Proof that data exists and hasn't been tampered with -- **Decentralized replication** - Data survives even if most miners go offline - -### Layer 2: Decentralized Compute (AO) -**Core Responsibility: Smart Contract Execution** - -AO's primary job is to **run programs that work with permanent data**: -- **Process execution** - Runs smart contracts and applications on permanent data -- **Message routing** - Enables communication between different processes -- **State management** - Maintains application state using permanent storage -- **Parallel computation** - Scales processing across multiple nodes - -### Layer 3: Decentralized Access (ar.io) -**Core Responsibility: Data Access & Discovery** - -Ar.io's primary job is to **make permanent data fast and accessible**: -- **Data retrieval** - Fetches and serves content from Arweave storage -- **Content indexing** - Organizes and catalogs data for search and discovery -- **ArNS resolution** - Converts human-readable names to Arweave transaction IDs -- **Performance optimization** - Caches popular content for faster access -- **Quality assurance** - Validates data integrity and provides reliable access - -### Layer 4: Applications & Users -**Core Responsibility: User Interface & Experience** - -Applications and users are responsible for **interacting with the permanent web**: -- **User interfaces** - Create familiar web experiences backed by permanent data -- **Data submission** - Upload new content and applications to the permaweb -- **Application logic** - Build decentralized apps using permanent storage and compute -- **Content consumption** - Browse, search, and interact with permanent web content - - - -## The Vision of the Permaweb - -The permaweb represents a fundamental shift from the ephemeral nature of today's internet to a permanent, censorship-resistant foundation for human knowledge and applications. By combining Arweave's immutable storage with AO's decentralized compute and ar.io's accessible gateway network, the permaweb creates an internet where data never disappears, applications run without central points of failure, and users maintain true ownership of their digital assets. - -This architecture enables a new generation of applications that can operate indefinitely without relying on traditional hosting services, where digital artifacts become truly permanent, and where the collective knowledge of humanity is preserved for future generations. The permaweb isn't just about storing data forever—it's about building a more resilient, equitable, and permanent foundation for the digital world. - -## Explore the Permaweb - - - } - /> - } - /> - } - /> - } - /> - diff --git a/content/learn/(introduction)/protocol-architecture.mdx b/content/learn/(introduction)/protocol-architecture.mdx new file mode 100644 index 000000000..c6bcf41f5 --- /dev/null +++ b/content/learn/(introduction)/protocol-architecture.mdx @@ -0,0 +1,135 @@ +--- +title: 'Protocol Architecture' +description: "Explore the technical architecture of the ar.io Solana programs and protocol components" +--- + +## Ar.io Protocol Architecture + +The ar.io protocol operates through four Solana programs (3+1) that work together via cross-program invocation (CPI). This architecture organizes protocol responsibilities into a coordinated set of programs with clear boundaries. + +For the deployed Solana mainnet program IDs and ARIO token mint, see [Mainnet Addresses](/learn/token#mainnet-addresses). + +```mermaid +graph TD + + subgraph CORE["ario-core"] + direction TB + BAL[SPL Token / Balances] + VAULTS[Vaults] + PRIMARY[Primary Names] + end + + subgraph GAR["ario-gar"] + direction TB + GW_REG[Gateway Registry] + STAKING[Staking & Delegation] + EPOCHS[Epochs & Rewards] + end + + subgraph ARNS["ario-arns"] + direction TB + NAME_REG[Name Registry] + DEMAND[Demand Factor] + PRICING[Pricing & Auctions] + end + + subgraph ANT["ario-ant"] + direction TB + NFT[Metaplex Core NFTs] + RECORDS[DNS-like Records] + CTRL[Controllers] + end + + CORE --> GAR + CORE --> ARNS + ARNS --> ANT + ARNS -.->|gateway data| GAR + + classDef coreProgram fill:#e3f2fd + classDef garProgram fill:#e8f5e8 + classDef arnsProgram fill:#fff3e0 + classDef antProgram fill:#f3e5f5 + + class BAL,VAULTS,PRIMARY coreProgram + class GW_REG,STAKING,EPOCHS garProgram + class NAME_REG,DEMAND,PRICING arnsProgram + class NFT,RECORDS,CTRL antProgram +``` + +## Programs + +### ario-core + +The core program manages the ARIO SPL Token, vaults, and primary names: + +- **SPL Token Operations**: ARIO is a standard SPL Token (6 decimals, 1 ARIO = 1,000,000 mARIO). Transfers, balances, and token accounts follow the SPL Token standard. +- **Vaults**: Time-locked token deposits for various purposes, including ecosystem programs and other protocol-managed incentives. + +Other programs (ario-gar, ario-arns) call into ario-core via CPI for all token operations. + +### ario-gar (Gateway Address Registry) + +Manages the network's gateway infrastructure, staking, delegation, and the epoch reward pipeline: + +- **Gateway Registry**: An onchain registry of network gateways. Each gateway is a PDA storing operator address, stake, observer address, settings, and performance stats. +- **Staking & Delegation**: Operator stakes, delegated stakes (separate PDA per gateway-delegator pair), withdrawals, redelegation, and allowlists. +- **Observer Address Uniqueness**: An ObserverLookup PDA enforces that no two gateways share the same observer address. +- **Epoch Pipeline**: A 6-step permissionless pipeline driven by [cranker bots](/learn/oip/epoch-pipeline): + 1. `create_epoch` — Initialize epoch, compute reward rate + 2. `tally_weights` — Batched weight computation + 3. `prescribe_epoch` — Select observers and prescribed names via weighted roulette + 4. `save_observations` — Observers submit pass/fail reports + 5. `distribute_epoch` — Batched reward distribution + 6. `close_epoch` — Reclaim rent from completed epoch accounts +- **Gateway Pruning**: Gateways that repeatedly fail observation are removed from the network and subject to stake slashing. + +### ario-arns (ArNS Registry) + +Manages the Ar.io Name System — name registration, leasing, pricing, and returned names: + +- **Name Registry**: An onchain registry of ArNS domains. Each name is an ArnsRecord PDA storing the owner, ANT mint address, lease type, and expiration. +- **Pricing**: Dynamic pricing via a DemandFactor PDA. +- **Returned Name Auctions**: When a name expires or is released, it enters a Dutch auction. +- **Validation**: Names must meet certain formatting criteria. +- **Cost Simulation**: `get_token_cost` view instruction available via `simulateTransaction` for fee estimation. + +### ario-ant (Ar.io Name Tokens) + +ANTs are [Metaplex Core](https://developers.metaplex.com/core) NFTs that represent ownership of ArNS names: + +- **NFT Standard**: Each ANT is a Metaplex Core asset — tradeable on Tensor, Magic Eden, and other NFT marketplaces. +- **DNS-like Records**: Each ANT stores routing records (AntRecord PDAs) mapping undernames to Arweave transaction IDs or IPFS CIDs, with configurable TTL values. +- **Controllers**: Controllers can manage records without holding the NFT itself. +- **Lazy Reconciliation**: When an ANT is transferred via a marketplace (outside the ar.io protocol), controllers are cleared on the next write operation, ensuring the new owner has full control. + +## State Model + +All protocol state is stored in Solana accounts using Program Derived Addresses (PDAs): + +| Account Type | Program | Derivation Seeds | Purpose | +|-------------|---------|-----------------|---------| +| ArioConfig | ario-core | `["config"]` | Global token configuration | +| Vault | ario-core | `["vault", owner, vault_id]` | Time-locked token deposit | +| PrimaryName | ario-core | `["primary_name", name_hash]` | Name → address mapping | +| PrimaryNameReverse | ario-core | `["primary_name_reverse", owner]` | Address → name reverse lookup | +| GatewayRegistry | ario-gar | `["gateway_registry"]` | Zero-copy gateway slot array | +| Gateway | ario-gar | `["gateway", operator]` | Individual gateway state | +| Delegation | ario-gar | `["delegation", gateway, delegator]` | Per-pair delegation | +| Withdrawal | ario-gar | `["withdrawal", owner, id]` | Pending withdrawal | +| ObserverLookup | ario-gar | `["observer_lookup", observer]` | Observer uniqueness check | +| Epoch | ario-gar | `["epoch", epoch_index]` | Epoch state and rewards | +| NameRegistry | ario-arns | `["name_registry"]` | Zero-copy name slot array | +| ArnsRecord | ario-arns | `["arns_record", name_hash]` | Individual name record | +| DemandFactor | ario-arns | `["demand_factor"]` | Current pricing multiplier | +| AntConfig | ario-ant | `["ant_config", mint]` | ANT metadata and settings | +| AntControllers | ario-ant | `["ant_controllers", mint]` | Controller list (max 10) | +| AntRecord | ario-ant | `["ant_record", mint, undername_hash]` | DNS-like routing record | + +## Security Model + +The protocol relies on a combination of Solana runtime guarantees, deterministic account ownership, and economic incentives: + +- **Signer and account checks**: Solana enforces transaction signatures, account ownership, and program execution rules. +- **Program Derived Addresses (PDAs)**: Protocol state lives in deterministic accounts controlled by the relevant ar.io programs. +- **Economic accountability**: Gateway operators stake ARIO, and repeated failure can lead to removal and slashing. +- **Permissionless verification**: Gateway observations and cranking keep network operations open and independently verifiable. diff --git a/content/learn/(introduction)/solana-developers.mdx b/content/learn/(introduction)/solana-developers.mdx new file mode 100644 index 000000000..d381741df --- /dev/null +++ b/content/learn/(introduction)/solana-developers.mdx @@ -0,0 +1,202 @@ +--- +title: "Ar.io for Solana Developers" +description: "A quick-start guide for Solana developers — map familiar concepts to ar.io and start building permanent applications" +--- + +import { Card, Cards } from 'fumadocs-ui/components/card'; +import { Upload, Globe, Server, Code } from 'lucide-react'; + +If you build on Solana, you already know most of what you need to use ar.io. This page maps familiar Solana concepts to ar.io equivalents and gets you to your first permanent deployment fast. + +## What ar.io Does + +Ar.io is the **access and naming layer for Arweave** — the permanent storage blockchain. It gives you: + +- **Permanent hosting** — Upload your frontend once, it lives forever. No hosting bills. +- **Human-readable URLs** — `yourapp.ardrive.net` instead of a 43-character transaction ID +- **Decentralized CDN** — Gateways worldwide serve your content, no single point of failure +- **On-chain naming** — ArNS names are Metaplex Core NFTs tradeable on Tensor/Magic Eden + +The ar.io protocol runs on Solana. You interact with it using your existing Solana wallet, SOL for fees, and ARIO tokens for protocol operations. + +## Concept Mapping + +If you know Solana, you already understand ar.io: + +| Solana Concept | ar.io Equivalent | Details | +|---|---|---| +| **SPL Token** | ARIO token | The protocol token. 6 decimals, 1B supply. Used for staking, ArNS names, delegation. | +| **Metaplex Core NFT** | ANT (Ar.io Name Token) | Each ArNS name is backed by an NFT. Trade on Tensor/Magic Eden. Holds DNS-like records. | +| **Anchor Program** | ario-core, ario-gar, ario-arns, ario-ant | Four programs that manage the protocol. Interact via SDK or CPI. | +| **PDA** | All protocol state | Gateways, delegations, vaults, name records, ANT configs — all PDAs. | +| **CPI** | Cross-program token ops | ario-gar and ario-arns call into ario-core for SPL Token transfers. | +| **Transaction fee (SOL)** | Same | Every protocol operation costs a small SOL fee (< 0.01 SOL). | +| **`@solana/kit`** | Signer creation | The SDK uses `@solana/kit` for keypair signing. | + +## What You Need + +| Requirement | Why | +|---|---| +| **Solana wallet** (Phantom, Solflare, Backpack) | Sign transactions for ArNS names, staking, records | +| **SOL** | Pay Solana transaction fees (< 0.01 SOL per operation) | +| **ARIO tokens** | Pay for ArNS names, staking. [Get ARIO tokens here](/token/get-ar.io-token) | +| **Node.js 18+** | Run the SDK and Turbo CLI | + +**What you DON'T need:** +- No Arweave wallet for protocol operations (staking, names, records) +- No AR tokens for ArNS or staking +- No special RPC — the SDK connects to Solana mainnet by default + + +You DO still need an Arweave wallet (or SOL via Turbo) to **upload data** to Arweave. Turbo accepts SOL directly — see the upload section below. + + +## Quick Start: 5 Minutes to Permanent + +### 1. Install + +```bash +npm install @ar.io/sdk @solana/kit @ardrive/turbo-sdk +``` + +### 2. Upload Your App to Arweave + +Use Turbo to upload your build folder. Turbo accepts SOL directly — no Arweave wallet needed: + +```typescript +import { TurboFactory } from '@ardrive/turbo-sdk'; + +// Authenticate with your Solana private key +const turbo = TurboFactory.authenticated({ + privateKey: bs58.encode(solanaSecretKey), + token: 'solana', +}); + +// Upload your build folder +const result = await turbo.uploadFolder({ + folderPath: './dist', + dataItemOpts: { + tags: [{ name: 'App-Name', value: 'my-cool-app' }], + }, +}); + +console.log('Manifest TX:', result.manifestResponse.id); +// This is the Arweave Transaction ID — your app is now permanent +``` + +### 3. Register an ArNS Name + +```typescript +import { ARIO } from '@ar.io/sdk'; +import { createKeyPairSignerFromBytes } from '@solana/kit'; +import fs from 'fs'; + +// Create signer from Solana keypair +const keypairBytes = new Uint8Array( + JSON.parse(fs.readFileSync('./solana-keypair.json', 'utf-8')), +); +const signer = await createKeyPairSignerFromBytes(keypairBytes); +const ario = ARIO.mainnet({ signer }); + +// Buy the name (mints an ANT as a Metaplex Core NFT) +const result = await ario.buyRecord({ + name: 'my-cool-app', + type: 'lease', + years: 1, +}); +``` + +### 4. Point Your Name to Your App + +```typescript +import { ANT } from '@ar.io/sdk'; + +// Get your ANT's mint address from the ArNS record +const record = await ario.getArNSRecord({ name: 'my-cool-app' }); + +// Initialize the ANT and set the record +const ant = ANT.init({ signer, processId: record.processId }); +await ant.setRecord({ + undername: '@', + transactionId: result.manifestResponse.id, // from the Turbo upload + ttlSeconds: 3600, +}); + +// Your app is now live at: +// https://my-cool-app.ardrive.net +// https://my-cool-app.turbo-gateway.com +// https://my-cool-app. +``` + +## How ArNS URLs Work + +Every ar.io gateway resolves ArNS names as subdomains: + +``` +https://my-cool-app.turbo-gateway.com → served by turbo-gateway.com gateway +https://my-cool-app.ardrive.net → served by ardrive.net gateway +https://my-cool-app.ardrive.net → served by ardrive.net gateway +``` + +All network gateways serve the same content. If one goes down, users access through another. Your app is truly decentralized. + +Undernames use underscores: +``` +https://docs_my-cool-app.ardrive.net → the "docs" undername +https://api_my-cool-app.ardrive.net → the "api" undername +``` + +## Costs at a Glance + +| Operation | Cost | +|---|---| +| Upload (< 100KB) | **Free** via Turbo | +| Upload (larger files) | Pay with SOL, ARIO, or fiat. [See pricing](https://console.ar.io) | +| ArNS name (1 year lease, 5+ chars) | ~200-2,500 ARIO depending on length × demand factor | +| ArNS name (permabuy, 5+ chars) | ~200-2,500 ARIO × demand factor | +| Set/update a record | SOL fee only (< 0.01 SOL) | +| Fetch data from a gateway | **Free** | +| Solana transaction fees | < 0.01 SOL per operation | + + +Use `ario.getTokenCost()` to check exact pricing before any purchase. ArNS prices adjust dynamically based on demand. + + +## What to Build + +Ar.io is ideal for: + +- **Permanent frontends** — Deploy your React/Next.js/Vue app with zero hosting costs forever +- **Immutable assets** — Store game assets, NFT metadata, or config files that can never be altered +- **Decentralized publishing** — Blog platforms, documentation sites, content archives +- **Censorship-resistant apps** — Apps that no single entity can take down +- **Version-controlled deployments** — Use undernames for staging, production, and rollbacks + +## Next Steps + + + } + /> + } + /> + } + /> + } + /> + diff --git a/content/learn/(introduction)/what-is-ario.mdx b/content/learn/(introduction)/what-is-ario.mdx index 24a3e71ba..a8e1aa88a 100644 --- a/content/learn/(introduction)/what-is-ario.mdx +++ b/content/learn/(introduction)/what-is-ario.mdx @@ -7,7 +7,9 @@ import { Code, Server, Globe, Database } from 'lucide-react'; Ar.io is the first permanent cloud network. -A decentralized infrastructure layer built on Arweave and AO, making permanent data accessible, discoverable, and usable. Think of it as the gateway to Arweave's permaweb, turning its tamper-proof storage into a fully functional, user-friendly ecosystem for apps, websites, and data. +A decentralized infrastructure layer built on Arweave for permanent data storage, with protocol execution on Solana. Think of it as the gateway to Arweave's permaweb, turning its tamper-proof storage into a fully functional, user-friendly ecosystem for apps, websites, and data. + +The permaweb is the permanent web of files, applications, and identities built on Arweave. Ar.io makes that permanent data usable by providing fast gateway access, indexing, human-readable names, and routing across a decentralized network.
@@ -29,9 +31,9 @@ A decentralized infrastructure layer built on Arweave and AO, making permanent d Ar.io operates a network of [gateways](/learn/gateways) — nodes that serve as entry points to Arweave’s data. These gateways fetch and deliver data quickly, supporting everything from static files to dynamic web apps. -### Arweave Name System (ArNS) +### Ar.io Name System (ArNS) -The [Arweave Name System (ArNS)](/learn/arns) is a decentralized naming system for Arweave. It allows users to register and resolve human-readable names to Arweave transaction IDs. +The [Ar.io Name System (ArNS)](/learn/arns) is a decentralized naming system for Arweave. It allows users to register and resolve human-readable names to Arweave transaction IDs. ### Data Access @@ -54,7 +56,7 @@ Ar.io builds on Arweave's permanent storage to create a decentralized, scalable - **Decentralized Gateways**: ar.io operates a network of gateways—nodes that serve as entry points to Arweave’s data. These gateways fetch and deliver data quickly, supporting everything from static files to dynamic web apps. -- **ArNS (Arweave Name Service)**: ar.io introduces decentralized domain names (e.g., yourname.arweave), mapping human-readable names to Arweave’s data IDs. This makes content easy to find and share, like URLs on the traditional web. +- **ArNS (Ar.io Name System)**: ar.io introduces decentralized domain names (e.g., yourname.arweave), mapping human-readable names to Arweave’s data IDs. This makes content easy to find and share, like URLs on the traditional web. - **Indexing and Querying**: ar.io enables efficient data indexing, allowing developers to search and retrieve specific content from Arweave’s vast storage without scanning the entire blockweave. diff --git a/content/learn/arns/ants.mdx b/content/learn/arns/ants.mdx index a79102532..d461d85bd 100644 --- a/content/learn/arns/ants.mdx +++ b/content/learn/arns/ants.mdx @@ -1,31 +1,35 @@ --- -title: "Arweave Name Tokens (ANTs)" -description: "Learn about Arweave Name Tokens (ANTs) - the ownership and control system for ArNS names" +title: "Ar.io Name Tokens (ANTs)" +description: "Learn about Ar.io Name Tokens (ANTs) - the ownership and control system for ArNS names" --- import { Card, Cards } from 'fumadocs-ui/components/card'; import { Calculator, FileText, Code } from 'lucide-react'; -To establish ownership of a record in the ArNS Registry, each record contains both a friendly name and a reference to an Arweave Name Token, ANT. Name Tokens are unique AO Computer based tokens/processes that give their owners the ability to update the Arweave Transaction IDs that their associated friendly names point to. +To establish ownership of a record in the ArNS Registry, each record contains both a friendly name and a reference to an Ar.io Name Token (ANT). ANTs are [Metaplex Core](https://developers.metaplex.com/core) NFTs on Solana that give their owners the ability to update the Arweave transaction IDs their associated friendly names point to. ## What is an ANT? -The ANT smart contract process is a standardized contract that implements the specific Arweave Name Process specification required by ar.io gateways who resolve ArNS names and their Arweave Transaction IDs. It also contains other basic functionality to establish ownership and the ability to transfer ownership and update the Arweave Transaction ID. +An ANT is a Metaplex Core NFT managed by the `ario-ant` Solana program. It implements the Ar.io Name Token specification required by ar.io gateways to resolve ArNS names to Arweave transaction IDs. The program handles record updates, controller management, metadata changes, and ownership reconciliation after transfers. -Name Tokens have an owner, who can transfer the token and control its modifiable settings. These settings include modifying the address resolution time to live (ttl) for each name contained in the ANT, and other settings like the ANT Name, Ticker, and an ANT Controller. +Name Tokens have an owner, who can transfer the token and control its modifiable settings. These settings include record targets, address-resolution TTL values, the ANT name and ticker, and controller permissions. ## Ownership and Control -The controller can only manage the ANT and set and update records, name, and the ticker, but cannot transfer the ANT. Note that ANTs are initially created in accordance with network standards by an end user who then has the ability to transfer its ownership or assign a controller as they see fit. +Controllers can manage records, name, and ticker settings, but cannot transfer the ANT or assign additional controllers. -Owners of names should ensure their ANT supports evolve ability if future modifications are desired. Loss of a private key for a permanently purchased name can result in the name being "bricked". +ANTs are minted in accordance with network standards by an end user who can transfer ownership or assign controllers as needed. Because ANT logic lives in the `ario-ant` program rather than inside each individual token, protocol-level updates can apply consistently across ANTs. -### Under_name Ownership +When an ANT is transferred through a marketplace outside the ar.io app, the `ario-ant` program reconciles ownership on the next interaction and clears previously assigned controllers so the new owner has clean control. -Undernames can have an `owner` set on them. This owner is empowered to set that undername as their primary name, can remove that undername as their primary name, and has full control over that Undername's metadata, such as: +Loss of a private key for a permanently purchased name can result in the name being inaccessible. -- Transaction Id - the data the record resolves to. -- TTL Seconds - the Time To Live in seconds the data is cached for by clients. +### Undername Ownership + +Undernames can have an `owner` set on them. This owner is empowered to set that undername as their primary name, remove that undername as their primary name, and control that undername's metadata, such as: + +- Transaction ID - the data the record resolves to. +- TTL seconds - the time to live in seconds the data is cached for by clients. - Owner - the owner of the record. - Description - the description of the record. - Display Name - the display name for the owner of the record. @@ -44,7 +48,7 @@ The table below indicates some of the possible interactions with the ArNS regist | Transfer ownership | ✔ | | | | | Add / remove controllers | ✔ | | | | | Approve/Remove Primary name | ✔ | | ✔ | | -| Reassign name to new ANT process | ✔ | | | | +| Reassign name to new ANT | ✔ | | | | | Return a permanent name | ✔ | | | | | Set records (pointers, record metadata) | ✔ | ✔ | ✔ | | | Update records, name, ticker | ✔ | ✔ | | | @@ -54,15 +58,17 @@ The table below indicates some of the possible interactions with the ArNS regist | Increase undernames | ✔ | ✔ | ✔ | ✔ | | Convert lease to permanent | ✔ | ✔ | ✔ | ✔ | -## Under_names +## Undernames -ANT owners and controllers can configure multiple subdomains for their registered ArNS name known as "under_names" or more easily written "undernames". These undernames are assigned individually at the time of registration or can be added on to any registered name at any time. +ANT owners and controllers can configure subdomains for their registered ArNS name, known as undernames. These undernames are assigned at registration or added later. -Under*names use an underscore "*" in place of a more typically used dot "." to separate the subdomain from the main ArNS domain. +Undernames use an underscore (`_`) in place of a dot (`.`) to separate the subdomain from the main ArNS domain. ## Secondary Markets -Secondary markets could be created by ecosystem partners that facilitate the trading of Name Tokens. Additionally, tertiary markets could be created that support the leasing of these friendly names to other users. Such markets, if any, would be created by third parties unrelated to and outside of the scope of this paper or control of the Foundation. +Since ANTs are standard Metaplex Core NFTs, they are tradeable on any compatible NFT marketplace, including **Tensor** and **Magic Eden**. When an ANT is sold on a marketplace, lazy reconciliation clears the existing controllers on the next write operation, ensuring the new owner gets clean control. + +Additionally, tertiary markets could be created that support the leasing of these friendly names to other users. ## Next Steps diff --git a/content/learn/arns/index.mdx b/content/learn/arns/index.mdx index a059d31a2..e6081d500 100644 --- a/content/learn/arns/index.mdx +++ b/content/learn/arns/index.mdx @@ -1,6 +1,6 @@ --- -title: "Arweave Name System (ArNS)" -description: "ArNS is a censorship-resistant naming system stored on Arweave, powered by ARIO tokens, enabled through ar.io gateway domains, and used to connect friendly domain names to permaweb apps, web pages, data, and identities." +title: "Ar.io Name System (ArNS)" +description: "ArNS is a censorship-resistant naming system powered by ARIO and ar.io gateways, connecting friendly names to permaweb apps, pages, data, and identities." --- import { Card, Cards } from "fumadocs-ui/components/card"; @@ -8,9 +8,9 @@ import { Globe, Key, DollarSign, ShoppingCart } from "lucide-react"; ## What is ArNS? -Arweave URLs and transaction IDs are long, difficult to remember, and occasionally categorized as spam. The Arweave Name System (ArNS) aims to resolve these problems in a decentralized manner. +Arweave URLs and transaction IDs are long, difficult to remember, and occasionally categorized as spam. The Ar.io Name System (ArNS) aims to resolve these problems in a decentralized manner. -ArNS is a **censorship-resistant naming system** stored on Arweave, powered by [ARIO tokens](/learn/token), enabled through [ar.io gateway](/learn/gateways) domains, and used to connect friendly domain names to permaweb apps, web pages, data, and identities. +ArNS is a **censorship-resistant naming system** powered by [ARIO tokens](/learn/token), enabled through [ar.io gateway](/learn/gateways) domains, and used to connect friendly domain names to permaweb apps, web pages, data, and identities. It's an open, permissionless, domain name registrar that doesn't rely on a single TLD. @@ -18,48 +18,27 @@ It's an open, permissionless, domain name registrar that doesn't rely on a singl This system works similarly to traditional DNS services, where users can purchase a name in a registry and DNS Name servers resolve these names to IP addresses. The system is flexible and allows users to purchase names permanently or lease them for a defined duration based on their use case. -With ArNS, the registry is stored permanently on Arweave via [AO](/glossary), making it immutable and globally resilient. This also means that apps and infrastructure cannot just read the latest state of the registry but can also check any point in time in the past, creating a "Wayback Machine" of permanent data. +With ArNS, the registry is managed on Solana by the `ario-arns` program, making ownership and registration state globally verifiable. Names are controlled by Ar.io Name Tokens (ANTs), which point to Arweave transaction IDs so gateways can route friendly names to permanent data. ```mermaid -graph TB - subgraph "ar.io Smart Contract" - Registry[ArNS Registry] - Registry --> Name1[ardrive] - Registry --> Name2[ao] - Registry --> Name3[gateways] - Registry --> NameN[...] - end - - subgraph "Arweave Name Token" - Name1 -.->|owned by| ANT1[Owner: 0x242424...
Record: @
Target: TxID_123] - Name2 -.->|owned by| ANT2[Owner: Zjgamagh...
Record: @
Target: TxID_456] - Name3 -.->|owned by| ANT3[Owner: Hboaalf...
Record: @
Target: TxID_789] - end - - subgraph "Arweave" - ANT1 -.->|points to| TX1[TxID_123
ArDrive App] - ANT2 -.->|points to| TX2[TxID_456
Email App] - ANT3 -.->|points to| TX3[TxID_789
Pages App] - end - - style Registry fill:#f9f,stroke:#333,stroke-width:3px,color:#333 - style ANT1 fill:#b3d9ff,stroke:#333,stroke-width:2px,color:#333 - style ANT2 fill:#b3d9ff,stroke:#333,stroke-width:2px - style ANT3 fill:#b3d9ff,stroke:#333,stroke-width:2px - style TX1 fill:#bfb,stroke:#333,stroke-width:2px,color:#333 - style TX2 fill:#bfb,stroke:#333,stroke-width:2px,color:#333 - style TX3 fill:#bfb,stroke:#333,stroke-width:2px,color:#333 +graph TD + User[User requests friendly name] --> Gateway[ar.io Gateway] + Gateway --> Registry[ArNS Registry
ario-arns] + Registry --> ANT[Ar.io Name Token
Metaplex Core NFT] + ANT --> Target[Arweave Transaction ID] + Gateway --> Content[Permaweb app, page, or data] + Target --> Content ``` ## Name Resolution -Users can register a name, like `ardrive`, within the ArNS Registry. Before owning a name, they must create an Arweave Name Token (ANT), an AO Computer based token and open-source protocol used by ArNS to track the ownership and control over the name. +Users can register a name, like `ardrive`, within the ArNS Registry. Ownership is represented by an Ar.io Name Token (ANT), a Metaplex Core NFT on Solana used by ArNS to track control over the name. ANTs allow the owner to set a mutable pointer to any type of permaweb data, like a page, app or file, via its Arweave transaction ID. -Each ar.io gateway acts as an ArNS Name resolver. They fetch the latest state of both the ArNS Registry and its associated ANTs from an AO compute unit (CU) and serve this information rapidly for apps and users. +Each ar.io gateway acts as an ArNS name resolver. Gateways fetch the latest state of both the ArNS Registry and associated ANTs from Solana and use that state to route users to the right Arweave transaction. -Ar.io gateways will also resolve that name as one of their own subdomains, e.g., `https://ardrive.arweave.net` and proxy all requests to the associated Arweave transaction ID. This means that ANTs work across all ar.io gateways that support them: `https://ardrive.ar-io.dev`, `https://ardrive.g8way.io/`, etc. +Ar.io gateways resolve names as gateway subdomains, e.g., `https://ardrive.ar.io`, and proxy requests to the associated Arweave transaction ID. This means an ArNS name can work across ar.io gateways that support ArNS. Users can easily reference these friendly names in their browsers, and other applications and infrastructure can build rich solutions on top of these ArNS primitives. @@ -68,10 +47,10 @@ sequenceDiagram participant User participant Gateway as ar.io Gateway participant Registry as ArNS Registry - participant ANT as Arweave Name Token + participant ANT as Ar.io Name Token participant Arweave - User->>Gateway: Request ardrive.arweave.net + User->>Gateway: Request ardrive.ar.io Gateway->>Registry: Query "ardrive" record Registry-->>Gateway: Returns ANT address Gateway->>ANT: Get target TxID @@ -88,10 +67,9 @@ sequenceDiagram - **Human-readable URLs** instead of complex transaction IDs - **Censorship-resistant** and decentralized -- **Permanent storage** on Arweave +- **Permanent data routing** to Arweave transaction IDs - **Cross-gateway compatibility** - works on all ar.io gateways -- **Historical data access** - check any point in time -- **Flexible ownership** - permanent or leased names +- **Flexible ownership** - lease names for a defined period or buy them permanently ## Explore ArNS @@ -103,7 +81,7 @@ sequenceDiagram icon={} /> } @@ -117,7 +95,7 @@ sequenceDiagram } /> diff --git a/content/learn/arns/meta.json b/content/learn/arns/meta.json index a5a92bd54..c1b5d242f 100644 --- a/content/learn/arns/meta.json +++ b/content/learn/arns/meta.json @@ -1,5 +1,6 @@ { - "title": "Arweave Name System (ArNS)", - "pages": ["name-registration", "ants", "pricing-model"], + "title": "Ar.io Name System (ArNS)", + "icon": "Globe", + "pages": ["name-registration", "ants", "pricing-model", "returned-names"], "defaultOpen": false } diff --git a/content/learn/arns/name-registration.mdx b/content/learn/arns/name-registration.mdx index a5cf8ffc2..75da146bb 100644 --- a/content/learn/arns/name-registration.mdx +++ b/content/learn/arns/name-registration.mdx @@ -3,31 +3,31 @@ title: "Name Registration" description: "Learn about registering ArNS names, including lease vs permanent options, validation rules, and the registration process" --- -There are two different types of name registrations that can be utilized based upon the needs of the user: +ArNS names can be registered as leases or permanent purchases, depending on how long the user needs the name and how much they want to commit up front. ## Registration Types ### Lease Registration -A name may be **leased on a yearly basis**. A leased name can have its lease extended or renewed but only up to a maximum active lease of **five (5) years** at any time. +A name may be **leased on a yearly basis**. Leases lower the barrier to entry, support temporary projects, and allow inactive names to eventually return to public availability. ### Permanent Registration (Permabuy) -A name may be **purchased for an indefinite duration** with no expiration date. +A name may be **purchased for an indefinite duration** with no expiration date. This is useful for long-lived apps, identities, and data references that should remain associated with a stable friendly name. Registering a name requires spending ARIO tokens corresponding to the name's character length and purchase type. ## Name Registry -The ArNS Registry is a list of all registered names and their associated ANT Process IDs. Key rules embedded within the smart contract include: +The ArNS Registry is a list of registered names and their associated ANT mint addresses, managed by the `ario-arns` Solana program. Key rules embedded within the protocol include: - **Genesis Prices**: Set within the contract as starting conditions - **Dynamic Pricing**: Varies based on name length, purchase type (lease vs buy), lease duration, and current Demand Factor -- **Name Records**: Include a pointer to the Arweave Name Token process identifier, lease end time (if applicable), and undername allocation +- **Name Records**: Include a pointer to the Ar.io Name Token (ANT) mint address, lease end time (if applicable), and undername allocation - **Reassignment**: Name registrations can be reassigned from one ANT to another - **Lease Extension**: Anyone with available ARIO Tokens can extend any name's active lease - **Lease to Permanent Buy**: Anyone with available ARIO Tokens can convert a name's lease to a permanent buy -- **Undername Capacity**: Additional undername capacity can be purchased for any actively registered name +- **Undername Capacity**: Additional undername capacity can be purchased for actively registered names - **Name Removal**: Name records can only be removed from the registry if a lease expires, or a permanent name is returned to the protocol ## Name Validation Rules @@ -38,19 +38,21 @@ All names registered must meet the following criteria: 2. **Dash placement**: Dashes cannot be leading or trailing characters 3. **Single character domains**: Dashes cannot be used in single character domains 4. **Length limits**: 1 character minimum, 51 characters maximum -5. **Reserved names**: Cannot be an invalid name predesignated to prevent unintentional use/abuse such as `www` +5. **43-character prohibition**: Names exactly 43 characters long are prohibited to prevent Arweave transaction ID collisions +6. **Lowercase enforcement**: Names must be lowercase at submission +7. **Reserved names**: Cannot be an invalid name predesignated to prevent unintentional use/abuse such as `www` ## Lease Management ### Lease Expirations -When a lease term ends, there is a **grace period of two (2) weeks** where the lease can be renewed before it fully expires. If this grace period elapses, the name is considered expired and returns to the protocol for public registration. Once expired, a name's associated undername registrations and capacity also expire. +When a lease term ends, there is a grace period where the lease can be renewed or converted to a permanent purchase before it fully expires. If this grace period elapses, the name is considered expired and returns to the protocol for public registration. Once expired, a name's associated undername registrations and capacity also expire. -A recently expired name's registration shall be priced subject to the "Returned Name Premium" mechanics. +A recently expired name enters a **Returned Name Dutch Auction**, where it starts at a premium and decays back toward standard pricing over time. Revenue from returned name purchases is split between the protocol and the previous owner. ### Lease to Permabuy Conversions -An actively leased name may be converted to a permanent registration. The price for this conversion shall be treated as if it were a new permanent name purchase. +An actively leased name may be converted to a permanent registration. The price for this conversion is treated as if it were a new permanent name purchase. This functionality allows users to transition from leasing to permanent ownership based on changing needs and available resources. It generates additional protocol revenue through conversion fees, contributing to the ecosystem's financial health and reward system. @@ -62,10 +64,14 @@ When a permanent name is returned, the name is subject to a "Returned Name Premi ## Primary Names -The Arweave Name System (ArNS) supports the designation of a "Primary Name" for users, simplifying how Arweave addresses are displayed across applications. A Primary Name is a user-friendly alias that replaces complex wallet addresses, making interactions and profiles easier to manage and identify. +The Ar.io Name System (ArNS) supports the designation of a "Primary Name" for users, simplifying how wallet addresses are displayed across applications. A Primary Name is a user-friendly alias that can replace complex wallet addresses, making interactions and profiles easier to manage and identify. -Users can set one of their owned ArNS names as their Primary Name, subject to a small fee. This allows applications to use a single, human-readable identifier for a wallet, improving user experience across the network. +Users can set one of their owned ArNS names as their Primary Name. The fee is equivalent to the cost of a single undername on a 51-character name of the same purchase type, adjusted by the current Demand Factor. + +Only one Primary Name can be set per wallet, and the same name cannot be the Primary Name for more than one wallet. The base name's ANT owner can remove any Primary Name set on one of its undernames. + +This allows applications to use a single, human-readable identifier for a wallet, improving user experience across the network. ## Next Steps -Now that you understand name registration, learn about [Arweave Name Tokens (ANTs)](/learn/arns/ants) to see how ownership and control work, or explore the [Pricing Model](/learn/arns/pricing-model) to understand how costs are calculated. +Now that you understand name registration, learn about [Ar.io Name Tokens (ANTs)](/learn/arns/ants) to see how ownership and control work, or explore the [Pricing Model](/learn/arns/pricing-model) to understand how costs are calculated. diff --git a/content/learn/arns/pricing-model.mdx b/content/learn/arns/pricing-model.mdx index 27add14ff..e1ac3d94c 100644 --- a/content/learn/arns/pricing-model.mdx +++ b/content/learn/arns/pricing-model.mdx @@ -7,198 +7,51 @@ import { Globe, Database, Code } from "lucide-react"; ## Addressing Variable Market Conditions -The future market landscape is unpredictable, and ar.io's smart contract is designed to be immutable, operating without governance or manual intervention. Using a pricing oracle to fix name prices relative to a stable currency is not viable due to the infancy of available solutions and reliance on external dependencies. +The future market landscape is unpredictable, and ArNS is designed to adapt without relying on a centralized pricing oracle. Instead of fixing name prices to an external currency, the protocol adjusts pricing based on network activity and registration demand. -To address these challenges, ArNS is self-contained and adaptive, with name prices reflecting network activity and market conditions over time. +This keeps ArNS self-contained while still allowing name prices to respond to changing market conditions over time. -To achieve this, ArNS incorporates: +ArNS pricing is built from a few core ideas: -1. A **dynamic pricing model** that adjusts fees using a "Demand Factor" based on ArNS purchase activity -2. A **Returned Name Premium (RNP)** system that applies a timed, descending multiplier to registration prices for names that have recently expired or been returned to the protocol +1. **Name length**: Shorter, more memorable names generally cost more. +2. **Registration type**: Names can be leased for a defined period or purchased permanently. +3. **Demand Factor**: A protocol multiplier adjusts prices based on recent ArNS activity. +4. **Returned Name Premium (RNP)**: Recently expired or returned names re-enter the market through a descending premium window. -This approach ensures that name valuations adapt to market conditions within the constraints of an immutable, maintenance-free smart contract framework. +This approach lets ArNS remain predictable enough for users while still adapting to namespace demand. -You can view current live pricing at [ArNS.app](https://arns.ar.io/#/prices) to see these formulas in action. +You can view current live pricing at [arns.ar.io](https://arns.ar.io/#/prices) to see these formulas in action. ## Key Definitions -- **Protocol Revenue:** Accumulated ARIO tokens from name registrations, lease extensions, and under_name sales -- **Period (P):** The time unit for DF adjustments, equivalent to one (1) day, denoted in milliseconds -- **n:** The current period indicator -- **Price:** The cost for permabuy or lease of a name -- **Under_names:** Subdomain equivalents, denoted by an underscore "\_" prefixing the base domain +- **Demand Factor:** A protocol multiplier that adjusts prices based on recent registration activity. +- **Base Fee:** The starting price for a name before dynamic adjustments. +- **Lease:** A time-limited registration. +- **Permabuy:** A permanent registration. +- **Undername:** A subdomain-style record written with an underscore (`_`) instead of a dot (`.`). +- **Protocol Revenue:** ARIO collected from ArNS actions, such as name registrations, lease extensions, and undername purchases. ## Dynamic Pricing Model -ArNS employs an adaptive pricing model to balance market demand with pricing fairness for name registration within the network. This model integrates static and dynamic elements, adjusting prices based on name length and purchase options like leasing, permanent acquisition, and undername amounts. +ArNS uses an adaptive model to balance name availability, demand, and long-term sustainability. Prices are influenced by name length, whether the name is leased or bought permanently, undername capacity, and the current Demand Factor. -### Core Pricing Components +The Demand Factor changes over time based on protocol activity. When demand is high, it can increase prices; when demand is low, it can decrease prices. If demand remains low for long enough, the protocol can step base fees downward so names stay accessible. -#### Base Registration Fee (BRF) - -The fundamental price for names, varying by character length, adjusted periodically. - -#### Genesis Registration Fee (GRF) - -The starting price for name registrations varies by character length. This is superseded by Base Registration Fees as the protocol evolves. - -**Table: Genesis Registration Fees** - -| Name Length | Fee (ARIO) | -| ----------- | ---------- | -| 1 | 1,000,000 | -| 2 | 200,000 | -| 3 | 20,000 | -| 4 | 10,000 | -| 5 | 2,500 | -| 6 | 1,500 | -| 7 | 800 | -| 8 | 500 | -| 9 | 400 | -| 10 | 350 | -| 11 | 300 | -| 12 | 250 | -| 13-51 | 200 | - -#### Demand Factor (DF) - -A global price multiplier, reflecting namespace demand, adjusted each period based on revenue trends. - -**DF Mechanics:** - -- **Intent:** The Demand Factor adjusts based on protocol revenue comparison to the Revenue Moving Average (RMA) -- **Increase DF:** When recent revenue is higher than or equal to (but non-zero) the RMA, the DF increases by 5.0% -- **Decrease DF:** When recent revenue is less than the RMA or both are zero, the DF decreases by 1.5% -- **Maximum DF Value:** Unbounded -- **Minimum DF Value:** 0.5 -- **Starting Demand Factor:** 1 (initial value at network launch) - -#### Revenue Moving Average (RMA) - -The average of protocol revenue from the past seven (7) periods. - -### Pricing Formulas - -#### Adjusted Registration Fee (ARF) - -```math -ARF = BRF × DF -``` - -#### Annual Fee - -```math -Annual ~Fee = ARF × 20% -``` - -#### Lease Pricing - -- **Lease Registration Price:** - -```math -Lease ~Price = ARF + (Annual ~Fee × Years) -``` - -- **Lease Extension/Renewal Price:** - -```math -Lease ~Renewal ~Price = Annual ~Fee × Years (max 5 years) -``` - -- **Grace period:** Two (2) weeks - -#### Permanent Purchases - -- **Permabuy Price:** - -```math -Permabuy ~Price = ARF + (Annual Fee × 20 years) -``` - -- **Lease to Permabuy Price:** Same as above - -#### Under Name Fees - -- **Initial Allocation:** 10 `under_names` are included with each name registration -- **For Leases:** - -```math -Lease ~Under ~Name ~Fee = BRF × DF × 0.1% -``` - -- **For Permabuys:** - -```math -Permabuy ~Under ~Name ~Fee = BRF × DF × 0.5% -``` - -#### Primary Name Fee - -Set or change primary name: The fee is equal to the associated fee for a single `under_name` purchase of a 51-character name of equivalent purchase type to the new primary name, regardless of the new primary name's length. - -### Step Pricing Mechanics - -- Synchronizes BRF (Base Rate Factor) with ARF (Adjusted Rate Factor) after seven (7) consecutive periods at the minimum DF value -- Resets DF to 1 following a step pricing adjustment +For exact costs, users should rely on the live ArNS app or SDK cost simulation rather than copying formulas into their own applications. ## Returned Name Premiums (RNP) -ArNS applies a **Returned Name Premium (RNP)** to names that re-enter the market after expiration or permanent return. This premium starts at a maximum value and decreases linearly over a predefined window, ensuring fair and transparent pricing for re-registered names. - -### RNP Mechanics - -#### Intent - -The premium starts at its maximum and decreases linearly until the name is purchased. If the name is not purchased before the premium window closes, it reverts to standard pricing and is no longer classified as "recently returned." - -#### RNP Window +ArNS applies a **Returned Name Premium (RNP)** to names that re-enter the market after expiration or permanent return. The premium starts high and decreases over a return window until the name reaches standard pricing again. -- **Duration:** Fourteen (14) periods +Returned name purchases split proceeds between the protocol balance and the previous owner. This discourages instant name sniping after expiry and gives owners a reason to release names they no longer need. -#### Returned Name Premium Formula - -The premium multiplier follows a linearly declining function: - -```math -RNP = 50 - (49 / 14) × t -``` - -Where: - -- **RNP:** The Returned Name Premium multiplier applied to the purchased name price -- **t:** Amount of time (or time-intervals) elapsed since the start of the return window - -#### RNP Registration Price - -```math -Price = RNP × (Lease~Or~Permabuy) ~Registration~ Price -``` - -#### Permanent Name Return Proceeds Split - -- **50%** goes to the returning name owner -- **50%** goes to the protocol balance - -The RNP multiplier is applied to the registration price of both permanently purchased and leased names. +For more detail, see [Returned Names](/learn/arns/returned-names). ## Gateway Operator ArNS Discount -Gateway operators who demonstrate consistent, healthy participation in the network are eligible for a **20% discount** on certain ArNS interactions. - -### Qualification Requirements - -To qualify for the discount: - -- The gateway must maintain a "Gateway Performance Ratio Weight" (GPRW) of **0.9 or higher** -- The gateway must have a "Tenure Weight" (TW) of **1.0 or greater** -- A gateway marked as "Leaving" shall not be eligible for this discount - -### Eligible Discounted Interactions +Gateway operators who demonstrate consistent, healthy participation in the network may be eligible for discounted ArNS interactions. This creates another incentive for gateways to provide reliable service while supporting ArNS usage. -- Purchasing a name -- Extending a lease -- Upgrading a lease to permabuy -- Increasing undernames capacity +Discount eligibility requires a **Gateway Performance Ratio Weight (GPRW) of 0.9** and a **Tenure Weight (TW) of 1.0**. Eligible operators receive a **20% discount** on new ArNS name registrations, lease extensions, lease upgrades, and undername purchases. ## Next Steps @@ -209,15 +62,15 @@ Congratulations! You now understand the complete ArNS pricing system. Ready to g See current ArNS pricing in real-time with the live pricing chart. -}> - Visit ArNS.app to register your first name and explore the pricing in action. +}> + Visit arns.ar.io to register your first name and explore the pricing in action. }> Learn about ar.io gateways and how they integrate with ArNS. - }> +}> Start building applications that leverage ArNS for decentralized naming. diff --git a/content/learn/arns/returned-names.mdx b/content/learn/arns/returned-names.mdx new file mode 100644 index 000000000..93402096d --- /dev/null +++ b/content/learn/arns/returned-names.mdx @@ -0,0 +1,75 @@ +--- +title: 'Returned Names' +description: 'How returned ArNS names are repriced through a Dutch auction mechanism' +--- + +## Overview + +When an ArNS name expires or is voluntarily returned to the protocol, it enters a **Returned Name Dutch Auction** before becoming available for standard registration. This mechanism prevents name squatting at expiry and provides fair pricing through a time-decaying premium. + +## Dutch Auction Mechanics + +Returned names start at a high premium and decay to the base price over a return window: + +- **Starting price**: premium above the base registration price +- **Ending price**: standard registration price +- **Decay**: decreases over time until standard pricing resumes + +```mermaid +graph LR + A[Name Expired
or Released] --> B[Dutch Auction
Starts at 50x] + B --> C[Price Decays
Over Time] + C --> D[Reaches
Standard Price] + D --> E[Standard
Registration] +``` + +## Revenue Split + +When a returned name is purchased during the Dutch auction: + +- **50%** goes to the **protocol balance** (funds epoch rewards) +- **50%** goes to the **previous owner** (the ANT holder at the time of return) + +This incentivizes name owners to voluntarily release names they no longer need, since they receive half the resale value. + +## How Names Enter the Returned Pool + +### Lease Expiration +When a leased name's term ends and the grace period elapses without renewal or conversion to permanent ownership, the name enters the returned pool. + +### Voluntary Release +Permanent name owners can voluntarily release their name back to the protocol. This places the name in the returned pool. + +## Lifecycle + +| Phase | Duration | Price | Action | +|-------|----------|-------|--------| +| **Active lease** | As registered | N/A | Name is in use | +| **Grace period** | After expiry | N/A | Owner can renew or convert | +| **Dutch auction** | Return window | Premium → standard price | Anyone can purchase | +| **Standard registration** | Indefinite | Base price | Normal ArNS purchase | + +## Querying Returned Names + +Use the SDK to check available returned names and their current auction price: + +```typescript +const ario = ARIO.mainnet(); + +// Get all active returned names +const returnedNames = await ario.getArNSReturnedNames({ + limit: 100, + sortBy: 'endTimestamp', + sortOrder: 'asc', +}); + +// Get a specific returned name +const name = await ario.getArNSReturnedName({ name: 'example' }); + +// Check current cost (includes auction premium) +const cost = await ario.getTokenCost({ + intent: 'Buy-Name', + name: 'example', + type: 'permabuy', +}); +``` diff --git a/content/learn/gateways/architecture.mdx b/content/learn/gateways/architecture.mdx index 5fee9c446..54007c770 100644 --- a/content/learn/gateways/architecture.mdx +++ b/content/learn/gateways/architecture.mdx @@ -5,7 +5,7 @@ description: "Learn about the technical architecture of ar.io gateways, their co import { Database, Shield, Layers, Server } from 'lucide-react'; -Ar.io gateways are sophisticated data access layers built on top of the Arweave network. They transform the raw Arweave blockchain into a performant, reliable, and developer-friendly platform for storing and retrieving data. Gateways act as bridges between applications and the permanent storage capabilities of Arweave. +Ar.io gateways are data access layers built on top of Arweave. They make permanent data easier to retrieve, cache, index, and serve through standard web interfaces. ```mermaid graph TB @@ -87,14 +87,14 @@ Ar.io gateways are built using modern, scalable technologies designed for high-p Several important design decisions shape how ar.io gateways operate: ### Data Retrieval Strategy -Ar.io gateways use a sophisticated **hierarchical fallback system** for data retrieval: +Ar.io gateways use a **hierarchical fallback system** for data retrieval: 1. **Trusted gateways**: Prioritize data from verified, high-performance peers 2. **Ar.io network**: Leverage the broader network of ar.io gateways 3. **Chunks data items**: Reconstruct data from individual chunks when needed 4. **Transaction data**: Fall back to raw Arweave transaction data -This approach ensures data availability while optimizing for speed and reliability. +This approach improves availability while optimizing for speed and reliability. ### Verification and Trust Model - **Multi-level cryptographic verification**: Data integrity is verified at multiple points @@ -102,12 +102,16 @@ This approach ensures data availability while optimizing for speed and reliabili - **Self-healing mechanisms**: Automatic recovery and re-verification of corrupted data - **Verification headers**: HTTP headers indicate the verification status of returned data +### Serving Capabilities + +Gateways expose a serving layer for applications and end users. This layer includes byte-range requests, signed and verifiable responses, x402 paid access, peer routing, and content moderation hooks. These capabilities let operators tune how data is delivered while preserving verifiability and local operator choice. + ### Worker Specialization Different background workers handle specific responsibilities: -- **Block synchronization workers**: Keep the gateway synchronized with Arweave blocks -- **Bundle processing workers**: Handle Layer 2 bundled data items (ANS-104) -- **Data verification workers**: Continuously verify stored data integrity +- **Block synchronization workers**: Keep the gateway aligned with Arweave blocks +- **Bundle processing workers**: Extract and index ANS-104 data items +- **Data verification workers**: Check cached data integrity - **Maintenance workers**: Perform cleanup and optimization tasks ## Scalability and Configuration @@ -119,7 +123,7 @@ Gateway services can be independently configured or disabled based on operator n - **Data serving**: Serve cached data to applications - **Data indexing**: Index and process new Arweave data - **Bundle processing**: Handle Layer 2 bundled transactions -- **ArNS routing**: Provide Arweave Name System resolution +- **ArNS routing**: Provide Ar.io Name System resolution ## Core Philosophy: Builder Independence @@ -134,7 +138,7 @@ Operators maintain complete control through **[Data Retrieval](/learn/gateways/d ### Network Resilience The modular design creates a resilient ecosystem where distributed infrastructure and customizable trust models prevent single points of failure. -This architecture ensures that builders can create powerful applications on Arweave while maintaining independence from any centralized infrastructure or service provider. +This architecture helps builders use Arweave without depending on a single infrastructure provider. ## Explore Gateway Capabilities diff --git a/content/learn/gateways/data-retrieval.mdx b/content/learn/gateways/data-retrieval.mdx index 94641bc00..04d55dd2f 100644 --- a/content/learn/gateways/data-retrieval.mdx +++ b/content/learn/gateways/data-retrieval.mdx @@ -5,7 +5,7 @@ description: "How ar.io gateways retrieve and share data from multiple sources i import { Shield, Cpu, Globe, Settings } from 'lucide-react'; -Ar.io gateways use a sophisticated multi-tier architecture to retrieve and serve Arweave data. This system ensures high availability, fast response times, and data integrity by leveraging multiple data sources with automatic fallback mechanisms. +Ar.io gateways retrieve and serve Arweave data from multiple sources. They prefer fast local or trusted sources when available, then fall back to broader network peers, chunks, or Arweave nodes as needed. ## How Gateways Retrieve Data @@ -37,7 +37,7 @@ graph TD classDef source fill:#2563eb,stroke:#1d4ed8,stroke-width:2px,color:#fff classDef process fill:#16a34a,stroke:#15803d,stroke-width:2px,color:#fff - class TRUSTED,NETWORK,CHUNKS,TXDATA source + class TRUSTED,NETWORK,CHUNKS,ARWEAVE source class VALIDATE,SERVE,STORE process ``` @@ -48,14 +48,14 @@ Ar.io gateways can retrieve data from multiple sources, each with different char ### 1. Trusted Gateways - **Purpose**: Peer-to-peer data sharing between verified ar.io gateways - **Benefits**: Distributed redundancy, load balancing, network resilience -- **Trust Mechanism**: Performance-based trust scores and reciprocity monitoring -- **Selection**: Prioritized based on established trust relationships +- **Trust Mechanism**: Operator-defined trust settings, observed performance, and reciprocity +- **Selection**: Prioritized based on local gateway configuration ### 2. ar.io (Untrusted Peers) - **Purpose**: Broader network of ar.io gateways without established trust - **Benefits**: Geographic distribution, expanded data availability -- **Selection**: Weighted random selection based on performance metrics -- **Validation**: Enhanced verification required due to untrusted nature +- **Selection**: Chosen based on availability, configuration, and routing strategy +- **Validation**: Verification is important because the peer may not be trusted ### 3. Chunk Assembly - **Purpose**: Direct reconstruction from Arweave chunks via known offsets @@ -92,7 +92,7 @@ Used specifically for unbundling and verification processes: ## Trust and Validation ### Peer Trust Management -Gateways maintain sophisticated trust relationships: +Gateways can maintain trust relationships with peer gateways: ```mermaid graph TD @@ -147,7 +147,7 @@ Every piece of retrieved data undergoes validation: --- -The data retrieval system is fundamental to ar.io's mission of providing reliable, performant access to the permaweb. This sophisticated architecture ensures that Arweave's permanent data remains accessible through a resilient, distributed gateway network. +The data retrieval system is central to ar.io's mission of providing reliable, performant access to the permaweb. Multiple retrieval paths help keep permanent data accessible through a distributed gateway network. ## Related Gateway Concepts diff --git a/content/learn/gateways/data-verification.mdx b/content/learn/gateways/data-verification.mdx index 5a46004d4..9ecdfe298 100644 --- a/content/learn/gateways/data-verification.mdx +++ b/content/learn/gateways/data-verification.mdx @@ -5,11 +5,13 @@ description: "How ar.io gateways ensure data integrity by verifying chunks are c import { Database, Cpu, Server, Cog } from 'lucide-react'; -Ar.io gateways continuously verify that data chunks are correctly stored and retrievable from Arweave. This ensures users receive authentic, uncorrupted data with cryptographic proof of integrity. The verification system is what makes ar.io gateways trustworthy data providers for the permaweb. +Ar.io gateways verify that retrieved and cached data matches what was committed to Arweave. Verification helps users receive authentic, uncorrupted data without trusting a single gateway operator. + +Gateway data verification is one layer of ar.io's broader verification architecture. For how gateway verification composes with signed response claims, client-side verification, and OIP accountability, see [Verification and Accountability](/learn/verification). ## How Gateways Verify Data -Data verification is an ongoing process that uses Merkle tree cryptography to provide mathematical proof of data integrity. The process involves multiple specialized components working together to ensure cached data matches what's stored on Arweave: +Data verification uses Arweave data roots, hashes, and Merkle proofs to check that cached data matches what was originally stored. A gateway can verify data before serving it or re-import data when verification fails: ```mermaid sequenceDiagram @@ -59,7 +61,7 @@ sequenceDiagram **The Verification Workflow:** -Gateways achieve verification through a systematic five-phase process orchestrated by the DataVerificationWorker. This process ensures that every piece of cached data cryptographically matches its original form on Arweave, providing mathematical proof of integrity before serving data to users. +At a high level, verification moves through discovery, retrieval, cryptographic computation, comparison, and recovery: **1. Discovery Phase** - Periodically scan for unverified data items @@ -92,7 +94,7 @@ Ar.io gateways handle different types of data verification based on the data's o ### Transaction Data Verification For individual Arweave transactions: -- **Direct root validation** against transaction data roots stored on-chain +- **Direct root validation** against transaction data roots stored onchain - **Complete data reconstruction** from chunks to ensure availability - **Cryptographic proof** that data matches what was originally stored @@ -116,12 +118,10 @@ At the most granular level: ### Cryptographic Trust Foundation - **Mathematical Proof**: Merkle tree cryptography provides irrefutable proof of data integrity - **Independent Validation**: Multiple gateways verify the same data independently -- **Network Consensus**: Distributed verification creates trust without central authority ### Data Integrity Guarantees - **Tamper Detection**: Any alteration to data is immediately detectable - **Corruption Recovery**: Automatic healing of corrupted data through re-import -- **Permanent Storage Validation**: Ensures Arweave's permanence promise is maintained ### Gateway Reliability - **Continuous Monitoring**: Ongoing verification catches issues before users encounter them diff --git a/content/learn/gateways/gateway-registry.mdx b/content/learn/gateways/gateway-registry.mdx index 9aee9e22e..f723a3205 100644 --- a/content/learn/gateways/gateway-registry.mdx +++ b/content/learn/gateways/gateway-registry.mdx @@ -1,89 +1,87 @@ --- title: "Gateway Registry" -description: "Ar.io consists of ar.io gateway nodes, which are identified by their registered Arweave wallet addresses and either their IP addresses or hostnames, as stored in the network" +description: "How ar.io gateways register, publish metadata, and participate in the network." --- -import { Network, Target, Shield, Settings } from 'lucide-react'; +import { Network, Target, Shield, Settings } from "lucide-react"; ## Overview -Ar.io consists of [ar.io gateway nodes](/learn/what-is-ario), which are identified by their registered Arweave wallet addresses and either their IP addresses or hostnames, as stored in the network's [smart contract](/learn/token) Gateway Address Registry (GAR). +The Gateway Address Registry (GAR) is the public registry of ar.io gateways. It records which gateways have joined the network, how they can be reached, and the metadata applications need to discover and evaluate them. -Any gateway operator that wishes to join ar.io must register their node in the ar.io smart contract's Gateway Address Registry. Registration involves staking a minimum amount of ARIO tokens and providing additional metadata describing the gateway service offered. - -These nodes adhere to ar.io's protocols, creating a collaborative environment of gateway nodes that vary in scale and specialization. The network promotes a fundamental level of service quality and trust minimization among its participants. +Gateway registry state is maintained by the ar.io Solana programs. Registered gateways are identified by Solana addresses and publish service information such as hostname, protocol settings, staking state, and operator-provided metadata.
ar.io Network Portal showing the Gateway Address Registry - The gateways.ar.io portal displays all gateways currently in the network, showing their stakes, performance scores, and operational metrics + The gateways.ar.io portal displays gateways currently in the network, including stake, performance, and operational metadata.
-### Benefits of Joining the Network +## Joining the Registry -Being part of the network grants ar.io gateways an array of advantages: -- Simplified advertising of services and discovery by end users via the Gateway Address Registry -- More rapid bootstrapping of key gateway operational data due to prioritized data request fulfillment among gateways joined to the network -- Sharing of data processing results -- Auditability and transparency through the use of AGPL-3 licenses, which mandate public disclosure of any software changes, thereby reinforcing the network's integrity and reliability -- Improved network reliability and performance through an incentive protocol, which uses a system of evaluations and rewards to encourage high-quality service from gateways -- Eligibility to accept delegated staking improving a gateway's discoverability and reward opportunities -- **Eligibility to receive distributions from the protocol balance** - Gateways that have joined the network are eligible to receive token distributions based on their performance and contributions to the network +To join the ar.io network, a gateway operator registers their gateway and locks ARIO as operator stake. Registration connects the gateway's public endpoint, operator address, observer address, and service metadata to the onchain registry. -### How the GAR Works +Once registered, a gateway can become eligible for network incentives, delegation, and observation through OIP. Gateway operators also need enough SOL to pay Solana transaction fees for network interactions. -After joining the network, the operator's gateway can be easily discovered by permaweb apps, its health can be observed, and it can participate in data sharing protocols. A gateway becomes eligible to participate in the network's incentive protocol in the epoch following the one they joined in. +The registry has a maximum capacity of **3,000 gateways**. Each gateway requires a **20,000 ARIO** minimum network-join stake, and each gateway can have up to **10,000** unique delegated stakers. -The GAR advertises the specific attributes of each gateway including its stake, delegates, settings and services. This enables permaweb apps and users to discover which gateways are currently available and meet their needs. Apps that read the GAR can sort and filter it using the gateway metadata, for example, ranking gateways with the highest stake, reward performance, or feature set at the top of the list. This allows users to prefer the higher staked, more rewarded gateways with certain capabilities over lower staked, less rewarded gateways. +## What the Registry Enables -## Token Incentives and Network Monitoring +### Discovery -The ar.io network uses a sophisticated incentive system to ensure gateway quality and reliability: +Apps, users, and other gateways can use the registry to find gateways by endpoint, service metadata, stake, observed performance, and supported capabilities. -- **Token Incentives**: Learn more about how gateways earn rewards and participate in the network economy in the [Token section](/learn/token/) -- **Observer Protocol**: The network employs an Observer system that monitors gateway performance and ensures quality of service. Learn more about the [Observer & Incentive Protocol](/learn/oip) and how it maintains network integrity +### Incentive Participation -## Recap +The registry connects gateway identity to staking, delegated stake, performance history, and reward eligibility. OIP uses this information to evaluate gateways and distribute rewards. + +### Network Transparency + +Gateway information is publicly visible through Solana state and network tooling such as [gateways.ar.io](https://gateways.ar.io). This makes participation, performance, and configuration easier to inspect. -The Gateway Registry is the foundation of the ar.io network's decentralized infrastructure. Key takeaways: +### Operator Choice -- **Network Participation**: Gateways must register and stake ARIO tokens to join the network -- **Protocol Distributions**: Registered gateways are eligible to receive token distributions from the protocol balance -- **Observer Monitoring**: The network employs an [Observer and Incentives Protocol](/learn/oip) that monitors gateway performance and ensures quality of service -- **Staking & Rewards**: Gateways earn rewards based on performance through a sophisticated [staking system](/learn/token/staking) that includes delegation opportunities -- **Discoverability**: The GAR enables apps and users to find suitable gateways based on their needs -- **Performance-Based Selection**: Gateway metadata allows for intelligent routing based on stake, performance, and capabilities -- **Transparent Ecosystem**: All gateway information is publicly accessible through the smart contract and at gateways.ar.io +Gateways can specialize. Some may focus on fast public access, some on indexing, some on private infrastructure, some on paid access, and some on specific moderation or compliance policies. + +## Relationship to OIP + +The registry is the set of gateways that OIP can observe and evaluate. Registered gateways are periodically checked for availability, correctness, and ArNS resolution behavior. Reliable gateways can earn rewards, while gateways that repeatedly fail can lose eligibility and eventually be removed through pruning. + +## Recap -By joining the network, gateways become part of a collaborative ecosystem that rewards quality service and ensures reliable access to the permaweb. +- The Gateway Address Registry is the public source of network gateway metadata. +- Gateways join by registering service details and locking operator stake. +- Registered gateways can participate in OIP, receive delegations, and become eligible for rewards. +- Apps can use registry data to discover, filter, and route through gateways. +- Poorly performing gateways can be pruned from the network after sustained failure. ## Explore the Gateway Ecosystem - } /> - } /> - } /> - } /> diff --git a/content/learn/gateways/index.mdx b/content/learn/gateways/index.mdx index ffb720591..7bf4fb7f6 100644 --- a/content/learn/gateways/index.mdx +++ b/content/learn/gateways/index.mdx @@ -1,105 +1,80 @@ --- title: "ar.io Gateways" -description: "Ar.io gateways bridge the Arweave network and applications, providing fast, reliable access to permanent data through specialized infrastructure." +description: "Ar.io gateways provide fast, reliable access to permanent Arweave data through decentralized infrastructure." --- import { Card, Cards } from "fumadocs-ui/components/card"; import { Building, Download, Shield, Server, CreditCard } from "lucide-react"; -## What Are ar.io Gateways? +## What are Gateways? -Ar.io gateways are specialized infrastructure nodes that serve as bridges between the Arweave network and applications. They transform raw Arweave blockchain data into a fast, reliable, and developer-friendly platform for storing and retrieving permanent data. +Ar.io gateways are infrastructure nodes that make Arweave data easy to access from web apps, APIs, and users. They retrieve data from Arweave, cache and index it, resolve ArNS names, and expose standard HTTP interfaces for the permaweb. -## Core Responsibilities - -Ar.io gateways handle three fundamental responsibilities: - -### Data Writing & Proxying - -- **Transaction relay**: Forward transaction headers to Arweave miners for mempool inclusion -- **Chunk distribution**: Proxy data chunks to Arweave nodes for storage and replication -- **Bundle processing**: Receive and bundle ANS-104 data items into base layer transactions - -### Data Retrieval & Serving - -- **Fast access**: Serve cached data with optimized performance and reliability -- **Multi-source fallback**: Retrieve data from trusted gateways, network peers, or directly from Arweave -- **Content delivery**: Stream complete transactions, individual chunks, or bundled data items +Gateways do not replace Arweave. Arweave provides permanent storage and data availability guarantees; gateways make that data fast, discoverable, and practical to use. -### Data Discovery & Indexing - -- **Structured queries**: Enable efficient searches across transactions, bundles, and wallet data -- **Real-time indexing**: Process incoming data streams and maintain searchable databases -- **ArNS routing**: Provide human-readable name resolution for Arweave content - -## Key Features - -### Modular Architecture +## Core Responsibilities -Gateways are built with interchangeable components that operators can customize: +### Data Access -- **Configurable services**: Enable or disable features based on specific needs -- **Scalable storage**: From SQLite for small deployments to cloud databases for enterprise scale -- **Flexible infrastructure**: Adaptable to different operational environments and requirements +- **Retrieve permanent data** from Arweave, peer gateways, local cache, or chunk-level sources +- **Serve content over HTTP** using familiar web patterns +- **Optimize delivery** through caching, streaming, and source selection -### Network Connectivity +### Indexing and Discovery -- **Decentralized network**: Connect to other ar.io gateways for data sharing and redundancy -- **Trust-minimized access**: Cryptographically verify data integrity without relying on central authorities -- **Performance optimization**: Intelligent caching and content delivery strategies +- **Index transactions and bundled data items** so apps can search and retrieve content efficiently +- **Resolve ArNS names** into the data records and routes they point to +- **Expose gateway metadata** so users and applications can discover available infrastructure -### Developer Experience +### Network Participation -- **HTTP APIs**: Standard web interfaces for all gateway functionality -- **Monitoring & telemetry**: Built-in observability for operational insights -- **Content moderation**: Configurable policies for community and compliance needs +- **Register in the Gateway Address Registry** to become discoverable as an ar.io network gateway +- **Stake ARIO** to participate in network incentives and signal operational commitment +- **Participate in OIP** by being observed, reporting observations when selected, and earning rewards for reliable service ## What Gateways Are Not -It's important to understand the boundaries of what ar.io gateways do and don't provide: - ### Not Storage Providers -- **Don't enforce Arweave protocol**: Gateways don't validate consensus or mining rules -- **Don't guarantee permanence**: Storage permanence comes from Arweave itself, not gateways -- **Don't replicate all data**: Gateways cache popular content but aren't full blockchain replicas +- **They do not create Arweave permanence**: permanence comes from Arweave itself +- **They do not need to store everything**: operators choose what to cache, index, and serve +- **They do not control user data**: data ownership remains with the original publisher and underlying protocols -### Not Compute Platforms +### Not Application Compute -- **Don't depend on AO**: Gateways operate independently of any compute layer -- **Don't execute smart contracts**: Computation happens on AO or other platforms, not gateways -- **Don't process application logic**: Gateways focus purely on data access and delivery +- **They do not run protocol logic**: ar.io protocol state, staking, ArNS, and incentives are coordinated by Solana programs +- **They do not process arbitrary app logic**: gateways focus on data access, indexing, routing, and delivery -### Not Centralized Services +### Not a Single Service -- **Don't control data**: Content ownership and control remain with original creators -- **Don't gate access**: Anyone can run a gateway and access Arweave data -- **Don't create vendor lock-in**: Gateway APIs and protocols are open and interoperable +- **Anyone can run one**: the gateway network is open to independent operators +- **Apps can choose among gateways**: routing can consider availability, geography, performance, policy, and payment requirements +- **Operators can specialize**: gateways may differ in scale, indexing choices, moderation policy, payment setup, and extensions ## Explore Gateways } /> } /> } /> } /> diff --git a/content/learn/gateways/meta.json b/content/learn/gateways/meta.json index 7472de369..31cfa7786 100644 --- a/content/learn/gateways/meta.json +++ b/content/learn/gateways/meta.json @@ -1,5 +1,6 @@ { "title": "Gateways", + "icon": "Network", "pages": [ "architecture", "data-retrieval", diff --git a/content/learn/meta.json b/content/learn/meta.json index 7d4fad3f1..600196324 100644 --- a/content/learn/meta.json +++ b/content/learn/meta.json @@ -8,6 +8,7 @@ "(introduction)", "cloud", "gateways", + "verification", "oip", "arns", "wayfinder", diff --git a/content/learn/oip/epoch-pipeline.mdx b/content/learn/oip/epoch-pipeline.mdx new file mode 100644 index 000000000..3cf4bece2 --- /dev/null +++ b/content/learn/oip/epoch-pipeline.mdx @@ -0,0 +1,102 @@ +--- +title: 'Epoch Pipeline' +description: 'The 6-step permissionless epoch pipeline for observation, reward calculation, and distribution on Solana' +--- + +## Overview + +On Solana, the ar.io epoch lifecycle is broken into six discrete, permissionless steps. Each step is a separate instruction that can be executed by anyone. + +All steps are **idempotent** (safe to run multiple times) and **permissionless** (anyone can crank them). This design ensures the protocol cannot be halted by a single point of failure. + +## Pipeline Steps + +```mermaid +graph LR + A[create_epoch] --> B[tally_weights] + B --> C[prescribe_epoch] + C --> D[save_observations] + D --> E[distribute_epoch] + E --> F[close_epoch] +``` + +### 1. create_epoch + +**Initializes the epoch account and computes the reward rate.** + +- Creates the epoch account +- Computes the epoch reward allocation from the protocol balance + +### 2. tally_weights + +**Batched computation of gateway weights for observer selection.** + +- Computes composite weights for gateways: + - **Stake weight**: Based on total stake (operator + delegated) + - **Tenure weight**: Based on how long the gateway has been in the network + - **Gateway performance**: Based on pass rate across recent epochs + - **Observer performance**: Based on observation submission history +- Batches work so large gateway sets can be processed safely on Solana + +### 3. prescribe_epoch + +**Selects observers and prescribed ArNS names via weighted roulette.** + +- Selects observers using weighted random selection +- Selects prescribed ArNS names that observers use as common test targets + +### 4. save_observations + +**Observers submit their pass/fail observation reports.** + +- Each selected observer submits compact pass/fail results for tested gateways +- This is the only step that requires a specific signer (the selected observer) +- Observations are stored on the Epoch account + +### 5. distribute_epoch + +**Batched reward distribution to gateways and their delegates.** + +- Functional gateways receive the Base Gateway Reward (BGR) +- Functional observers receive the Base Observer Reward (BOR) +- Deficient observers do not receive observer rewards +- Operator rewards auto-compound into operator stake +- Delegate rewards are tracked via the reward-per-share accumulator (settled lazily) +- Leaving gateways receive 0 rewards + +### 6. close_epoch + +**Reclaims rent from completed epoch accounts.** + +- Recovers SOL rent from completed epoch accounts +- Keeps onchain state lean over time + +## Timing + +The pipeline steps can be executed as the epoch progresses: + +| Step | When | Batched? | +|------|------|----------| +| create_epoch | After previous epoch ends | No | +| tally_weights | After create_epoch | Yes | +| prescribe_epoch | After all weights tallied | No | +| save_observations | During observation window | No (per observer) | +| distribute_epoch | After observation window | Yes | +| close_epoch | After epoch state is no longer needed | No | + +## Who Cranks? + +A cranker is a permissionless actor that executes the epoch pipeline on Solana. Since Solana programs cannot execute on a timer, an external wallet must call each instruction to advance the epoch lifecycle. + +The key property is that cranking is **permissionless**: any wallet with SOL for transaction fees can run it. No ARIO tokens, gateway registration, or special authorization is required. + +Without crankers, the epoch pipeline would stall. Observations would not be prescribed, rewards would not be distributed, and completed epoch state would not be closed. Multiple independent crankers provide redundancy so the network is not dependent on a single operator. + +Cranking can run as a standalone bot that watches epoch state and submits whichever pipeline instruction is needed next. It can also be embedded directly in ar.io observers: + +```bash +# In your observer's .env file +ENABLE_EPOCH_CRANKING=true +``` + +Multiple crankers can run at the same time without coordination. The first successful transaction advances the pipeline, and later attempts see that the step is already complete. Because the instructions are idempotent, duplicate calls do not double-distribute rewards or corrupt epoch state. diff --git a/content/learn/oip/index.mdx b/content/learn/oip/index.mdx index 9a510b5d3..9fd5facbc 100644 --- a/content/learn/oip/index.mdx +++ b/content/learn/oip/index.mdx @@ -8,109 +8,89 @@ import { Target, BarChart, FileText, Coins } from "lucide-react"; ## Overview -The Observation and Incentive Protocol ensures network quality through peer monitoring and performance-based rewards. Gateways are incentivized to maintain high performance while also serving as "observers" that evaluate their peers' ArNS resolution capabilities and data integrity verification. +The Observation and Incentive Protocol (OIP) helps maintain gateway quality through peer monitoring and performance-based rewards. Gateways are incentivized to serve data reliably while also acting as observers that evaluate their peers. -The protocol operates on **24-hour epochs** where up to 50 gateways are selected as observers to test other gateways against ArNS name resolution criteria and chunk/offset validation. This creates a self-regulating ecosystem with transparent, consensus-based performance evaluation. +The protocol runs in epochs. During each epoch, selected gateways observe other gateways, submit reports, and participate in an onchain pass/fail voting process. Functional gateways and observers become eligible for ARIO rewards, while deficient gateways and observers miss rewards and can lose future selection weight. ## Architecture Overview -The Observer Protocol operates through a systematic process where selected gateways monitor their peers and report findings to maintain network quality: +The protocol follows a repeatable flow: ```mermaid sequenceDiagram - participant SC as ar.io Smart Contract + participant SC as ar.io Protocol participant OBS as Observer Gateway participant GW as Target Gateway participant AR as Arweave Network Note over SC,AR: Epoch Start - SC->>SC: Select up to 50 observers
(weighted random) - SC->>SC: Generate 2 prescribed ArNS names + SC->>SC: Select observers
(weighted random) + SC->>SC: Prescribe ArNS names SC->>OBS: Notify selection & provide names Note over SC,AR: Observation Phase - OBS->>OBS: Choose 8 additional ArNS names - OBS->>OBS: Select subset for chunk validation loop For each gateway to test - OBS->>GW: Test ArNS resolution (10 names) + OBS->>GW: Test ArNS resolution GW-->>OBS: Response data OBS->>OBS: Score: Pass/Fail end - loop For selected gateways - OBS->>GW: Test chunk/offset validation - GW-->>OBS: Chunk data + Merkle proof - OBS->>OBS: Verify cryptographic proof - end - Note over SC,AR: Reporting Phase OBS->>AR: Upload detailed JSON report AR-->>OBS: Confirm storage - OBS->>SC: Submit failed gateways list + OBS->>SC: Submit compact pass/fail results SC-->>OBS: Confirm interaction Note over SC,AR: Evaluation & Rewards SC->>SC: Tally all observer votes - SC->>SC: Determine gateway status
(≥50% pass = functional) - SC->>SC: Calculate reward distribution + SC->>SC: Tally observer votes + SC->>SC: Determine rewards SC->>OBS: Distribute observer rewards SC->>GW: Distribute gateway rewards
(if functional) ``` -## Epoch Cycle and Responsibilities +## Epoch Cycle -Each 24-hour epoch follows a structured process with specific responsibilities for gateways and observers: +Each epoch follows a structured process with specific responsibilities for gateways and observers: ### Epoch Start -- **Smart Contract**: Selects up to 50 observers using weighted random selection -- **Smart Contract**: Generates 2 prescribed ArNS names for all observers to test +- **ar.io protocol**: Selects observers using weighted random selection +- **ar.io protocol**: Prescribes common ArNS names for observers to test - **Selected Observers**: Receive notification of selection and prescribed names ### Observation Phase -- **Observers**: Choose 8 additional ArNS names to test (total of 10 names per gateway) -- **Observers**: Select subset of gateways for chunk/offset validation based on sampling rate -- **Observers**: Test assigned gateways for ArNS resolution, wallet ownership, content hashes, and response times -- **Observers**: Validate chunk/offset data integrity using cryptographic Merkle proofs -- **Target Gateways**: Respond to resolution requests, serve content, and provide chunk data with proofs +- **Observers**: Test assigned gateways for ArNS resolution and response quality +- **Observers**: Document pass/fail findings and failure reasons +- **Target Gateways**: Respond to resolution requests and serve the requested content ### Reporting Phase - **Observers**: Upload detailed JSON reports to Arweave for transparency -- **Observers**: Submit failed gateway lists to the ar.io Smart Contract for consensus voting +- **Observers**: Submit compact onchain pass/fail results for consensus voting ### Evaluation and Distribution -- **Smart Contract**: Tallies all observer votes (≥50% pass = functional gateway) -- **Smart Contract**: Distributes rewards at epoch end based on performance -- **Functional Gateways/Observers**: Receive ARIO token rewards automatically +- **ar.io protocol**: Tallies observer votes to classify gateways +- **ar.io protocol**: Distributes rewards based on performance +- **Functional Gateways/Observers**: Become eligible for ARIO rewards ## Key Features - **Decentralized Monitoring**: Peer-to-peer evaluation ensures no single point of failure -- **Consensus-Based Scoring**: Majority rule (≥50% pass votes) determines gateway functionality +- **Consensus-Based Scoring**: Observer submissions determine gateway functionality - **Performance Incentives**: Only functional gateways and observers receive ARIO token rewards -- **Data Integrity Validation**: Cryptographic verification of chunk/offset data using Merkle proofs - **Transparent Accountability**: All reports permanently stored on Arweave and viewable at [gateways.ar.io](https://gateways.ar.io) - **Sustainable Funding**: Protocol balance funded by ArNS name purchases, aligning rewards with network usage -## Chunk/Offset Validation - -The protocol includes advanced data integrity verification through chunk/offset observation. Observers validate that gateways can correctly serve and verify Arweave data chunks using cryptographic proofs: - -### Validation Process - -- **Sampling**: A subset of gateways is selected for chunk validation each epoch -- **Offset Testing**: Random offsets within the stable weave range are tested -- **Merkle Proof Verification**: Cryptographic validation ensures chunk authenticity -- **Binary Search Optimization**: Efficient transaction lookup using cached metadata +## What Observers Evaluate -### Technical Implementation +Observer criteria can evolve over time without requiring every detail to live in the protocol itself. At a high level, observers evaluate whether gateways can: -- **Chunk Retrieval**: `GET /chunk/{offset}` returns chunk data and Merkle proof -- **Proof Validation**: Uses Arweave's `validatePath()` function for cryptographic verification -- **Performance Optimization**: LRU caching for blocks, transactions, and metadata -- **Early Stopping**: Tests stop immediately upon first successful validation +- Resolve prescribed ArNS names correctly +- Return expected transaction IDs and response data +- Serve data with acceptable availability and responsiveness +- Produce reports that can be independently inspected by users and applications --- @@ -121,25 +101,25 @@ The protocol includes advanced data integrity verification through chunk/offset } /> } /> } /> } /> diff --git a/content/learn/oip/meta.json b/content/learn/oip/meta.json index 33e3cb26c..4589cce1f 100644 --- a/content/learn/oip/meta.json +++ b/content/learn/oip/meta.json @@ -1,10 +1,14 @@ { - "title": "Observation & Incentive Protocol", + "title": "Observation & Incentive Protocol (OIP)", + "icon": "Activity", "defaultOpen": false, "pages": [ + "epoch-pipeline", "observer-selection", "reporting", "performance-evaluation", - "reward-distribution" + "reward-distribution", + "staking", + "pruning" ] } diff --git a/content/learn/oip/observer-selection.mdx b/content/learn/oip/observer-selection.mdx index 1a18ef958..f128c6919 100644 --- a/content/learn/oip/observer-selection.mdx +++ b/content/learn/oip/observer-selection.mdx @@ -1,33 +1,32 @@ --- title: "Observer Selection" -description: "Learn about how gateways are selected as observers each epoch and how ArNS names are chosen using weighted random selection and Hashchain entropy" +description: "Learn how gateways are selected as observers each epoch using weighted random selection" --- ## Epochs and Selection Timeline -The ar.io network operates on **24-hour epochs**, during which the observer selection and evaluation process takes place. At the start of each epoch: +The ar.io network operates in epochs, during which the observer selection and evaluation process takes place. At the start of each epoch: -- **50 observers** are selected to monitor the network -- **2 prescribed ArNS names** are chosen for all observers to test -- **8 additional names** are selected by each observer individually -- **Gateway subset** is selected for chunk/offset validation based on sampling rate +- Observers are selected to monitor the network +- Prescribed ArNS names are chosen for all observers to test +- Observers may evaluate additional names or criteria as defined by the current observation process -This creates a consistent evaluation framework where all observers test the same baseline names while having flexibility to choose additional targets for comprehensive network monitoring, plus advanced data integrity verification. +This creates a consistent evaluation framework where all observers test the same baseline names while leaving room for the observation process to evolve over time. ## Selection Process -Up to **fifty (50)** gateways are selected as observers per epoch using a sophisticated weighted random selection system. The selection uses **hashchain entropy** from previous ar.io contract state messages to ensure unpredictable and tamper-resistant selection. +Gateways are selected as observers using weighted random selection. The process combines randomness with gateway-specific weights so observer duties are unpredictable, but still influenced by stake, tenure, and historical performance. -The hashchain-based entropy provides cryptographic randomness for selecting: +The entropy provides cryptographic randomness for selecting: -- **Observer Gateways**: The 50 gateways chosen to perform observations -- **Prescribed ArNS Names**: The 2 common names all observers must evaluate +- **Observer Gateways**: Gateways chosen to perform observations +- **Prescribed ArNS Names**: Common names all observers must evaluate This approach prevents manipulation while maintaining weighted probabilities based on gateway performance and commitment. ![Current epoch observers showing their observation chance (normalized composite weight) and report status](/content/observers.png) -

{" "} shows the current epoch prescribed observers and arns names, as well as their submission status -

+
## Weighted Selection Criteria @@ -57,45 +56,11 @@ The selection considers four key factors that are multiplied together to create - **Gateway Performance Ratio Weight (GPRW)**: Historical gateway performance - **Observer Performance Ratio Weight (OPRW)**: Historical observer performance -These weights are then normalized across all eligible gateways to create selection probabilities. For detailed weight calculations and formulas, see [Performance Evaluation](/learn/oip/performance-evaluation). +These weights are then normalized across all eligible gateways to create selection probabilities. For more on how performance affects those weights, see [Performance Evaluation](/learn/oip/performance-evaluation). -## Hashchain Random Selection +## Random Selection -The selection process uses **hashchain entropy** from previous ar.io contract state messages to achieve cryptographically secure randomness: - -### How Hashchain Selection Works - -1. **Entropy Source**: Random numbers are generated from the hashchain of previous contract state messages -2. **Weighted Mapping**: Each random number maps to normalized weight ranges of eligible gateways -3. **Observer Selection**: The gateway whose weight range contains each random number is selected -4. **Prescribed Names**: The same entropy selects 2 ArNS names that all observers must test - -This creates tamper-resistant selection where higher-weighted gateways have proportionally better chances of selection, while maintaining true randomness that cannot be predicted or manipulated. - -## Chunk/Offset Sampling - -In addition to observer selection, the protocol includes a separate sampling process for chunk/offset validation: - -### Gateway Selection for Chunk Validation - -- **Deterministic Selection**: Uses PRNG seeded with observation entropy to select gateway subset -- **Sampling Rate**: Configurable percentage of gateways tested per observation (default: 1%) -- **Minimum Guarantee**: At least 1 gateway is always selected for testing -- **Offset Selection**: Random offsets within the stable weave range are chosen for each selected gateway - - - **Initial Implementation**: During the initial rollout phase, only a very - small portion of gateways will be checked for chunk/offset validation, and the - current validation criteria are extremely lenient to ensure smooth network - operation. - - -### Validation Process - -- **Chunk Retrieval**: Observers request chunk data using `GET /chunk/{offset}` -- **Merkle Proof Verification**: Cryptographic validation ensures data integrity -- **Early Stopping**: Tests stop immediately upon first successful validation -- **Performance Optimization**: Uses LRU caching for efficient transaction lookup +The selection process maps random values onto normalized gateway weight ranges. Gateways with higher weights have proportionally better chances of selection, while randomness prevents predictable or easily manipulated assignments. ## Fairness and Meritocracy @@ -103,7 +68,7 @@ This system ensures: - **Meritocratic Selection**: Higher-performing gateways have better selection odds - **Fair Opportunity**: All gateways maintain non-zero selection probability -- **Tamper Resistance**: Hashchain entropy prevents manipulation +- **Tamper Resistance**: Entropy prevents predictable observer assignment - **Consistent Standards**: Prescribed names create common evaluation baseline The selection is saved in the contract state at epoch start to ensure that activities during the epoch do not affect selection or reward distribution. diff --git a/content/learn/oip/performance-evaluation.mdx b/content/learn/oip/performance-evaluation.mdx index d28249b3d..b5b944345 100644 --- a/content/learn/oip/performance-evaluation.mdx +++ b/content/learn/oip/performance-evaluation.mdx @@ -9,54 +9,30 @@ Consider the following classifications: - **Functional or Passed Gateways**: are gateways that meet or surpass the network's performance and quality standards, including ArNS resolution and chunk/offset validation (if selected). - **Deficient or Failed Gateways**: are gateways that fall short of the network's performance expectations, including failures in ArNS resolution or chunk/offset validation. -- **Functional or Submitted Observers**: are selected observers who diligently perform their duties and submit observation reports and contract interactions. -- **Deficient or Failed Observers**: are selected observers who do not fulfill their duty of submitting observation reports and contract interactions. +- **Functional or Submitted Observers**: are selected observers who diligently perform their duties and submit observation reports and onchain observations. +- **Deficient or Failed Observers**: are selected observers who do not fulfill their duty of submitting observation reports and onchain observations. ## Evaluation Process -At the end of an epoch, the ar.io Smart Contract processes observer submissions to determine gateway performance through a consensus-based vote tallying system. This evaluation transforms individual observer reports into network-wide performance assessments. +At the end of an epoch, the ar.io protocol processes observer submissions to determine gateway performance through a consensus-based vote tallying system. This evaluation transforms individual observer reports into network-wide performance assessments. ### Vote Tallying and Gateway Classification -After observers submit their detailed reports (see [Reporting](/learn/oip/reporting) for submission details), the smart contract performs consensus calculation: +After observers submit their detailed reports (see [Reporting](/learn/oip/reporting) for submission details), the protocol performs consensus calculation: **Vote Processing:** -- **Data Collection**: All observer contract interactions for each gateway are collected +- **Data Collection**: All observer onchain observations for each gateway are collected - **Vote Counting**: Each observer submission contributes either a PASS or FAIL vote -- **Majority Determination**: If ≥50% of submitted observer interactions indicate PASS, the gateway is considered Functional +- **Majority Determination**: If enough submitted observer interactions indicate PASS, the gateway is considered Functional - **Binary Classification**: Gateways are classified as either Functional (eligible for rewards) or Deficient (ineligible for rewards) **Consensus Mechanism:** - Multiple observers evaluate each gateway independently, ensuring reliable assessment -- The 50% threshold requires majority agreement for positive performance determination +- A majority-style threshold requires broad observer agreement for positive performance determination - Binary scoring provides clear, unambiguous performance classification -- Vote tallying occurs after the 40-minute confirmation period to ensure all interactions are finalized - -## Chunk/Offset Validation Criteria - -For gateways selected for chunk/offset validation, additional performance criteria are evaluated: - -### Validation Requirements - -- **Chunk Retrieval**: Gateway must successfully respond to `GET /chunk/{offset}` requests -- **Data Integrity**: Chunk data must be non-empty and within reasonable size limits (<1MB) -- **Merkle Proof Validation**: Cryptographic proof must decode correctly and validate against transaction data_root -- **Performance Standards**: Response times must meet network expectations - - - **Initial Implementation**: During the initial rollout phase, only a very - small portion of gateways will be checked for chunk/offset validation, and the - current validation criteria are extremely lenient to ensure smooth network - operation. - - -### Assessment Process - -- **Binary Scoring**: Each offset test results in pass/fail determination -- **Consensus Integration**: Chunk/offset results are integrated into overall gateway assessment -- **Performance Tracking**: Individual offset assessments are tracked and reported +- Vote tallying occurs after submissions are finalized ## Weight Impact on Gateway Performance @@ -64,13 +40,7 @@ Gateway performance directly affects multiple weighted factors that influence fu ### Gateway Performance Ratio Weight (GPRW) -A gateway's evaluation results directly impact their Gateway Performance Ratio Weight, which affects their likelihood of being selected as an observer in future epochs: - -```math -GPRW = \frac{1 + \text{Passed Epochs}}{1 + \text{Participated Epochs}} -``` - -**Impact:** +A gateway's evaluation results directly impact their Gateway Performance Ratio Weight, which affects their likelihood of being selected as an observer in future epochs. - **Functional Gateways**: Increase their passed epochs count, improving their GPRW - **Deficient Gateways**: Decrease their GPRW as participated epochs increase without corresponding passes @@ -78,32 +48,20 @@ GPRW = \frac{1 + \text{Passed Epochs}}{1 + \text{Participated Epochs}} ### Observer Performance Ratio Weight (OPRW) -For gateways selected as observers, their performance in submitting reports affects future selection: - -```math -OPRW = \frac{1 + \text{Submitted Epochs}}{1 + \text{Selected Epochs}} -``` - -**Impact:** +For gateways selected as observers, their performance in submitting reports affects future selection. - **Functional Observers**: Who submit reports increase their OPRW - **Deficient Observers**: Who fail to submit reports see their OPRW decrease - **Future Selection**: Higher OPRW improves chances of future observer selection -### Composite Weight Calculation - -All performance factors combine to determine overall network influence: - -```math -CW = SW \times TW \times GPRW \times OPRW -``` +### Composite Weight -Where: +Performance factors combine to determine overall network influence: -- **SW** = Stake Weight (financial commitment) -- **TW** = Tenure Weight (network longevity) -- **GPRW** = Gateway Performance Ratio Weight -- **OPRW** = Observer Performance Ratio Weight +- **Stake Weight**: Financial commitment +- **Tenure Weight**: Network longevity +- **Gateway Performance Ratio Weight**: Gateway reliability history +- **Observer Performance Ratio Weight**: Observation duty history **Long-term Effects:** @@ -113,13 +71,13 @@ Where: ## Evaluation Timeline -Rewards are distributed **at the end of each epoch** by the ar.io Smart Contract directly based on the tallied observer votes. The smart contract processes all observer submissions and automatically distributes rewards to functional gateways and observers based on their performance during the epoch. +Rewards are distributed **at the end of each epoch** by the ar.io protocol based on the tallied observer votes. The protocol processes observer submissions and distributes rewards to functional gateways and observers based on their performance during the epoch. ## Key Features - **Majority Rule**: Gateway performance is determined by majority vote from observers - **Binary Scoring**: Simple pass/fail system for clear performance assessment -- **Network Confirmation**: Delay ensures all votes are confirmed before evaluation +- **Network Confirmation**: Evaluation happens after observer submissions are finalized - **Transparent Process**: All evaluations are based on onchain data ## Consequences of Performance @@ -140,10 +98,10 @@ Rewards are distributed **at the end of each epoch** by the ar.io Smart Contract - Functional observers receive observer rewards - Deficient observers forfeit observer rewards -- Deficient observers who are also functional gateways have their gateway reward reduced by 25% +- Deficient observers who are also functional gateways can have their gateway reward reduced for that epoch --- ## Next Steps -Ready to understand how rewards are distributed? Learn about [Reward Distribution](/learn/oip/reward-distribution) to see the formulas and mechanics, or go back to [Observer Selection](/learn/oip/observer-selection) to review the selection process. +Ready to understand how rewards are distributed? Learn about [Reward Distribution](/learn/oip/reward-distribution), or go back to [Observer Selection](/learn/oip/observer-selection) to review the selection process. diff --git a/content/learn/oip/pruning.mdx b/content/learn/oip/pruning.mdx new file mode 100644 index 000000000..2c8b33b04 --- /dev/null +++ b/content/learn/oip/pruning.mdx @@ -0,0 +1,58 @@ +--- +title: "Gateway Pruning" +description: "How persistently underperforming gateways can be removed from the ar.io network." +--- + +## Overview + +Gateway pruning is the mechanism that removes gateways that continue to fail network performance checks. It helps keep the Gateway Address Registry focused on infrastructure that is reachable, useful, and aligned with the network's quality expectations. + +Pruning is based on OIP performance results. A gateway is not removed for a single bad result, but sustained failure can make it eligible for removal. + +## How Pruning Works + +When a gateway remains deficient for a sustained period: + +1. **The gateway becomes prunable** after repeated failed observations. +2. **A permissionless instruction can remove it** from the Gateway Address Registry. +3. **The operator's required network-join stake is slashed** to the protocol balance. +4. **Eligible excess and delegated stake follow the normal withdrawal process** instead of being slashed. + +```mermaid +graph TD + A[Gateway participates in OIP] --> B{Performance result} + B -->|Pass| C[Failure history resets] + B -->|Fail| D[Failure history accumulates] + D --> E{Sustained failure?} + E -->|No| A + E -->|Yes| F[Gateway becomes prunable] + F --> G[Gateway removed from registry] + G --> H[Required operator stake slashed] + G --> I[Eligible excess and delegated stake withdrawable] +``` + +## Impact on Operators + +Pruning is designed to make gateway operation economically accountable. Operators are expected to keep their gateways online, correctly configured, able to resolve ArNS names, and able to participate in observation duties when selected. + +When a gateway is pruned, the operator loses the required stake associated with joining the network. Any eligible excess stake follows the normal withdrawal flow. + +## Impact on Delegators + +Delegated stake is not the target of pruning slashing. If a gateway is pruned, delegated stake follows the normal withdrawal process for leaving gateways. Delegators should still monitor gateway performance because pruning can interrupt reward eligibility and require withdrawal or redelegation decisions. + +## Prevention + +Gateway operators can reduce pruning risk by: + +- Maintaining reliable uptime and public reachability +- Keeping ArNS resolution working correctly +- Monitoring OIP performance results and gateway health +- Funding the observer wallet with SOL for required transaction fees +- Reviewing gateway status in [gateways.ar.io](https://gateways.ar.io) + +## Related Concepts + +- [Gateway Registry](/learn/gateways/gateway-registry) +- [Observation and Incentive Protocol](/learn/oip) +- [Staking](/learn/oip/staking) diff --git a/content/learn/oip/reporting.mdx b/content/learn/oip/reporting.mdx index a551e6668..2f5b82dfb 100644 --- a/content/learn/oip/reporting.mdx +++ b/content/learn/oip/reporting.mdx @@ -1,6 +1,6 @@ --- title: "Reporting" -description: "Learn about observer responsibilities for submitting reports to Arweave and the ar.io Smart Contract" +description: "Learn about observer responsibilities for submitting reports to Arweave and compact onchain results to the ar.io protocol" --- ## Observer Responsibilities @@ -17,7 +17,7 @@ Observers must submit their findings through both channels to fulfill their duti - **Purpose**: Permanent audit trail and transparency - **Content**: Complete test results, timing data, and failure details -### 2. Contract Interactions to ar.io Smart Contract +### 2. Onchain Observations to the ar.io Protocol - **Format**: List of failed gateways - **Purpose**: Efficient vote tallying for consensus @@ -25,7 +25,7 @@ Observers must submit their findings through both channels to fulfill their duti ## Observer Evaluations -Observers test assigned gateways against 10 ArNS names (2 prescribed + 8 chosen) and document their findings: +Observers test assigned gateways against prescribed ArNS names and document their findings: ![Observer Report Overview showing multiple gateway evaluations](/content/observer-report.png) @@ -38,10 +38,10 @@ Observers test assigned gateways against 10 ArNS names (2 prescribed + 8 chosen) alt="Successful gateway evaluation report" style={{ width: "100%" }} /> -

+

Passing Report: Gateway successfully resolves ArNS names - with correct status codes (200), transaction IDs, and data hashes. -

+ with expected status codes, transaction IDs, and response data. +
Failed gateway evaluation report -

+

Failing Report: Gateway fails ArNS resolution tests due - to ownership issues, timeouts (5000ms), or missing content. -

+ to ownership issues, timeouts, or missing content. +
@@ -62,46 +62,6 @@ Observers evaluate gateways based on: - **ArNS Resolution**: Tests successful name-to-transaction resolution - **Content Hash Verification**: Ensures data integrity - **Response Times**: Measures performance within limits -- **Chunk/Offset Validation**: Cryptographic verification of data chunks (for selected gateways) - -## Chunk/Offset Assessment Reporting - -For gateways selected for chunk/offset validation, observers perform additional testing and reporting: - -### Validation Process - -- **Offset Selection**: Random offsets within the stable weave range are chosen for testing -- **Chunk Retrieval**: Observers request chunk data using `GET /chunk/{offset}` endpoint -- **Merkle Proof Verification**: Cryptographic validation ensures chunk authenticity -- **Binary Search**: Efficient transaction lookup using cached metadata for proof validation - -### Reporting Details - -- **Individual Assessments**: Each offset test is tracked with pass/fail/skipped status -- **Enforcement Status**: Reports include whether chunk/offset failures affect gateway status -- **Performance Metrics**: Response times and validation results are documented -- **Early Stopping**: Tests stop immediately upon first successful validation - -### Report Structure - -```json -{ - "offsetAssessments": { - "plannedOffsets": [12345, 67890, ...], - "actualAssessments": [...], - "validatedOffset": 12345, - "pass": true, - "enforcementEnabled": true - } -} -``` - - - **Initial Implementation**: During the initial rollout phase, only a very - small portion of gateways will be checked for chunk/offset validation, and the - current validation criteria are extremely lenient to ensure smooth network - operation. - ## Observer Rewards and Penalties @@ -109,14 +69,14 @@ Observer performance directly impacts rewards and future participation: ### Successful Observer Performance -- **Observer Reward**: Observers who submit both reports and contract interactions receive the Observer Reward +- **Observer Reward**: Observers who submit both reports and onchain observations receive the Observer Reward - **Future Selection**: Successful reporting improves Observer Performance Ratio Weight (OPRW) - **Increased Chances**: Higher OPRW increases likelihood of future observer selection and more reward opportunities ### Failed Observer Performance - **No Observer Reward**: Observers who fail to submit required reports forfeit their Observer Reward -- **Gateway Penalty**: If the deficient observer is also a functional gateway, their gateway reward is **reduced by 25%** +- **Gateway Penalty**: If the deficient observer is also a functional gateway, their gateway reward can be reduced for that epoch - **Reduced Selection**: Failed submissions decrease OPRW, diminishing future observer selection chances - **Lost Opportunities**: Lower selection probability means fewer chances to earn Observer Rewards @@ -124,7 +84,7 @@ Observer performance directly impacts rewards and future participation: The system tracks observer performance to ensure network quality: -- **Submission Tracking**: Both Arweave reports and contract interactions must be submitted +- **Submission Tracking**: Both Arweave reports and onchain observations must be submitted - **Performance History**: Observer submission record affects future selection probability - **Reward Impact**: Consistent reporting builds credibility and increases earning potential diff --git a/content/learn/oip/reward-distribution.mdx b/content/learn/oip/reward-distribution.mdx index 7692526b5..dccbcb088 100644 --- a/content/learn/oip/reward-distribution.mdx +++ b/content/learn/oip/reward-distribution.mdx @@ -9,7 +9,7 @@ The ar.io network maintains a protocol balance that funds all gateway and observ ## Epoch Allocation -Each epoch, a portion of the protocol balance is earmarked for distribution as rewards. This value shall begin at 0.1% per epoch for the first year of operation, then linearly decline down to and stabilize at 0.05% over the following 6 months. +Each epoch, a portion of the protocol balance is earmarked for distribution as rewards. The protocol uses this allocation to reward functional gateways and observers. ### Funding Sources - **ArNS Name Purchases**: Primary funding mechanism - fees from ArNS name registrations and renewals @@ -22,19 +22,11 @@ From this allocation, two distinct reward categories are derived: ### Base Gateway Reward (BGR) -This is the portion of the reward allocated to each Functional Gateway within the network and is calculated as: - -```math -BGR = \frac{\text{Epoch Reward Allocation} \times 0.9}{\text{Total Gateways in the Network}} -``` +This is the portion of the epoch reward allocation distributed to functional gateways. ### Base Observer Reward (BOR) -Observers, due to their additional responsibilities, have a separate reward calculated as: - -```math -BOR = \frac{\text{Epoch Reward Allocation} \times 0.1}{\text{Total Selected Observers for the Epoch}} -``` +Observers, due to their additional responsibilities, have a separate reward category for successfully performing observation duties. ## Distribution Based on Performance @@ -43,29 +35,38 @@ The reward distribution is contingent on the performance classifications derived - **Functional Gateways**: Gateways that meet the performance criteria receive the Base Gateway Reward. - **Deficient Gateways**: Gateways falling short in performance do not receive any gateway rewards. - **Functional Observers**: Observers that fulfilled their duty receive the Base Observer Reward. -- **Deficient Observers**: Observers failing to meet their responsibilities do not receive observer rewards. Furthermore, if they are also Functional Gateways, their gateway reward is reduced by **25%** for that epoch as a consequence for not performing their observation duty. +- **Deficient Observers**: Observers failing to meet their responsibilities do not receive observer rewards. If they are also functional gateways, their gateway reward can be reduced for that epoch as a consequence for not performing their observation duty. ![Epoch reward distributions showing eligible vs distributed ARIO tokens](/content/epoch-distributions.png) -

+

Epoch reward distributions showing the relationship between eligible rewards (total available) and distributed rewards (actually paid out) across epochs. The difference represents rewards not distributed due to gateway or observer deficiencies. -

+
-## Auto-Staking +## Epoch Pipeline -Gateways shall be given the option to have their reward tokens "auto-staked" to their existing stake or sent to their wallet as unlocked tokens. The default setting shall be "auto-staked". +On Solana, reward distribution is driven by a permissionless 6-step [epoch pipeline](/learn/oip/epoch-pipeline) rather than a single atomic operation: -## Distribution to Delegates +1. **create_epoch** — Initialize epoch, compute reward rate +2. **tally_weights** — Batched weight computation +3. **prescribe_epoch** — Select observers and prescribed names via weighted roulette +4. **save_observations** — Observers submit pass/fail reports +5. **distribute_epoch** — Batched reward distribution +6. **close_epoch** — Reclaim rent from completed epoch accounts + +All steps are permissionless and idempotent — anyone can crank them, and running them multiple times is safe. -The protocol will automatically distribute a Functional Gateway's shared rewards with its delegates. The distribution will consider the gateway's total reward for the period (including observation rewards), the gateway's "Delegate Reward Share Ratio", and each delegate's stake proportional to the total delegation. +## Operator Rewards -Each individual delegate reward is calculated as: +Operator rewards always **auto-compound** into the operator's stake. There is no toggle — operators must call `decrease_operator_stake` to realize rewards as liquid tokens. -```math -DR_i = \text{Total Rewards} \times \text{Reward Share Ratio} \times \frac{\text{Delegate's Stake}}{\text{Total Delegated Stake}} -``` +## Distribution to Delegates + +Delegate rewards use a **reward-per-share accumulator** pattern. Rather than distributing rewards to each delegate individually each epoch, the protocol tracks a cumulative reward-per-token on each gateway. Pending rewards are settled when the delegator interacts with the protocol (e.g., delegate more, withdraw, or claim rewards). -Unlike gateways, token reward distributions to delegated stakers will only be "auto-staked" in that they will be automatically added to the delegate's existing stake associated with the rewarded gateway. The delegated staker is then free to withdraw their staked rewards at any time (subject to withdrawal delays). +Leaving gateways receive 0 rewards for any epoch in which they have initiated withdrawal. + +Delegate reward distribution considers the gateway's total reward, the gateway's delegate reward share setting, and each delegate's proportional stake. Delegated rewards are added to the delegate's existing stake for that gateway and can later be withdrawn subject to normal withdrawal rules. ## Undistributed Rewards @@ -77,9 +78,7 @@ Note that if a gateway (and its delegates) leaves the network or a delegate full ## Handling Deficient Gateways -To maintain network efficiency and reduce contract state bloat, gateways that are marked as deficient, and thus fail to receive rewards, for **thirty (30)** consecutive epochs will automatically trigger a "Network Leave" action and be subject to the associated stake withdrawal durations for both gateway stake and any delegated stake. - -In addition, the gateway shall have its **minimum network-join stake slashed by 100%**. The slashed stake shall be immediately sent to the protocol balance. +To maintain network efficiency and reduce state bloat, gateways that remain deficient for a sustained period can be removed from the network. When this happens, their minimum network-join stake is slashed to the protocol balance, while eligible excess and delegated stake follow the standard withdrawal process. --- diff --git a/content/learn/oip/staking.mdx b/content/learn/oip/staking.mdx new file mode 100644 index 000000000..5177e5a32 --- /dev/null +++ b/content/learn/oip/staking.mdx @@ -0,0 +1,188 @@ +--- +title: "Staking" +description: "Learn how staking ARIO tokens enables gateway operators and delegators to participate in ar.io and earn protocol rewards" +--- + +import { Card, Cards } from 'fumadocs-ui/components/card'; +import { Target, Network, Shield, Coins } from 'lucide-react'; + +## Overview + +Staking tokens within ar.io serves a dual primary purpose: it signifies a public commitment by gateway operators and qualifies them and their delegates for reward distributions. + +In the ar.io ecosystem, "staking" refers to the process of locking ARIO tokens into protocol-controlled vaults on Solana. This act signifies an opportunity cost for the staker, acting both as a motivator and a public pledge to uphold the network's collective interests. Once staked, tokens remain locked until the staker initiates an unstake or withdrawal action, or reaches the end of the vault's lock period. + +It is important to note that the ARIO Token is non-inflationary, distinguishing ar.io's staking mechanism from yield-generation tools found in other protocols. Staking in this context is about eligibility for potential rewards rather than direct token yield. By staking tokens, gateway operators (and their delegates) demonstrate their commitment to the network, thereby gaining eligibility for protocol-driven rewards and access to the network's shared resources. + + +SOL is required for Solana transaction fees when staking, delegating, redelegating, or withdrawing ARIO. + + +
+