diff --git a/docs/manual-browser-qualification.md b/docs/manual-browser-qualification.md index a1f3495..2fe3905 100644 --- a/docs/manual-browser-qualification.md +++ b/docs/manual-browser-qualification.md @@ -1,276 +1,327 @@ --- title: Manual browser qualification -description: Owner-run browser checklist for every implemented CDNFoundry panel workflow. +description: Owner-run browser and operator checkpoints for the post-baseline CDNFoundry roadmap. --- # Manual browser qualification -This is a manual, owner-run release job. Coding agents must not automate it. -Record date, operator, exact commit, topology, browser/version, desktop and -mobile viewport, operation IDs, domain revisions, certificate fingerprints, -screenshots, actual results, and failures. - -Use disposable application accounts and domains. Do not delete Compose volumes -or use customer data. - -## Preparation - -1. Start the persistent development or approved production-like topology. -2. Apply Laravel and PowerDNS migrations explicitly. -3. Confirm `/api/health`, `/api/ready`, DNSdist UDP/TCP, Horizon, and scheduler. -4. Create one administrator with `cdnf:admin:create`. -5. Prepare one unused domain-user email, one delegated test domain, two edge - addresses, two origin fixtures, and two viewport sizes. -6. Sign in at `/admin`; confirm keyboard focus is visible, labels are associated, - tables scroll on a narrow viewport, and no secret appears in page source or logs. - -## Phase 1: foundation and access - -### Administrator and user - -1. Open **Customers → Users → New user**. -2. Enter name, unique email, type **User**, password, and confirmation. -3. Save. Expect one user row and no plaintext password display. -4. Edit the user and choose disable. Expect disabled status. -5. In another browser profile, attempt `/app` login. Expect denial. -6. Re-enable the user. Expect login to `/app`; no API token is recreated. -7. Confirm an administrator cannot disable, demote, or delete their own row. - -### Profile and tokens - -1. Open **Account → Profile** in each panel. -2. Change display name and save. Expect the header/account widget to update. -3. Change password with current password and matching confirmation. -4. Open **Account → API tokens**; create a named token. -5. Record that plaintext appears once; navigate away and back. -6. Expect only metadata/final six characters. Revoke it and confirm it disappears. - -### System DNS identity - -1. Open **Control plane → System DNS identity**. -2. Enter platform zone, two nameservers, IPv4 and IPv6 glue for each, proxy - hostname, SOA contact, refresh, retry, expire, minimum, and TTL values. -3. Preview. Expect normalized records and a confirmation token. -4. Change one field after preview and try the old token. Expect rejection. -5. Preview again, apply the exact token, and record the returned operation. -6. Expect pending/running then succeeded deployment state without a blocked page. - -### Phase 1 completion gate - -| Gate | Result | -| --- | --- | -| Implementation | Present | -| Documentation | Current | -| Automated/runtime qualification | Record current run | -| Manual browser qualification | Pass only when every Phase 1 step above is recorded | - -## Phase 2: domains and authoritative DNS - -### DNS cluster - -1. Open **Control plane → DNS clusters → New cluster**. -2. Enter unique name, location, HTTPS API URL, API key, server ID, nameserver - list, and optional operational notes. -3. Save. Expect disabled/pending health and no API key in the table or edit form. -4. Edit and choose **Test connection**. Record the operation. -5. After success, enable the cluster and reconcile all zones. -6. Expect active checksums and no replacement of a healthy cluster's old state on a deliberately failed second target. - -### Domain lifecycle and assignment - -1. Open **Domains → New domain** and enter the delegated test domain. -2. Expect `pending_verification`, revision, nameserver guidance, and no origin. -3. Attach the domain user from the **Users** relation. -4. Sign in as that user. Expect only the assigned domain. -5. Choose **Verify nameservers**, wait for success, then **Activate**. -6. Expect active lifecycle and acknowledged DNS deployment. - -### DNS records - -1. Add A, AAAA, CNAME, MX, TXT, NS, CAA, and SRV examples with valid fields. -2. Confirm TTL minimum 30; MX requires priority; SRV requires - `_service._protocol`, priority, weight, and port. -3. Try CNAME beside another record at the same owner. Expect inline rejection. -4. Try an owner outside the zone and a non-SRV underscore. Expect rejection. -5. As a domain user, try to change apex delegation NS. Expect the action hidden or denied. -6. Bulk delete selected non-delegation records. Expect one revision change. -7. Import a small BIND zone with append, then export it. Expect deterministic text. -8. Import with replacement and confirm the final record set. - -### Phase 2 completion gate - -| Gate | Result | -| --- | --- | -| Implementation | Present | -| Documentation | Current | -| Automated/runtime qualification | Record PHP, DNS, and scale evidence | -| Manual browser qualification | Pass only when every Phase 2 step above is recorded | - -## Phase 3: Geo-DNS - -1. Create an A record with mode **Geo-DNS**. -2. Add a default address, one continent override, and one country override. -3. Save. Expect normalized configuration and one domain revision. -4. Use **Preview** with an IP for each rule and an unknown/documentation IP. -5. Expect country before continent before default, with displayed classification. -6. Repeat with AAAA and confirm IPv6 input. -7. Try duplicate codes, no default target, too many targets, and CAA Geo-DNS. -8. Expect validation and no revision change. -9. Verify real DNS from approved external vantage points and record ECS behaviour. - -### Phase 3 completion gate - -| Gate | Result | -| --- | --- | -| Implementation | Present | -| Documentation | Current | -| Automated/runtime qualification | Record PHP and Geo-DNS runtime evidence | -| Manual browser/external vantage qualification | Pass only when every Phase 3 step is recorded | - -## Phase 4: proxy and edge - -### Pools, edges, and enrollment - -1. Create one shared and one quarantine pool with unique stable names. -2. Create two edges with name, country, continent, unique IPv4, and optional IPv6. -3. Record each UUID/bootstrap token once; reload and expect token secrecy. -4. Edit each edge's cells and enter unique public IPv4/optional IPv6. -5. Enroll the real agents. Expect registered identity, fresh heartbeat, - `listener_ready`, ready cells, version, sequence, and bounded capacity. -6. Rotate one identity. Expect the old agent to fail and a new token to appear once. -7. Restore enrollment, then drain/undrain a cell and one edge. Expect durable operations and routing exclusion/reinclusion. - -### Proxied hostname - -1. On the domain DNS relation, add a proxied A, AAAA, or CNAME. -2. Enter origin host, HTTP/HTTPS scheme, host header, verified HTTPS SNI, - connect timeout, response timeout, retry count, WebSocket flag, and optional health-check path/interval. -3. Expect DNS content to become platform managed and the explicit origin to remain visible. -4. Try loopback, link-local, metadata, multicast, edge listener, and proxy hostname origins. Expect rejection. -5. Run **Test origin**. Expect an asynchronous result with address, latency, - status, or stable failure reason. -6. Inspect edge delivery. Expect both eligible agents to acknowledge the revision. -7. Move the domain to the quarantine pool. Expect target ready before source drain. -8. Roll back to a retained revision. Expect a new higher revision, not a decremented number. - -### Phase 4 completion gate - -| Gate | Result | -| --- | --- | -| Implementation | Present | -| Documentation | Current | -| Automated/runtime qualification | Record PHP, Go, control, mTLS, and OpenResty evidence | -| Manual browser/real traffic qualification | Pass only when every Phase 4 step is recorded | - -## Phase 5: TLS, cache, and purge - -### Managed and custom TLS - -1. With an active proxied domain, open the TLS section. -2. Expect managed mode and a visible pending/succeeded order without private data. -3. Verify public apex/wildcard HTTPS and record the fingerprint. -4. Use **Renew**, then **Reissue**; record operation IDs and final certificate state. -5. Upload a valid leaf, chain, and matching private key. Expect custom mode and metadata only. -6. Try a wrong key, wrong name, invalid chain, expired certificate, and oversized PEM. Expect rejection. -7. Remove the custom certificate. Expect managed mode and asynchronous reconciliation. -8. Set TLS disabled and confirm intended HTTPS behaviour; return to managed. - -### Cache - -1. Open **Cache settings** and set enabled, edge/browser TTLs, one allowed object - size, origin-header respect, query policy, bypass cookies, and stale grace. -2. Save and wait for edge acknowledgement. -3. Request a cacheable object twice. Expect MISS then HIT and correct browser headers. -4. Enable development mode for a short duration. Expect absolute expiry and BYPASS. -5. Disable it and expect caching to resume. -6. Purge one exact URL. Expect one task per eligible edge and a later MISS. -7. Purge everything. Expect an incremented epoch, no filesystem scan, and later MISS. -8. Create a controlled cell-delivery failure. Expect visible retry of the same task. - -### Phase 5 completion gate - -| Gate | Result | -| --- | --- | -| Implementation | Present | -| Documentation | Current | -| Automated/runtime qualification | Record PHP, Pebble, cache, purge, and OpenResty evidence | -| Manual browser/public HTTPS qualification | Pass only when every Phase 5 step is recorded | - -## Phase 6: security and DDoS readiness - -1. Open **Security profile and limits**. -2. Select standard, protected, quarantine, then manual. Expect fixed profile - values and manual values bounded by platform ceilings. -3. Select a quarantine policy, allowed methods, and trusted proxy CIDRs; save. -4. Add IP, CIDR, country, and continent allow/block rules with priorities. -5. Import rules with append and replacement. Expect duplicate/limit validation. -6. Send matching IPv4 and IPv6 traffic. Expect rule order and reason visibility. -7. Use **Restrict domain**, **Quarantine domain**, then **Release domain**. -8. Expect target-first placement and `recovering` before normal where applicable. -9. Apply and clear an expiring emergency mode to an edge and cell. -10. Withdraw and restore a pool. Expect DNS publication to change without corrupting another pool. -11. Confirm traffic to an unrelated cell/domain remains healthy throughout. - -### Phase 6 completion gate - -| Gate | Result | -| --- | --- | -| Implementation | Present | -| Documentation | Current | -| Automated/runtime qualification | Record PHP and security real-runtime evidence | -| Manual browser/real attack-pattern qualification | Pass only when every Phase 6 step is recorded | - -## Phase 7: analytics and usage - -1. Generate controlled DNS and HTTP traffic, cache hits/misses, origin failure, - TLS, and security events. -2. As the domain user, open **Observe → Analytics**. -3. Check summary, timeseries, status, cache, countries, hostnames, URLs, origin, - edges, and DNS with range and unit labels. -4. Open request, DNS, error, and security logs where surfaced. Expect opaque - pagination, no query strings/secrets, and masked IPv4/IPv6. -5. Download domain usage CSV. Expect stable columns and only the assigned domain. -6. As administrator, open **Observe → Telemetry** and global views/export. -7. Stop ClickHouse in the controlled environment. Expect a visible outage, not - a broken DNS or edge service. -8. Restore ClickHouse and confirm Vector backlog behaviour and partial interval labels. - -### Phase 7 completion gate - -| Gate | Result | -| --- | --- | -| Implementation | Present | -| Documentation | Current | -| Automated/runtime qualification | Record PHP and analytics outage/recovery evidence | -| Manual browser qualification | Pass only when every Phase 7 step is recorded | - -## Phase 8: operations and release - -1. Open the administrator dashboard. Confirm overall status, component detail, - all four queue lanes, recent operations, and no secret values. -2. Open **Operations**. Inspect pending, succeeded, and failed rows; retry one supported failure. -3. Open failed jobs through the API/operational surface. Confirm payloads are bounded and redacted. -4. Open **Platform settings**. Change a non-runtime group and a runtime group. -5. Expect typed bounds, audit history, and a global operation only for runtime-affecting policy. -6. Create a backup. Expect snapshot metadata and verification without download. -7. Start restore preflight with exact confirmation and current password. -8. On an isolated host, perform the maintenance restore and complete the recovery checklist. -9. Run one canary upgrade and rollback with prior/current compatible images. -10. Verify desktop/mobile layout, keyboard operation, error focus, tables, - destructive confirmations, CSV downloads, and one-time secret boundaries. - -### Phase 8 completion gate - -| Gate | Result | -| --- | --- | -| Implementation | Present | -| Documentation | Current | -| Automated/runtime qualification | Record operations, recovery, upgrade, throughput, MMDB, Compose, and image evidence | -| Manual browser/production qualification | Pass only when every Phase 8 and external release step is recorded | - -## Result record - -For each failed checkpoint, record expected result, actual result, sanitized -evidence, operation/task/revision IDs, severity, owner, and retest. The release -remains unqualified until every current checkpoint passes or an explicit -product-contract change removes it. +## Purpose + +This document mirrors `docs/roadmap.md` and covers only the new post-baseline +phases. The completed original platform remains covered by regression checks and +is not repeated as roadmap work here. + +Browser qualification is owner-run. Coding agents maintain this checklist but +must report it as not executed unless the owner records the evidence. + +Do not invent UI controls before they exist. During implementation, replace each +phase's activation contract with exact menu names, fields, buttons, URLs, and +expected results from the real UI. + +## Evidence template + +Record the following for every qualification run: + +```text +Date: +Operator: +Commit SHA: +Release/image tags: +Environment: +Control-plane URL: +Browser/version: +Desktop viewport: +Mobile viewport: +Edges/POPs: +Cell slots per edge: +Pools and routing modes: +Public IPv4/IPv6 endpoints: +Test domains: +Origin fixtures: +Start time: +End time: +Overall result: passed / failed / blocked +``` + +For every checkpoint, record: + +- expected result; +- actual result; +- sanitized screenshot or screen recording; +- relevant operation, revision, edge, pool, cell, endpoint, certificate, purge, + origin, WAF, or rollout identifier; +- browser console/network error when relevant; +- failure severity and owner; +- retest evidence. + +Never record passwords, tokens, API keys, private keys, signing keys, backup +credentials, raw customer telemetry, or customer data. + +## Global preparation + +1. Use a disposable or approved production-like environment. +2. Preserve persistent Compose volumes and existing development data. +3. Confirm the exact commit and immutable image tags under qualification. +4. Confirm implemented health/readiness endpoints are healthy. +5. Prepare one administrator and one disposable domain user. +6. Prepare delegated test domains and IPv4/IPv6 origin fixtures. +7. Prepare one failing origin and approved external DNS/HTTP vantage points. +8. Use desktop and narrow mobile viewports. +9. Check keyboard navigation, focus, labels, validation focus, responsive tables, + loading, empty, degraded, error, destructive-confirmation, and one-time-secret states. +10. Confirm no secret appears in page source, browser storage beyond intended + session data, console output, rendered errors, audit details, or exports. +11. Run the completed baseline regression smoke suite before releasing any new phase. + +# Phase 1 qualification — Edge gateway ingress + +Activate this section when gateway endpoint and routing management exist in the UI. + +- [ ] Create or edit an edge gateway configuration. +- [ ] Add valid IPv4 and IPv6 service addresses. +- [ ] Confirm duplicate, malformed, and conflicting addresses are rejected. +- [ ] Inspect generated Host/SNI routing state without exposing secrets. +- [ ] Apply a valid routing revision and observe pending, active, and acknowledged states. +- [ ] Attempt an invalid revision and confirm the previous valid map remains active. +- [ ] Send real HTTP Host traffic to each configured service address. +- [ ] Send real HTTPS SNI traffic to each configured service address. +- [ ] Confirm unknown Host, unknown SNI, and unassigned destination traffic are rejected. +- [ ] Restart the gateway and confirm active routing is restored. +- [ ] Verify gateway health, listener, revision, connection, and error visibility. +- [ ] Confirm existing baseline edge traffic remains functional. + +**Phase 1 result:** passed / failed / blocked / not activated + +# Phase 2 qualification — Bounded cell inventory + +Activate this section when bounded cell-slot installation and inventory exist. + +- [ ] Configure a bounded slot count for a disposable edge. +- [ ] Install or reconcile the edge and confirm exactly that number of slots exists. +- [ ] Confirm every slot has unique identity, ports, paths, health, and resource limits. +- [ ] Inspect assigned, unassigned, ready, degraded, drained, and stopped states. +- [ ] Assign and unassign a slot through the supported workflow. +- [ ] Drain and restore one slot. +- [ ] Restart one slot and confirm unrelated slots continue serving. +- [ ] Trigger a controlled unhealthy slot and confirm gateway and agent remain healthy. +- [ ] Confirm the agent UI/API exposes no unrestricted container command surface. +- [ ] Verify cache, temporary, and log quota visibility. +- [ ] Confirm enrollment, mTLS identity, and snapshot recovery regression checks pass. + +**Phase 2 result:** passed / failed / blocked / not activated + +# Phase 3 qualification — Multi-cell pools and placement + +Activate this section when pool membership and placement management exist. + +- [ ] Create or inspect shared, reserved, dedicated, and quarantine pool kinds. +- [ ] Add at least three cells from one edge to one shared pool. +- [ ] Configure minimum-ready-cell and capacity policy. +- [ ] Assign several test domains and confirm stable placement. +- [ ] Add an unrelated domain and confirm existing placements do not reshuffle unnecessarily. +- [ ] Confirm a dedicated pool rejects a second domain. +- [ ] Move a domain to another cell and observe target preparation before route switch. +- [ ] Force target preparation failure and confirm source placement remains active. +- [ ] Complete a successful move and confirm source state is removed only after cutover. +- [ ] Move a domain into and out of quarantine. +- [ ] Confirm non-participating cells do not receive domain artifacts. +- [ ] Verify unrelated domains and caches remain healthy during movement. + +**Phase 3 result:** passed / failed / blocked / not activated + +# Phase 4 qualification — Pool endpoints and Geo-Unicast + +Activate this section when pool endpoint management exists. + +- [ ] Add a dual-stack service endpoint to a shared pool on one edge. +- [ ] Add a different service pair to a reserved pool on the same edge. +- [ ] Confirm management and service addresses remain distinct. +- [ ] Confirm address conflicts are rejected before activation. +- [ ] Enable the endpoints and observe gateway binding and readiness. +- [ ] Verify system-managed DNS publishes only ready endpoints. +- [ ] Test country, continent, and fallback Geo-Unicast answers. +- [ ] Withdraw one endpoint and confirm only its pool changes. +- [ ] Restore the endpoint and confirm DNS and gateway state converge. +- [ ] Test IPv4-only, IPv6-only, and dual-stack endpoint behavior. +- [ ] Restart/reconcile gateway, DNS, and cells and confirm consistent state. + +**Phase 4 result:** passed / failed / blocked / not activated + +# Phase 5 qualification — Simple Anycast + +Activate this section only when an approved multi-POP routing environment exists. + +- [ ] Create or edit an eligible pool with Simple Anycast routing mode. +- [ ] Configure one shared IPv4 and optional IPv6 service pair. +- [ ] Attach at least two participating POPs/edges. +- [ ] Confirm the same pair is bound locally on each participating edge. +- [ ] Confirm assigned domains publish the shared pair. +- [ ] Verify ready, degraded, and withdrawn states are clear. +- [ ] Test traffic from approved external vantage points. +- [ ] Remove one POP through the external routing process and record convergence. +- [ ] Confirm another POP continues serving valid local state. +- [ ] Restore the POP and record route and serving recovery. +- [ ] Confirm the UI states that the operator/provider owns BGP advertisement. +- [ ] Confirm no router credentials or arbitrary routing command surface exists. + +**Phase 5 result:** passed / failed / blocked / not activated + +# Phase 6 qualification — Cache v2 + +Activate this section when Cache v2 settings and status exist. + +- [ ] Select each available bounded pool cache profile. +- [ ] Configure cache size, temporary quota, inactive time, object limit, and free-space policy. +- [ ] Exercise include-all, ignore-all, include-selected, and ignore-selected query policies. +- [ ] Confirm deterministic cache-key behavior with repeated requests. +- [ ] Verify approved status-code TTL policies. +- [ ] Confirm MISS, HIT, BYPASS, STALE, and revalidation states where applicable. +- [ ] Exercise stale-if-error and stale-while-revalidate. +- [ ] Exercise cache-only and stale-only emergency modes. +- [ ] Restart a cell and confirm persistent cache behavior. +- [ ] Execute exact URL purge and full epoch purge. +- [ ] Trigger low-disk and high-variant protection in the approved lab. +- [ ] Confirm one domain cannot exhaust unrelated cache resources beyond declared limits. +- [ ] Verify disk, temporary storage, admissions, evictions, and hit-ratio visibility. + +**Phase 6 result:** passed / failed / blocked / not activated + +# Phase 7 qualification — Gzip and Brotli + +Activate this section when compression profiles and telemetry exist. + +- [ ] Test off, standard, and maximum-savings profiles where permitted. +- [ ] Request identical content with identity, Gzip, and Brotli support. +- [ ] Confirm decoded content is identical. +- [ ] Confirm one canonical cache object serves different encodings correctly. +- [ ] Verify MIME allowlist and minimum-size behavior. +- [ ] Confirm images, video, archives, and other compressed formats are not recompressed. +- [ ] Test HEAD, 304, ETag, Vary, stale, and purge behavior. +- [ ] Test range and large-response handling. +- [ ] Trigger compression concurrency or CPU-pressure limits in the approved lab. +- [ ] Confirm safe fallback or emergency disable without traffic interruption. +- [ ] Verify encoding, ratio, origin bytes, served bytes, savings, and fallback telemetry. +- [ ] Confirm shared pools cannot select unsafe compression settings. + +**Phase 7 result:** passed / failed / blocked / not activated + +# Phase 8 qualification — Origin failover + +Activate this section when backup-origin configuration exists. + +- [ ] Configure one valid primary and one valid backup origin. +- [ ] Confirm unsafe, looping, private/disallowed, and malformed origins are rejected. +- [ ] Verify normal traffic uses the primary. +- [ ] Trigger a qualified primary failure and observe transition to backup. +- [ ] Confirm transition reason and timestamps are visible without secrets. +- [ ] Restore primary health and verify hold-down and delayed failback. +- [ ] Confirm repeated health changes do not cause flapping. +- [ ] Fail both origins and verify stale, cache-only, or maintenance behavior. +- [ ] Stop the control plane and confirm local failover behavior remains available. +- [ ] Attempt invalid backup changes and confirm active valid state remains. +- [ ] Confirm one failing origin does not exhaust unrelated origin budgets. + +**Phase 8 result:** passed / failed / blocked / not activated + +# Phase 9 qualification — Managed OWASP CRS WAF + +Activate this section when managed WAF profiles exist. + +- [ ] Apply off, monitor, balanced, and strict profiles to approved test domains/pools. +- [ ] Confirm monitor mode records safe test detections without blocking. +- [ ] Confirm blocking profiles reject approved test attack patterns with stable reasons. +- [ ] Verify privacy-safe rule ID, category, score, action, and timing visibility. +- [ ] Add an approved bounded exclusion with reason, owner, and expiry. +- [ ] Confirm the exclusion affects only its intended scope. +- [ ] Confirm expired exclusions stop applying and remain audited. +- [ ] Verify arbitrary rules, raw directives, uploads, and runtime downloads are unavailable. +- [ ] Test oversized and malformed body limits in the approved lab. +- [ ] Run a monitor-only canary of a new WAF image/ruleset. +- [ ] Force canary failure and confirm the previous valid WAF runtime remains active. +- [ ] Confirm unrelated non-WAF pools remain healthy during WAF load or failure. + +**Phase 9 result:** passed / failed / blocked / not activated + +# Phase 10 qualification — Observability and capacity + +Activate this section when the new operational views and analytics exist. + +- [ ] Inspect gateway listener, route, map, connection, error, and revision status. +- [ ] Inspect service endpoint, Anycast, pool readiness, and placement status. +- [ ] Inspect cell CPU, memory, connections, cache disk, temporary space, and saturation. +- [ ] Inspect compression bytes, ratios, concurrency, and fallback. +- [ ] Inspect origin health, active origin, failover, and failback state. +- [ ] Inspect WAF profile, action, score, rule category, and processing cost. +- [ ] Confirm domain users see only assigned-domain data. +- [ ] Confirm raw logs and exports remain bounded and redacted. +- [ ] Stop ClickHouse/Vector and confirm serving continues with visible degraded analytics. +- [ ] Restore telemetry and verify bounded backlog recovery. +- [ ] Trigger representative alerts and confirm each links to an actionable runbook. +- [ ] Confirm queries remain responsive at the qualification dataset. + +**Phase 10 result:** passed / failed / blocked / not activated + +# Phase 11 qualification — Fleet rollout automation + +Activate this section when rollout management exists. + +- [ ] Inspect desired and current gateway, agent, normal-cell, and WAF-cell versions. +- [ ] Configure one canary edge/POP and later rollout waves. +- [ ] Start a canary rollout and observe progress and health gates. +- [ ] Confirm later waves do not start before canary success. +- [ ] Force a controlled canary failure and confirm automatic pause. +- [ ] Confirm the failure and pause reason are visible. +- [ ] Roll back and verify the previous compatible runtime is restored. +- [ ] Resume a successful rollout and verify bounded mixed-version serving. +- [ ] Confirm version drift and incompatibility are visible. +- [ ] Confirm no arbitrary command execution or unbounded container creation is exposed. +- [ ] Verify every rollout, pause, resume, and rollback is audited. + +**Phase 11 result:** passed / failed / blocked / not activated + +# Phase 12 qualification — Final production release + +Activate this section after Phases 1 through 11 are implemented and individually qualified. + +- [ ] Record at least two POPs/edges and at least eight cell slots per edge. +- [ ] Record one multi-cell shared pool, one reserved pool, and one quarantine pool. +- [ ] Verify several service pairs on the same edge. +- [ ] Verify Geo-Unicast and approved Simple Anycast traffic externally. +- [ ] Verify stable placement, migration, drain, quarantine, rollback, and recovery. +- [ ] Verify Cache v2 persistence, purge, stale behavior, and pressure controls. +- [ ] Verify Gzip and Brotli behavior and savings. +- [ ] Verify primary/backup origin failover and failback. +- [ ] Verify managed WAF monitor, block, exclusion, canary, and rollback. +- [ ] Stop selected control-plane and telemetry dependencies and confirm existing serving continues. +- [ ] Apply invalid gateway, cell, and WAF candidates and confirm last-valid state. +- [ ] Saturate one approved lab cell and confirm unrelated pools continue. +- [ ] Perform fleet canary upgrade and rollback. +- [ ] Perform clean-host control-plane restore and derived-state reconciliation. +- [ ] Run the completed baseline regression smoke suite. +- [ ] Confirm all relevant documentation and runbooks match the final product. +- [ ] Record measured limits, topology, hardware, RPO, RTO, throughput, latency, + saturation points, known limitations, and unresolved risks. +- [ ] Confirm no unresolved critical or high-severity failure remains. + +**Phase 12 result:** passed / failed / blocked / not activated + +## Release record + +```text +Roadmap phase: +Implementation: passed / failed / blocked +Unit and feature tests: passed / failed / blocked / not applicable +Real-runtime E2E: passed / failed / blocked / not applicable +IPv4 and IPv6: passed / failed / blocked / not applicable +Scale: passed / failed / blocked / not applicable +Failure and recovery: passed / failed / blocked / not applicable +Isolation: passed / failed / blocked / not applicable +Observability: passed / failed / blocked / not applicable +Documentation: passed / failed / blocked +Manual qualification: passed / failed / blocked +Baseline regression: passed / failed / blocked +Release decision: release / do not release +Evidence links: +Known limitations: +Owner approval: +``` diff --git a/docs/roadmap.md b/docs/roadmap.md index 447dee6..09f039c 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -1,226 +1,476 @@ --- -title: Roadmap and implementation status -description: CDNFoundry product contract, current implementation boundary, completion gates, and future admission rules. +title: CDNFoundry next development roadmap +description: Post-baseline roadmap for the new edge, cache, origin resilience, WAF, and fleet capabilities. --- -# Roadmap and implementation status - -This page is the product contract referenced by `AGENTS.md`. The original -long-form roadmap, appendices, and dated evidence are preserved at -`docs/legacy/roadmap.md`; they are historical, not current documentation. - -## Product contract - -CDNFoundry is a small production-grade private CDN with predictable failure -behaviour, bounded resource use, and low operational complexity. - -- One Laravel modular monolith and two Filament panels own management. -- PostgreSQL owns desired state. -- DNS, HTTP, security decisions, certificate selection, and raw telemetry never - pass through Laravel. -- External effects are asynchronous, revisioned, idempotent, coalesced, and - last-valid-state preserving. -- Scale comes from workers, DNS capacity, ClickHouse capacity, edges, and - bounded OpenResty cells. -- A domain does not receive a default process, container, worker, timer, Nginx - server block, cache directory, or reload. - -Part One is implemented through the code surfaces described below. A phase is -release complete only when implementation, current documentation, -agent-owned automated/runtime qualification, and owner-run manual browser -qualification are each recorded. - -## Phase 1: foundation, access, and system identity - -Implemented behaviour: - -- administrator and assigned-domain-user panels; -- Sanctum login, profile, password, token, disable/enable, audit, and idempotency; -- policy-aware domain and operation binding; -- typed system DNS identity preview, exact confirmation, asynchronous apply; -- health, readiness, operation, Horizon, scheduler, and Compose foundations. - -### Phase 1 completion gate - -| Gate | Status | -| --- | --- | -| Implementation | Present in current code | -| Documentation | Current in this site | -| Automated/runtime qualification | Covered by current PHP and `tests/e2e/e2e.py`; rerun for each release | -| Manual browser qualification | Not recorded for the current revision | - -## Phase 2: domains and authoritative DNS - -Implemented behaviour: - -- normalized domain lifecycle, assignment, delegation verification, activation, - delayed deprovisioning, tombstones, and reclaim cooldown; -- A, AAAA, CNAME, MX, TXT, NS, CAA, SRV, and reverse-zone PTR; -- transactional bulk changes, BIND import/export, monotonic serials; -- DNS cluster management, health tests, deterministic reconciliation, drift and - last-valid-zone preservation; -- DNSdist-only public ingress and separate PowerDNS runtime migrations. - -### Phase 2 completion gate - -| Gate | Status | -| --- | --- | -| Implementation | Present in current code | -| Documentation | Current in domain, DNS, deployment, and reference guides | -| Automated/runtime qualification | Covered by PHP, `phase2_dns.py`, and separate `phase2_scale.py`; rerun for each release | -| Manual browser qualification | Not recorded for the current revision | - -## Phase 3: Geo-DNS - -Implemented behaviour: - -- country, continent, then default answer selection; -- bounded targets and geography vocabulary; -- IPv4/IPv6 preview and unknown fallback; -- deterministic PowerDNS Lua compilation; -- validated, atomically updated local MMDB with last-valid preservation. - -CAA remains DNS-only because it is not in the qualified Geo-DNS runtime type -list. - -### Phase 3 completion gate - -| Gate | Status | -| --- | --- | -| Implementation | Present in current code | -| Documentation | Current in Geo-DNS and DNS references | -| Automated/runtime qualification | Covered by PHP and `phase3_geo_dns.py`; external vantage points remain owner evidence | -| Manual browser qualification | Not recorded for the current revision | - -## Phase 4: proxy, edge routing, and edge agent - -Implemented behaviour: - -- safe explicit origin per proxied hostname; -- shared, quarantine, and exceptional dedicated pool model; -- edge/cell desired state, addresses, enrollment, mutual TLS, identity rotation; -- signed incremental artifacts and bounded full recovery snapshots; -- atomic agent activation, persistent acknowledgements, heartbeat/capacity; -- target-first placement and source drain; -- data-driven OpenResty request path with no per-domain reload. - -### Phase 4 completion gate - -| Gate | Status | -| --- | --- | -| Implementation | Present in Laravel, Go, Nginx, and Lua | -| Documentation | Current in edge, proxy, architecture, API, and deployment guides | -| Automated/runtime qualification | Covered by PHP, Go, `phase4_control_plane.py`, `phase4_mtls.py`, and `phase4_runtime.py`; rerun for each release | -| Manual browser qualification | Not recorded for the current revision | - -## Phase 5: TLS, cache, and purge - -Implemented behaviour: - -- managed ACME DNS-01 only for eligible proxied domains; -- bounded name sets, renewal, retries, cleanup, alerts, and certificate reuse; -- validated encrypted custom certificate upload; -- deterministic cache settings, bypass/admission, development mode, stale grace; -- epoch full purge, exact URL purge, durable per-edge delivery and retry; -- revision rollback without decrementing revision numbers. - -### Phase 5 completion gate - -| Gate | Status | -| --- | --- | -| Implementation | Present in current code | -| Documentation | Current in TLS, cache, API, operations, and troubleshooting | -| Automated/runtime qualification | Covered by PHP, `phase5_tls.py`, and cumulative edge/runtime programs; rerun for each release | -| Manual browser/public HTTPS qualification | Not recorded for the current revision | - -## Phase 6: security and DDoS readiness - -Implemented behaviour: - -- ordered bounded allow/block rules and imports; -- standard, protected, quarantine, and bounded manual profiles; -- IPv4/IPv6 trusted client and rule handling; -- readiness states, bounded reason codes, target-first quarantine and recovery; -- persisted expiring emergency actions for domains, cells, edges, and pools; -- cell resource isolation and explicit volumetric-limit statement. - -### Phase 6 completion gate - -| Gate | Status | -| --- | --- | -| Implementation | Present in control and edge runtimes | -| Documentation | Current in security, hardening, limits, and runbooks | -| Automated/runtime qualification | Covered by PHP and `phase6_security.py`; rerun for each release | -| Manual browser/real-traffic qualification | Not recorded for the current revision | - -## Phase 7: logs, analytics, and usage export - -Implemented behaviour: - -- direct Vector-to-ClickHouse edge and DNS telemetry; -- bounded/redacted schemas and disk buffers; -- raw logs, aggregates, outage/partial labels, IPv4/IPv6 masking; -- domain/admin analytics separation; -- idempotent hourly PostgreSQL usage rollups and stable JSON/CSV export; -- telemetry failure independent from serving. - -### Phase 7 completion gate - -| Gate | Status | -| --- | --- | -| Implementation | Present in current code and telemetry configuration | -| Documentation | Current in analytics, schema, monitoring, and troubleshooting | -| Automated/runtime qualification | Covered by PHP and `phase7_analytics.py`; rerun for each release | -| Manual browser qualification | Not recorded for the current revision | - -## Phase 8: operations and production qualification - -Implemented behaviour: - -- component/queue/capacity health, token-protected metrics, alerts; -- failed-job and operation management, bounded global reconciliation; -- encrypted Restic backup, authenticated restore preflight, maintenance executor; -- explicit migration, production image, split-host, IPv6, and external-data contracts; -- upgrade, recovery, throughput, MMDB, restart, and isolation programs. - -### Phase 8 completion gate - -| Gate | Status | +# CDNFoundry next development roadmap + +## Starting point + +The original production roadmap is complete. The existing control plane, +authoritative DNS, Geo-DNS, proxy, TLS, baseline cache and purge, security, +telemetry, analytics, backup, and operations are the regression baseline. + +This roadmap contains only the new work discussed after that baseline. + +CDNFoundry remains a simple but solid private CDN. It must stay understandable, +bounded, production-safe, and easy to operate. It is not intended to become a +general cloud platform or a Cloudflare replacement. + +## Project boundaries + +- Laravel and Filament remain the single control plane. +- PostgreSQL remains the desired-state source of truth. +- DNS and HTTP traffic never pass through Laravel. +- The edge gateway only binds service addresses and routes by destination IP, + Host, and TLS SNI. +- OpenResty cells own TLS, cache, compression, security, WAF, origin proxying, + and request telemetry. +- Cell slots are bounded and created during edge installation. +- The edge agent never receives unrestricted Docker access. +- No default per-domain process, container, worker, timer, cache directory, + server block, or reload is allowed. +- External effects remain asynchronous, revisioned, idempotent, coalesced, + acknowledged, and last-valid-state preserving. +- No microservices, Kafka, Kubernetes requirement, CQRS, event sourcing, + GraphQL, custom RBAC, plugin runtime, reseller hierarchy, billing engine, + custom WAF language, or second dashboard. +- Simple Anycast uses operator/provider routing. CDNFoundry does not become a + BGP controller. +- DDoS readiness never claims upstream volumetric scrubbing after physical + capacity is saturated. +- Roadmap phase numbers and lifecycle suffixes must not appear in production + code or filenames. + +## Required completion gate + +Every phase must independently record: + +| Gate | Required evidence | | --- | --- | -| Implementation | Present in current code and infrastructure | -| Documentation | Current in deployment, operations, security, development, and upgrade guides | -| Automated/runtime qualification | Covered by PHP and phase 8 Python programs; full external environment evidence remains operator-owned | -| Manual browser and production release qualification | Not recorded for the current revision | - -## Current release boundary - -Because the manual browser and external production evidence is not recorded for -the current revision, Part One is implementation-complete but not represented -here as owner release-qualified. Historical evidence remains in -`docs/legacy/phase-*-qualification.md`. - -## Future admission - -Potential future work includes HTTP/3, alternative certificate workflows, -private origin connectivity, larger fleet automation, extended disaster -recovery, and proven customer extensions. None is implemented or promised by -the current product. - -A future capability is admitted only when: - -- a real operator/user problem and measurable acceptance criteria exist; -- it preserves the product invariants and request-path independence; -- it has typed bounded state, authorization, failure behaviour, rollback, - observability, real-runtime tests, documentation, and manual qualification; -- disabling it leaves Part One functional; -- it does not introduce speculative microservices, Kubernetes requirements, - plugins, billing hierarchies, custom expression languages, or per-domain - runtime processes. - -## Final architecture rule - -Keep management in one understandable Laravel application. Keep DNS and HTTP -traffic independent. Store desired behaviour as validated bounded data. Use the -same reconciliation discipline for every external change. Preserve the last -valid state, isolate edge resources by bounded cells, and never claim -volumetric protection after physical capacity is saturated. +| Implementation | State, migrations, authorization, API, UI where needed, jobs, reconciliation, metrics, audit, and rollback | +| Unit and feature tests | Happy path, permissions, validation, bounds, idempotency, and stable errors | +| Real-runtime E2E | Real DNS, HTTP, HTTPS, TLS, cache, compression, WAF, restart, and failure behavior where applicable | +| IPv4 and IPv6 | Both families pass whenever the phase handles addresses or traffic | +| Scale | Dataset, topology, hardware, concurrency, result, saturation point, and accepted limit | +| Failure and recovery | Retry, obsolete work, invalid candidate, dependency outage, restart, rollback, and last-valid state | +| Isolation | Failure or load in one domain, pool, cell, edge, or component does not unnecessarily affect unrelated traffic | +| Observability | Health, metrics, logs, alerts, capacity, degraded states, and stable reason codes | +| Documentation | User, administrator, API/OpenAPI, architecture, deployment, operations, troubleshooting, and runbooks | +| Manual qualification | Owner-run evidence from `docs/manual-browser-qualification.md` | +| Regression | The completed baseline continues to work | +| Release decision | Passed, failed, blocked, or removed from scope | + +A phase is not complete when an applicable test was not executed. Every phase +must finish as a usable production increment and must not depend on unfinished +work from a later phase. + +# Phase 1 — Edge gateway ingress + +**Goal:** introduce one minimal gateway per edge so one server can bind several +public IPv4/IPv6 service pairs and route traffic to bounded OpenResty cells. + +**Implementation:** + +- Bind operator-configured IPv4 and IPv6 service addresses. +- Route HTTP by destination address and validated Host. +- Route HTTPS by destination address and TLS SNI without terminating customer TLS. +- Forward trusted client identity to cells through a fixed internal contract. +- Reject unknown addresses, Host values, and SNI names before expensive work. +- Activate generated routing maps atomically and retain the previous valid map. +- Expose listener, revision, route, connection, error, and readiness metrics. +- Keep cache, TLS certificates, WAF, origin logic, and control-plane calls out of + the gateway. + +**Scale target:** at least 50,000 Host/SNI mappings and multiple dual-stack +service pairs on one edge, with throughput, latency, CPU, memory, and saturation +recorded. + +**Completion checklist:** + +- [ ] Real HTTP Host and HTTPS SNI traffic reaches the intended cell. +- [ ] IPv4 and IPv6 pass. +- [ ] Unknown and invalid traffic is rejected. +- [ ] Invalid maps never replace active maps. +- [ ] Restart restores or rebuilds the last valid map. +- [ ] Existing baseline edge traffic remains functional during migration. +- [ ] Tests, scale, documentation, and manual qualification pass. + +# Phase 2 — Bounded cell inventory + +**Goal:** replace the fixed shared/quarantine assumption with a bounded set of +generic OpenResty cell slots created during edge installation. + +**Implementation:** + +- Configure a bounded slot count such as `cell-01` through `cell-N`. +- Give every slot stable identity, ports, runtime path, cache path, temporary + path, status endpoint, and resource limits. +- Register assigned, unassigned, ready, degraded, drained, and stopped states. +- Remove readiness logic tied to one hardcoded shared cell. +- Let the agent configure, reload, drain, restart, and report existing slots. +- Keep agent resources separate from cell resources. +- Do not mount the container-engine socket into the agent. + +**Scale target:** qualify at least eight slots on one edge and record idle and +active overhead per slot. + +**Completion checklist:** + +- [ ] Fresh installation creates exactly the configured slots. +- [ ] Every slot has unique identity, paths, ports, health, and limits. +- [ ] One crashed or saturated cell does not stop the gateway, agent, or other cells. +- [ ] Cell restart and rollback preserve unrelated traffic. +- [ ] Cache, temporary, and log storage remain bounded. +- [ ] Existing enrollment, mTLS, and snapshot recovery still pass. +- [ ] Tests, scale, documentation, and manual qualification pass. + +# Phase 3 — Multi-cell pools and stable placement + +**Goal:** allow one pool to use several cells on the same edge while preserving +cache locality and predictable isolation. + +**Implementation:** + +- Support shared, reserved, dedicated, and quarantine pool kinds. +- Separate stable cell identity from pool identity. +- Add explicit edge participation and cell assignment per pool. +- Allow several cells from one edge in one pool. +- Add minimum-ready-cell and capacity policies. +- Place a normal domain on one stable cell per edge by default. +- Keep replicated placement exceptional and bounded. +- Move domains target-first, switch gateway routing, then drain the source. +- Deliver artifacts only to participating and migration-target cells. + +**Scale target:** at least 20,000 domains across several cells and a controlled +burst of 10,000 placement-affecting changes without unnecessary reshuffling. + +**Completion checklist:** + +- [ ] One shared pool uses at least three cells on one edge. +- [ ] Placement remains stable across unrelated changes. +- [ ] Reserved, dedicated, and quarantine rules are enforced. +- [ ] Failed target readiness keeps the source active. +- [ ] Successful movement removes old state only after the target is serving. +- [ ] Non-participating cells receive no artifacts. +- [ ] Pool readiness evaluates all required cells. +- [ ] Tests, scale, documentation, and manual qualification pass. + +# Phase 4 — Pool service endpoints and Geo-Unicast + +**Goal:** support several public service IP pairs on one edge, with each pair +owned by one pool endpoint backed by one or more cells. + +**Implementation:** + +- Add per-edge pool endpoints with IPv4, optional IPv6, gateway state, + participating cells, and readiness. +- Allow separate service pairs for shared, reserved, dedicated, and quarantine pools. +- Keep management addresses separate from service endpoints. +- Publish only ready, non-withdrawn endpoints through system-managed DNS. +- Preserve country, continent, and global Geo-DNS fallback behavior. +- Prevent duplicate or conflicting address ownership. +- Reconcile gateway bindings, pool readiness, and DNS publication through one + revisioned workflow. + +**Scale target:** several pools and service pairs on each of at least two edges, +with endpoint-health changes measured without rewriting every domain. + +**Completion checklist:** + +- [ ] Three shared cells serve one dual-stack pair. +- [ ] A reserved pool serves a different pair on the same edge. +- [ ] DNS publishes the correct ready endpoints. +- [ ] Withdrawal affects only the intended pool. +- [ ] IPv4-only, IPv6-only, and dual-stack endpoints pass. +- [ ] Conflicts fail before activation. +- [ ] Restart and reconciliation converge gateway, DNS, pool, and cell state. +- [ ] Tests, scale, documentation, and manual qualification pass. + +# Phase 5 — Simple Anycast pools + +**Goal:** allow selected pools to use one shared IPv4/IPv6 pair across several +POPs while route advertisement remains owned by the network operator/provider. + +**Implementation:** + +- Add `simple_anycast` routing mode beside Geo-Unicast. +- Store one pool-level IPv4 and optional IPv6 pair. +- Attach explicit POPs/edges and readiness requirements. +- Bind the same pair on participating gateways. +- Publish the shared pair for assigned domains. +- Expose clear ready, degraded, and withdrawn signals. +- Do not add FRR, BIRD, router credentials, arbitrary commands, or BGP control. + +**Scale target:** at least two approved POPs using the same dual-stack service +pair, tested from multiple external vantage points with controlled POP loss. + +**Completion checklist:** + +- [ ] One Anycast pool serves from multiple POPs. +- [ ] Geo-Unicast and Anycast coexist on the same fleet. +- [ ] POP failure does not corrupt another POP's local state. +- [ ] External route withdrawal and restoration are recorded. +- [ ] UI and docs clearly state that CDNFoundry does not announce BGP routes. +- [ ] Uplink and upstream-scrubbing limitations remain explicit. +- [ ] Tests, network evidence, documentation, and manual qualification pass. + +# Phase 6 — Cache v2 + +**Goal:** turn the baseline cache into a persistent, bounded, production-strength +cell cache without a distributed cache or per-domain directories. + +**Implementation:** + +- Use persistent per-cell cache volumes with explicit size, inactive time, + temporary quota, and minimum free space. +- Add small, standard, large, and streaming pool profiles. +- Keep stable domain-to-cell placement. +- Add query policies: include all, ignore all, include selected, ignore selected. +- Keep deterministic scheme, Host, path, query, and epoch behavior. +- Add bounded TTL policies for approved status codes. +- Add admission, object-size, low-disk, range, and variant protections. +- Add stale-if-error, stale-while-revalidate, cache-only, and stale-only modes. +- Preserve exact URL purge and epoch full purge. + +**Scale target:** mixed HIT/MISS traffic, quota pressure, restart persistence, +purge fan-out, and high-cardinality abuse with throughput, latency, IOPS, CPU, +memory, disk, and hit ratio recorded. + +**Completion checklist:** + +- [ ] Cache survives routine restart and remains rebuildable after loss. +- [ ] Profiles enforce disk, temporary, object, and admission ceilings. +- [ ] Query policies do not create unbounded variants. +- [ ] TTL and stale behavior match policy. +- [ ] Low disk and abuse bypass safely. +- [ ] Purge remains durable across participating cells. +- [ ] One domain cannot exhaust unrelated cache resources beyond accepted limits. +- [ ] Tests, load evidence, documentation, and manual qualification pass. + +# Phase 7 — Gzip and Brotli compression + +**Goal:** reduce delivered bandwidth through safe compression integrated with +Cache v2 and bounded by pool and cell resources. + +**Implementation:** + +- Store one canonical uncompressed cache object by default. +- Request identity encoding from origins where required. +- Enable Gzip as the normal default. +- Add optional Brotli through one immutable, pinned, tested edge image/module. +- Expose only off, standard, and maximum-savings profiles. +- Use a tested MIME allowlist and minimum response size. +- Avoid recompressing images, video, archives, and other compressed formats. +- Handle Accept-Encoding, Vary, ETag, HEAD, 304, stale, purge, and fallback. +- Bound range traffic, large responses, concurrency, and CPU usage. +- Add emergency disable and CPU-pressure fallback. +- Record encoding, bytes, ratio, profile, and fallback telemetry. + +**Scale target:** mixed identity, Gzip, and Brotli clients against HIT and MISS +traffic, with bandwidth saved, throughput, latency, and CPU cost recorded. + +**Completion checklist:** + +- [ ] Identity, Gzip, and Brotli decode to identical content. +- [ ] One canonical object serves different encodings correctly. +- [ ] Compressed and range content follows safe policy. +- [ ] Vary, ETag, revalidation, stale, and purge remain correct. +- [ ] Shared pools cannot select unsafe levels. +- [ ] CPU pressure falls back without stopping traffic. +- [ ] Compression analytics are accurate. +- [ ] Tests, load evidence, documentation, and manual qualification pass. + +# Phase 8 — Primary and backup origin failover + +**Goal:** add one simple active-passive backup origin per proxied hostname. + +**Implementation:** + +- Support one primary and one optional backup origin. +- Reuse strict origin, TLS, timeout, header, and loop validation. +- Add bounded health checks and request-path failure evidence. +- Add failure threshold, recovery threshold, hold-down, and failback delay. +- Keep active-origin state local to cells during control-plane loss. +- Fail over without calling Laravel in the request path. +- Prefer stale or cache-only behavior before retry storms. +- Expose active origin and transition reason without secrets. +- Do not add weighted, percentage, geographic, or arbitrary origin pools. + +**Scale target:** controlled failover and recovery under concurrent HIT/MISS +traffic, with transition time, origin pressure, errors, and isolation recorded. + +**Completion checklist:** + +- [ ] Healthy primary receives normal traffic. +- [ ] Qualified failure moves traffic to backup. +- [ ] Recovery does not flap. +- [ ] Both-origin failure follows stale or maintenance policy. +- [ ] Invalid backup state never replaces valid primary state. +- [ ] Control-plane outage does not remove local failover. +- [ ] One failing origin cannot exhaust unrelated origin budgets. +- [ ] Tests, load evidence, documentation, and manual qualification pass. + +# Phase 9 — Managed OWASP CRS WAF + +**Goal:** add optional managed application-signature protection without exposing +a custom WAF language or raw ModSecurity configuration. + +**Implementation:** + +- Pin ModSecurity v3 and OWASP Core Rule Set releases. +- Build an immutable WAF-capable OpenResty cell image/profile. +- Support off, monitor, balanced, and strict profiles. +- Map profiles to tested thresholds, paranoia levels, body limits, and blocking. +- Prefer reserved, dedicated, or quarantine WAF-capable cells where isolation is needed. +- Allow only bounded exclusions by approved dimensions, reason, owner, and expiry. +- Reject arbitrary SecRule directives, customer rule uploads, runtime downloads, + and custom expression languages. +- Add rule, score, action, processing-time, body-limit, and exclusion telemetry. +- Roll new CRS/image versions through monitor-only canaries. +- Preserve the previous image and ruleset after failure. + +**Scale target:** safe attack and false-positive corpora plus HIT/MISS load, with +latency, CPU, memory, throughput, detection, false positives, and accepted limits recorded. + +**Completion checklist:** + +- [ ] Off, monitor, balanced, and strict behave as documented. +- [ ] Monitor detects without blocking. +- [ ] Blocking uses stable, privacy-safe reasons. +- [ ] Exclusions are bounded, audited, and expiring. +- [ ] Oversized or malformed bodies remain bounded. +- [ ] Failed canaries keep the previous valid WAF runtime. +- [ ] Non-WAF pools remain healthy during WAF load or failure. +- [ ] Tests, security evidence, documentation, and manual qualification pass. + +# Phase 10 — Observability and capacity control + +**Goal:** make gateway, endpoint, pool, cell, cache, compression, failover, +Anycast, and WAF behavior operationally visible and capacity-manageable. + +**Implementation:** + +- Add bounded gateway, endpoint, pool, placement, cell, cache, compression, + origin, and WAF telemetry. +- Extend ClickHouse schemas and aggregates with bounded retention and queries. +- Add useful administrator and domain analytics without leaking unrelated data. +- Add alerts for stale maps, endpoint mismatch, cell exhaustion, cache pressure, + compression pressure, origin failover, WAF errors, and Anycast disagreement. +- Keep telemetry best-effort and outside serving decisions. + +**Scale target:** at least 20,000 active proxied domains across several pools, +cells, endpoints, and edges, including ClickHouse/Vector outage and recovery. + +**Completion checklist:** + +- [ ] Every new component has healthy, degraded, and unavailable states. +- [ ] Metrics identify pool, cell, edge, and revision. +- [ ] Domain users cannot see unrelated data. +- [ ] Raw logs remain bounded and redacted. +- [ ] Telemetry outage never blocks serving. +- [ ] Queries remain bounded at the qualification dataset. +- [ ] Alerts link to actionable runbooks. +- [ ] Tests, scale evidence, documentation, and manual qualification pass. + +# Phase 11 — Bounded fleet rollout automation + +**Goal:** automate proven edge upgrades without introducing general remote +execution or dynamic containers. + +**Implementation:** + +- Manage immutable gateway, agent, normal-cell, and WAF-cell versions. +- Define compatibility ranges and a bounded mixed-version window. +- Support canary edges/POPs and rollout waves. +- Pause on health, error, readiness, revision, or capacity thresholds. +- Roll back to the last compatible image and configuration. +- Preserve the fixed slot topology. +- Expose desired/current version, wave, progress, pause, failure, and rollback. +- Audit every rollout decision. + +**Scale target:** multi-edge, multi-POP rollout with mixed normal and WAF cells, +including failed canary, automatic pause, and rollback. + +**Completion checklist:** + +- [ ] Canary completes before later waves. +- [ ] Failed canary pauses automatically. +- [ ] Rollback restores the previous compatible runtime. +- [ ] Traffic continues during the mixed-version window. +- [ ] No arbitrary command execution or dynamic unbounded containers exist. +- [ ] Version drift and incompatibility are visible. +- [ ] Tests, recovery evidence, documentation, and manual qualification pass. + +# Phase 12 — Final production qualification + +**Goal:** prove the complete post-baseline architecture as one deployable, +recoverable, simple, and solid product. + +**Required topology:** + +- At least two POPs/edges. +- At least eight bounded cell slots per edge. +- One shared pool using at least three cells per edge. +- One reserved pool using a separate IPv4/IPv6 pair. +- One quarantine pool. +- Geo-Unicast endpoints. +- One Simple Anycast pool where an approved routing environment exists. +- Persistent Cache v2, Gzip, Brotli, backup origin, and managed WAF. +- Real IPv4 and IPv6 traffic where available. + +**Final checklist:** + +- [ ] Clean edge installation and registration pass. +- [ ] All pool kinds and endpoint modes pass. +- [ ] Multiple service pairs work on one edge. +- [ ] Placement, movement, drain, quarantine, and rollback pass. +- [ ] Geo-Unicast and Anycast external checks pass. +- [ ] Cache persistence, purge, stale, pressure, Gzip, and Brotli pass. +- [ ] Origin failover and failback pass. +- [ ] WAF monitor, block, exclusion, canary, and rollback pass. +- [ ] Existing traffic continues through controlled control-plane and telemetry outages. +- [ ] Invalid gateway, cell, and WAF candidates preserve previous valid state. +- [ ] One saturated cell does not stop unrelated pools. +- [ ] Fleet canary upgrade and rollback pass. +- [ ] Clean-host control-plane restore and derived-state reconciliation pass. +- [ ] All affected API, UI, architecture, deployment, operations, security, + troubleshooting, and runbook documentation is current. +- [ ] Every completed test is linked and every unexecuted test is marked clearly. +- [ ] Owner-run browser and real-traffic qualification is recorded. +- [ ] Release notes state measured capabilities and limitations honestly. + +# Future candidates + +The following remain outside the committed roadmap until real demand justifies a +separate bounded implementation and qualification contract: + +- secondary ACME authority; +- DNSSEC; +- HTTP/3 and QUIC; +- private origin connector; +- immutable backup storage and warm control-plane standby; +- long-retention analytics archive; +- replicated placement for exceptional high-volume domains; +- additional bounded placement policies; +- origin shield or hierarchical cache. + +# Explicitly out of scope + +- weighted or percentage origin balancing; +- general BGP/router management; +- volumetric DDoS scrubbing guarantees; +- CAPTCHA, browser challenge, or bot-scoring platforms; +- customer-written WAF rules or edge scripts; +- serverless workers or plugin marketplaces; +- object-storage product features; +- per-domain containers or processes by default; +- Kubernetes as a deployment requirement; +- billing, reseller, organization, team, or custom-role systems. + +## Final rule + +Keep the completed platform intact. Add one bounded production capability per +phase. Keep the gateway simple, keep cells isolated, preserve last-valid state, +test real traffic and failure behavior, record scale honestly, update every +affected document, and never move DNS or HTTP traffic through Laravel.