From 24c14efe478c0e672ae608e19a3d95548e80512b Mon Sep 17 00:00:00 2001 From: vaheeD khoshnouD Date: Mon, 27 Jul 2026 13:08:37 +0330 Subject: [PATCH 1/7] docs: rebuild product roadmap --- docs/roadmap.md | 717 ++++++++++++++++++++++++++++++++++++------------ 1 file changed, 541 insertions(+), 176 deletions(-) diff --git a/docs/roadmap.md b/docs/roadmap.md index 447dee6..2320e26 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -1,226 +1,591 @@ --- -title: Roadmap and implementation status -description: CDNFoundry product contract, current implementation boundary, completion gates, and future admission rules. +title: CDNFoundry product roadmap +description: Ordered implementation and qualification contract for a simple, solid, production-grade private CDN. --- -# Roadmap and implementation status +# CDNFoundry product roadmap -This page is the product contract referenced by `AGENTS.md`. The original -long-form roadmap, appendices, and dated evidence are preserved at -`docs/legacy/roadmap.md`; they are historical, not current documentation. +## 1. Product goal -## Product contract +CDNFoundry is a small, production-grade private CDN and authoritative DNS platform for operators that own or manage their infrastructure. -CDNFoundry is a small production-grade private CDN with predictable failure -behaviour, bounded resource use, and low operational complexity. +The product favors: -- One Laravel modular monolith and two Filament panels own management. -- PostgreSQL owns desired state. -- DNS, HTTP, security decisions, certificate selection, and raw telemetry never - pass through Laravel. -- External effects are asynchronous, revisioned, idempotent, coalesced, and - last-valid-state preserving. -- Scale comes from workers, DNS capacity, ClickHouse capacity, edges, and - bounded OpenResty cells. -- A domain does not receive a default process, container, worker, timer, Nginx - server block, cache directory, or reload. +- low feature count; +- clear operations; +- bounded resource usage; +- deterministic desired state; +- failure isolation; +- last-valid-state recovery; +- real production qualification. -Part One is implemented through the code surfaces described below. A phase is -release complete only when implementation, current documentation, -agent-owned automated/runtime qualification, and owner-run manual browser -qualification are each recorded. +The product does not attempt to reproduce Cloudflare, Fastly, Akamai, or a general cloud platform. -## Phase 1: foundation, access, and system identity +A phase is complete only when its implementation, documentation, automated tests, real-runtime qualification, scale checkpoint, and owner-run browser qualification are all recorded. -Implemented behaviour: +## 2. Non-negotiable boundaries -- administrator and assigned-domain-user panels; -- Sanctum login, profile, password, token, disable/enable, audit, and idempotency; -- policy-aware domain and operation binding; -- typed system DNS identity preview, exact confirmation, asynchronous apply; -- health, readiness, operation, Horizon, scheduler, and Compose foundations. +1. One Laravel modular monolith and two Filament panels form the control plane. +2. PostgreSQL is the durable source of desired state. +3. DNS, HTTP, HTTPS, TLS selection, cache decisions, security decisions, and raw telemetry never pass through Laravel. +4. PowerDNS runtime state, edge artifacts, generated gateway maps, and analytics aggregates are derived and rebuildable. +5. External effects are asynchronous, revisioned, idempotent, coalesced, acknowledged, and last-valid-state preserving. +6. DNSdist is the only public authoritative DNS endpoint. +7. Edge traffic uses a small gateway and bounded identical OpenResty cells. +8. Cell containers are created during edge installation. The edge agent assigns and configures cells; it does not receive unrestricted Docker access. +9. No domain receives a default process, container, worker, timer, cache directory, Nginx server block, or reload. +10. Per-domain, per-pool, per-cell, queue, log, cache, import, purge, rule, artifact, and query limits are explicit. +11. DDoS readiness reduces application and noisy-neighbour blast radius but does not claim upstream volumetric scrubbing after physical capacity is saturated. +12. No microservices, Kafka, Kubernetes requirement, CQRS, event sourcing, GraphQL, custom RBAC, plugin runtime, reseller hierarchy, billing engine, custom expression language, or additional dashboard is introduced without an explicit product-contract change. +13. Production code and filenames never use roadmap phase numbers or lifecycle suffixes such as `V2`, `New`, or `Final`. +14. Development PostgreSQL and named Compose volumes remain persistent across all work. Destructive refreshes are forbidden. -### Phase 1 completion gate +## 3. Required completion gate for every phase -| Gate | Status | +Every phase must record the following independently: + +| Gate | Required evidence | | --- | --- | -| Implementation | Present in current code | -| Documentation | Current in this site | -| Automated/runtime qualification | Covered by current PHP and `tests/e2e/e2e.py`; rerun for each release | -| Manual browser qualification | Not recorded for the current revision | +| Implementation | Durable state, API, policies, UI where required, jobs/reconciliation, metrics, audit, and rollback behavior | +| Unit and feature tests | Happy path, permissions, validation, idempotency, bounds, and stable errors | +| Runtime and E2E tests | Real DNS, HTTP, HTTPS, TLS, cache, security, telemetry, restart, and failure behavior where applicable | +| Scale checkpoint | Dataset, concurrency, hardware/topology, measured result, saturation point, and accepted limit | +| Failure and recovery | Retry, obsolete work, last-valid state, dependency outage, restart, rollback, and reconciliation | +| Documentation | User, administrator, API, architecture, operations, metrics, troubleshooting, and runbook updates | +| Browser qualification | Owner-run steps and evidence in `docs/manual-browser-qualification.md` | +| Release decision | Passed, failed, blocked, or explicitly removed from scope | -## Phase 2: domains and authoritative DNS +A phase cannot be marked complete when a required test was not executed. -Implemented behaviour: +## 4. Qualification rules -- normalized domain lifecycle, assignment, delegation verification, activation, - delayed deprovisioning, tombstones, and reclaim cooldown; -- A, AAAA, CNAME, MX, TXT, NS, CAA, SRV, and reverse-zone PTR; -- transactional bulk changes, BIND import/export, monotonic serials; -- DNS cluster management, health tests, deterministic reconciliation, drift and - last-valid-zone preservation; -- DNSdist-only public ingress and separate PowerDNS runtime migrations. +- Tests use the smallest realistic topology that proves the behavior, then a separate scale program proves the declared limits. +- Runtime behavior is not accepted from mocks alone. +- IPv4 and IPv6 are tested whenever the feature handles addresses or traffic. +- Invalid candidates never replace active DNS, gateway, cell, TLS, cache, WAF, or routing state. +- Rapid updates prove coalescing and obsolete-revision exit. +- A failure in one domain, pool, cell, edge, DNS cluster, telemetry component, or queue lane must not unnecessarily affect unrelated traffic. +- Every phase updates OpenAPI, route coverage, current documentation, examples, environment references, metrics, alerts, and runbooks when affected. +- Browser qualification is manual and owner-run. Coding agents maintain the checklist but report it as not run unless the owner records evidence. +- Non-browser E2E and real-runtime qualification is agent-owned under `tests/e2e`. -### Phase 2 completion gate +# Part One — Simple but solid production platform -| Gate | Status | -| --- | --- | -| Implementation | Present in current code | -| Documentation | Current in domain, DNS, deployment, and reference guides | -| Automated/runtime qualification | Covered by PHP, `phase2_dns.py`, and separate `phase2_scale.py`; rerun for each release | -| Manual browser qualification | Not recorded for the current revision | +Part One is the ordered production roadmap. Phases 1 through 8 preserve and requalify the current platform foundation. Phases 9 through 14 implement the agreed edge, Anycast, cache, origin, and managed WAF improvements. -## Phase 3: Geo-DNS +## Phase 1 — Foundation, access, and system identity -Implemented behaviour: +### Outcome -- country, continent, then default answer selection; -- bounded targets and geography vocabulary; -- IPv4/IPv6 preview and unknown fallback; -- deterministic PowerDNS Lua compilation; -- validated, atomically updated local MMDB with last-valid preservation. +A recoverable Laravel/Filament control plane with exactly two user types, typed system identity, audit, idempotency, queues, scheduler, health, and safe Compose foundations. -CAA remains DNS-only because it is not in the qualified Geo-DNS runtime type -list. +### Scope -### Phase 3 completion gate +- Administrator and assigned-domain-user access. +- Sanctum browser sessions and API tokens. +- Policy-aware route binding. +- User lifecycle, profile, password, token, and audit behavior. +- Typed platform DNS identity preview, exact confirmation, asynchronous apply, and rollback. +- Horizon queue lanes, scheduler freshness, health, readiness, and protected metrics. +- Development and production Compose definitions with explicit networks, volumes, limits, health checks, and migrations. -| Gate | Status | -| --- | --- | -| Implementation | Present in current code | -| Documentation | Current in Geo-DNS and DNS references | -| Automated/runtime qualification | Covered by PHP and `phase3_geo_dns.py`; external vantage points remain owner evidence | -| Manual browser qualification | Not recorded for the current revision | +### Completion checklist -## Phase 4: proxy, edge routing, and edge agent +- [ ] Administrator and domain-user boundaries pass API and browser tests. +- [ ] Idempotency conflict and replay behavior pass. +- [ ] Secrets appear only at their allowed one-time boundary. +- [ ] System identity preview and exact-confirmation flow pass. +- [ ] Queue and scheduler outage states are visible and bounded. +- [ ] SQLite-isolated Laravel tests fail closed when the safe test database is not active. +- [ ] Compose, OpenAPI, formatting, static, documentation, and link checks pass. +- [ ] Clean startup, graceful shutdown, restart, and last-valid settings tests pass. +- [ ] Manual browser Phase 1 is recorded. -Implemented behaviour: +## Phase 2 — Domains and authoritative DNS -- safe explicit origin per proxied hostname; -- shared, quarantine, and exceptional dedicated pool model; -- edge/cell desired state, addresses, enrollment, mutual TLS, identity rotation; -- signed incremental artifacts and bounded full recovery snapshots; -- atomic agent activation, persistent acknowledgements, heartbeat/capacity; -- target-first placement and source drain; -- data-driven OpenResty request path with no per-domain reload. +### Outcome -### Phase 4 completion gate +A complete authoritative DNS workflow with durable desired state, deterministic deployment, drift detection, and real DNS qualification. -| Gate | Status | -| --- | --- | -| Implementation | Present in Laravel, Go, Nginx, and Lua | -| Documentation | Current in edge, proxy, architecture, API, and deployment guides | -| Automated/runtime qualification | Covered by PHP, Go, `phase4_control_plane.py`, `phase4_mtls.py`, and `phase4_runtime.py`; rerun for each release | -| Manual browser qualification | Not recorded for the current revision | +### Scope -## Phase 5: TLS, cache, and purge +- Domain lifecycle, assignment, delegation verification, activation, delayed deprovisioning, tombstones, and reclaim safety. +- A, AAAA, CNAME, MX, TXT, NS, CAA, SRV, and reverse-zone PTR. +- Transactional bulk changes and bounded BIND import/export. +- Monotonic serials and deterministic zone rendering. +- Multiple DNS clusters with target acknowledgements. +- DNSdist-only public ingress and private PowerDNS/runtime databases. +- Zone rebuild from control-plane PostgreSQL. -Implemented behaviour: +### Scale checkpoint -- managed ACME DNS-01 only for eligible proxied domains; -- bounded name sets, renewal, retries, cleanup, alerts, and certificate reuse; -- validated encrypted custom certificate upload; -- deterministic cache settings, bypass/admission, development mode, stale grace; -- epoch full purge, exact URL purge, durable per-edge delivery and retry; -- revision rollback without decrementing revision numbers. +- At least 500,000 domains. +- At least 1,000,000 DNS records. +- At least 50,000 DNS changes per day. +- A controlled burst of at least 10,000 mutations. +- Multiple DNS clusters and mixed IPv4/IPv6 data. -### Phase 5 completion gate +### Completion checklist -| Gate | Status | -| --- | --- | -| Implementation | Present in current code | -| Documentation | Current in TLS, cache, API, operations, and troubleshooting | -| Automated/runtime qualification | Covered by PHP, `phase5_tls.py`, and cumulative edge/runtime programs; rerun for each release | -| Manual browser/public HTTPS qualification | Not recorded for the current revision | +- [ ] Record validation, CNAME coexistence, zone boundaries, Punycode, duplicates, and TTL limits pass. +- [ ] Bulk operations are bounded, atomic, idempotent, and revisioned. +- [ ] One failed DNS target does not replace another target's valid state. +- [ ] PowerDNS runtime deletion and complete rebuild pass. +- [ ] Real UDP/TCP `dig` tests pass over IPv4 and IPv6. +- [ ] Scale and mutation-coalescing programs pass with measured evidence. +- [ ] DNS user/admin/API/operations documentation and runbooks are current. +- [ ] Manual browser Phase 2 is recorded. -## Phase 6: security and DDoS readiness +## Phase 3 — Geo-DNS -Implemented behaviour: +### Outcome -- ordered bounded allow/block rules and imports; -- standard, protected, quarantine, and bounded manual profiles; -- IPv4/IPv6 trusted client and rule handling; -- readiness states, bounded reason codes, target-first quarantine and recovery; -- persisted expiring emergency actions for domains, cells, edges, and pools; -- cell resource isolation and explicit volumetric-limit statement. +Bounded country, continent, and default DNS answers without runtime dependency on Laravel or external GeoIP services. -### Phase 6 completion gate +### Scope -| Gate | Status | -| --- | --- | -| Implementation | Present in control and edge runtimes | -| Documentation | Current in security, hardening, limits, and runbooks | -| Automated/runtime qualification | Covered by PHP and `phase6_security.py`; rerun for each release | -| Manual browser/real-traffic qualification | Not recorded for the current revision | +- Country, continent, then default selection. +- A and AAAA Geo-DNS records. +- Valid ECS use when present and trusted. +- Resolver-address fallback with honest accuracy documentation. +- Shared local MMDB with atomic update and last-valid preservation. +- Deterministic PowerDNS Lua compilation. -## Phase 7: logs, analytics, and usage export +### Completion checklist -Implemented behaviour: +- [ ] Country, continent, default, and unknown behavior pass. +- [ ] IPv4 and IPv6 previews and real queries pass. +- [ ] Duplicate, missing-default, invalid-type, and excessive-target inputs fail safely. +- [ ] MMDB corruption/provider outage retains the previous valid database. +- [ ] Geo changes do not rewrite unrelated zones. +- [ ] External-vantage evidence records resolver/ECS limitations. +- [ ] Documentation and browser qualification are current. -- direct Vector-to-ClickHouse edge and DNS telemetry; -- bounded/redacted schemas and disk buffers; -- raw logs, aggregates, outage/partial labels, IPv4/IPv6 masking; -- domain/admin analytics separation; -- idempotent hourly PostgreSQL usage rollups and stable JSON/CSV export; -- telemetry failure independent from serving. +## Phase 4 — Proxy, baseline edge pools, and edge agent -### Phase 7 completion gate +### Outcome -| Gate | Status | -| --- | --- | -| Implementation | Present in current code and telemetry configuration | -| Documentation | Current in analytics, schema, monitoring, and troubleshooting | -| Automated/runtime qualification | Covered by PHP and `phase7_analytics.py`; rerun for each release | -| Manual browser qualification | Not recorded for the current revision | +Safe reverse proxying through signed, data-driven OpenResty runtime state with stable baseline pool placement. -## Phase 8: operations and production qualification +### Scope -Implemented behaviour: +- One explicit origin per proxied hostname. +- Origin safety, forwarding-header normalization, timeouts, retries, WebSocket behavior, and health tests. +- Shared, quarantine, and exceptional dedicated pools. +- Edge registration, one-time bootstrap, mTLS identity, revocation, and rotation. +- Signed deltas and bounded full snapshots. +- Atomic activation, acknowledgement, previous-valid retention, and offline serving. +- Target-first placement and source drain. +- Generic OpenResty configuration without per-domain reload. -- component/queue/capacity health, token-protected metrics, alerts; -- failed-job and operation management, bounded global reconciliation; -- encrypted Restic backup, authenticated restore preflight, maintenance executor; -- explicit migration, production image, split-host, IPv6, and external-data contracts; -- upgrade, recovery, throughput, MMDB, restart, and isolation programs. +### Completion checklist -### Phase 8 completion gate +- [ ] Unsafe origins and proxy loops are rejected at save and connect time. +- [ ] Agent enrollment, rotation, revocation, and replay limits pass. +- [ ] Corrupt, unsigned, oversized, incompatible, and obsolete artifacts are rejected. +- [ ] A fresh edge restores from a bounded full snapshot. +- [ ] Control-plane, queue, and network interruption do not stop existing traffic. +- [ ] One cell/domain failure does not stop an unrelated cell/domain. +- [ ] Real HTTP/HTTPS, IPv4/IPv6, restart, drain, migration, and rollback tests pass. +- [ ] Documentation and browser qualification are current. -| Gate | Status | -| --- | --- | -| Implementation | Present in current code and infrastructure | -| Documentation | Current in deployment, operations, security, development, and upgrade guides | -| Automated/runtime qualification | Covered by PHP and phase 8 Python programs; full external environment evidence remains operator-owned | -| Manual browser and production release qualification | Not recorded for the current revision | - -## Current release boundary - -Because the manual browser and external production evidence is not recorded for -the current revision, Part One is implementation-complete but not represented -here as owner release-qualified. Historical evidence remains in -`docs/legacy/phase-*-qualification.md`. - -## Future admission - -Potential future work includes HTTP/3, alternative certificate workflows, -private origin connectivity, larger fleet automation, extended disaster -recovery, and proven customer extensions. None is implemented or promised by -the current product. - -A future capability is admitted only when: - -- a real operator/user problem and measurable acceptance criteria exist; -- it preserves the product invariants and request-path independence; -- it has typed bounded state, authorization, failure behaviour, rollback, - observability, real-runtime tests, documentation, and manual qualification; -- disabling it leaves Part One functional; -- it does not introduce speculative microservices, Kubernetes requirements, - plugins, billing hierarchies, custom expression languages, or per-domain - runtime processes. - -## Final architecture rule - -Keep management in one understandable Laravel application. Keep DNS and HTTP -traffic independent. Store desired behaviour as validated bounded data. Use the -same reconciliation discipline for every external change. Preserve the last -valid state, isolate edge resources by bounded cells, and never claim -volumetric protection after physical capacity is saturated. +## Phase 5 — TLS, baseline cache, and purge + +### Outcome + +Recoverable managed/custom TLS and deterministic basic caching with durable bounded purge delivery. + +### Scope + +- Managed ACME DNS-01 only for eligible proxied domains. +- Renewal spreading, retries, cleanup, certificate reuse, and expiry alerts. +- Validated encrypted custom certificate upload. +- Cache enablement, TTLs, object limits, origin-header behavior, query inclusion, cookie bypass, development mode, and stale-if-error. +- Epoch-based full purge and exact URL purge. +- Durable per-edge purge tasks with retry and status. + +### Completion checklist + +- [ ] Managed and custom TLS lifecycle passes with no private-key exposure. +- [ ] Wrong key, chain, name, expiry, and oversized PEM are rejected. +- [ ] Existing valid certificates continue during ACME failure. +- [ ] Cache MISS/HIT/BYPASS/STALE behavior matches the deterministic key. +- [ ] URL and full purge reach every eligible target and retry safely. +- [ ] Restart and failed-delivery tests preserve serving and task state. +- [ ] Public HTTPS, documentation, and browser qualification are current. + +## Phase 6 — Security and DDoS readiness + +### Outcome + +Bounded early rejection, application protection, quarantine, and emergency controls without pretending to provide upstream scrubbing. + +### Scope + +- Ordered IP, CIDR, country, and continent allow/block rules. +- Standard, protected, quarantine, and bounded manual profiles. +- Client/domain request, connection, TLS, body, header, timeout, origin, and cache-admission ceilings. +- Trusted proxy handling. +- Restrict, quarantine, recover, and release states. +- Expiring emergency actions for domains, cells, edges, and pools. +- Pool service-address withdrawal from DNS. +- Security telemetry with stable reason codes. + +### Completion checklist + +- [ ] Rule order, permissions, imports, limits, IPv4, and IPv6 pass. +- [ ] Unknown Host/SNI and malformed traffic are rejected before expensive work. +- [ ] Quarantine migration is target-first and recoverable. +- [ ] Emergency actions persist across restart and expire safely. +- [ ] One attacked domain cannot exhaust unrelated cell resource budgets in qualification. +- [ ] Physical-uplink saturation limitations remain explicit. +- [ ] Security docs, runbooks, metrics, alerts, and browser qualification are current. + +## Phase 7 — Telemetry, analytics, and usage export + +### Outcome + +Direct bounded telemetry and accurate operational analytics independent from serving. + +### Scope + +- OpenResty and DNS structured telemetry through Vector directly to ClickHouse. +- Bounded disk buffers, retries, drops, retention, and redaction. +- Request, DNS, cache, origin, TLS, security, deployment, edge, cell, and pool fields. +- Domain-scoped and administrator-global analytics. +- Bounded raw-log access. +- Idempotent PostgreSQL usage rollups and stable JSON/CSV exports. +- Partial/outage labeling. + +### Scale checkpoint + +- At least 20,000 domains with active analytics. +- Bounded query ranges, filters, result sizes, and execution limits. +- Controlled ClickHouse outage and backlog recovery. + +### Completion checklist + +- [ ] Raw traffic never passes through Laravel queues or PostgreSQL. +- [ ] Secrets, request bodies, tokens, cookies, and certificate material are absent. +- [ ] Generated traffic matches request, byte, cache, origin, security, and DNS totals. +- [ ] ClickHouse/Vector outage does not affect serving. +- [ ] Backlog recovery does not starve live traffic. +- [ ] Usage rebuild and export are stable and idempotent. +- [ ] Documentation and browser qualification are current. + +## Phase 8 — Operations, recovery, and baseline release qualification + +### Outcome + +A deployable, observable, recoverable baseline platform with measured release evidence. + +### Scope + +- Component health, queue health, drift, capacity, alerts, and operations. +- Bounded global reconciliation. +- Failed-job and failed-operation inspection/retry. +- Encrypted off-host backup and authenticated restore preflight. +- Clean-host restore. +- Expand/contract migrations, canary upgrade, and rollback. +- Clock synchronization and drift monitoring. +- Capacity planning and failure runbooks. + +### Completion checklist + +- [ ] Existing DNS and edge traffic continues during control-plane and ClickHouse outage. +- [ ] Encrypted backup and clean replacement-host restore pass. +- [ ] PowerDNS and edge state rebuild pass. +- [ ] Queue loss is repaired by reconciliation. +- [ ] Canary upgrade and rollback pass without database restore. +- [ ] RPO, RTO, topology, hardware, and measured throughput are recorded. +- [ ] Production installation, upgrade, backup, recovery, monitoring, and capacity documentation are current. +- [ ] Baseline release browser qualification is recorded. + +## Phase 9 — Edge gateway and bounded cell inventory + +### Outcome + +Each edge has one minimal public gateway and a bounded inventory of pre-created generic OpenResty cell slots. + +### Scope + +- One edge gateway container/process group. +- Fixed configurable cell-slot count created during edge installation. +- Stable cell identities independent from pool names. +- Gateway listeners on one or more service IPv4/IPv6 pairs. +- Host routing for HTTP and SNI preread routing for HTTPS. +- Trusted client-address preservation to cells. +- Unknown destination, Host, and SNI rejection. +- Signed/revisioned gateway maps with atomic activation and last-valid rollback. +- Agent discovery and reporting of all installed cell slots. +- No Docker socket in the normal edge agent. + +### Scale and failure checkpoint + +- Maximum declared service IP pairs, host mappings, cells, and connections per edge. +- Gateway restart and invalid-map rejection. +- One cell restart without gateway or unrelated-cell interruption. +- Gateway CPU/memory reservation under full cell load. + +### Completion checklist + +- [ ] Installation creates exactly the configured bounded slots. +- [ ] Agent registers inventory without dynamically creating containers. +- [ ] Gateway routes HTTP Host and TLS SNI to the assigned cell. +- [ ] Real client IP reaches the trusted cell runtime. +- [ ] Unknown or mismatched destination/Host/SNI is rejected. +- [ ] Invalid gateway state preserves the previous map. +- [ ] Gateway and cell health, saturation, revisions, and metrics are visible. +- [ ] Compose, API, UI, docs, runbooks, E2E, scale, and browser qualification are current. + +## Phase 10 — Multi-cell pools, endpoints, and stable domain placement + +### Outcome + +A pool can use multiple cells on each participating edge and one service IP pair can front those cells. + +### Scope + +- Pool kinds: shared, reserved, dedicated, and quarantine. +- Explicit participating edges. +- One or more cell slots per pool per edge. +- Pool service endpoints with IPv4/IPv6 ownership separate from individual cells. +- Stable domain-to-cell placement inside a pool. +- Optional explicitly bounded replicated placement for exceptional high-capacity domains. +- Different service IP pairs for different pools on the same edge. +- Target-first domain and cell migration. +- Selective artifact/certificate delivery only to participating cells and migration targets. +- Minimum-ready-cell policy per pool and edge. +- Slot drain, release, reassignment, and capacity reservation. + +### Scale checkpoint + +- Declared maximum pools, endpoints, cells, domains, and mappings per edge. +- Rebalancing does not reshuffle unrelated domains. +- Adding/removing a cell does not require full-fleet artifact delivery. +- Shared, reserved, dedicated, and quarantine isolation load tests. + +### Completion checklist + +- [ ] One IP pair fronts three shared cells successfully. +- [ ] A second IP pair fronts reserved customer cells on the same edge. +- [ ] Stable domain placement preserves cache locality. +- [ ] Dedicated pools enforce the single-domain contract. +- [ ] Reserved pools accept only explicitly assigned domains. +- [ ] Cell shortage and minimum shared/quarantine reservations fail safely. +- [ ] Only participating cells receive domain certificates and artifacts. +- [ ] Migration, drain, rollback, scale, docs, UI, API, and browser qualification pass. + +## Phase 11 — Geo-Unicast and Simple Anycast routing modes + +### Outcome + +Pools support either normal Geo-Unicast endpoints or externally routed Simple Anycast without introducing BGP control into Laravel. + +### Scope + +- `geo_unicast` and `simple_anycast` pool routing modes. +- Geo-Unicast publishes ready per-edge pool endpoints. +- Simple Anycast uses one shared IPv4 and one shared IPv6 across participating POPs. +- Anycast disables GeoDNS edge selection for the pool. +- CDNFoundry manages desired membership, configuration, TLS, purge, security, revision, and readiness. +- Provider/network tooling manages BGP, FRR, BIRD, route announcement, and withdrawal. +- Agent reports service-address presence, gateway readiness, cell readiness, active revision, and external route-advertised signal. +- A POP is ready only after required runtime state is active. +- A failed POP is excluded only after route withdrawal is confirmed. +- Shared Anycast service addresses are forbidden as origins. + +### Completion checklist + +- [ ] Geo-Unicast and Anycast modes cannot be ambiguously combined. +- [ ] Every Anycast POP activates the same required domain revision before ready. +- [ ] DNS returns the shared Anycast addresses without country/continent edge selection. +- [ ] Route withdrawal signal prevents unsafe POP exclusion ordering. +- [ ] One POP failure leaves healthy POPs serving. +- [ ] Control-plane outage does not stop externally announced valid POPs. +- [ ] BGP credentials and neighbor configuration are absent from CDNFoundry. +- [ ] External multi-POP, IPv4/IPv6, failure, scale, docs, and browser qualification pass. + +## Phase 12 — Cache v2 and response compression + +### Outcome + +A persistent, bounded, observable cache with correct variants, origin protection, Gzip, and optional Brotli. + +### Scope + +- Persistent per-cell cache storage with explicit maximum size, inactive period, minimum free disk, temporary-storage quota, and eviction visibility. +- Small, standard, large, and streaming pool cache resource profiles. +- Canonical uncompressed cached objects. +- Gzip enabled by default for approved compressible MIME types. +- Optional Brotli from an immutable tested edge image. +- Off, standard, and maximum-savings compression profiles. +- Minimum/maximum compressible response size. +- CPU/concurrency ceilings and emergency compression disable. +- No compression for already-compressed, range/video, or excluded content. +- Correct `Vary: Accept-Encoding`. +- Query policy: all, none, include-list, or ignore-list. +- Bounded cookie bypass and cache variants. +- Status-code TTL policy for explicitly supported responses. +- Cache lock/request collapsing, admission limits, stale-if-error, and stale-while-revalidate. +- Exact URL and full epoch purge preserved. +- Compression and bandwidth-savings telemetry. + +### Scale and failure checkpoint + +- Cache HIT load with identity, Gzip, and Brotli clients. +- CPU saturation and fallback behavior. +- Disk-full, minimum-free-space, restart, corrupt-cache, and eviction behavior. +- High-cardinality URL/query abuse. +- Large-object and range-request qualification. + +### Completion checklist + +- [ ] Identical decoded content is served for identity, Gzip, and Brotli. +- [ ] Cache key does not fragment unintentionally by `Accept-Encoding`. +- [ ] MIME, size, range, and precompressed exclusions work. +- [ ] CPU pressure remains bounded and can disable Brotli/compression safely. +- [ ] Query and cookie variants remain within configured limits. +- [ ] Disk pressure cannot exhaust the host or unrelated cells. +- [ ] MISS/HIT/STALE/revalidate/purge behavior remains correct with compression. +- [ ] Metrics report origin bytes, uncompressed bytes, served bytes, encoding, and savings. +- [ ] UI, API, docs, troubleshooting, load tests, and browser qualification pass. + +## Phase 13 — Simple origin resilience + +### Outcome + +A proxied hostname can use one primary and one backup origin with active-passive failover only. + +### Scope + +- Primary and optional backup origin. +- Same safety validation for both origins. +- Bounded active and passive health evidence. +- Explicit failover and recovery thresholds. +- Last-known-good origin state. +- No weighted traffic, percentage splitting, Geo origin steering, discovery, or arbitrary origin pools. +- Origin selection remains local to the cell and independent from Laravel availability. +- Failover/recovery telemetry and audit. + +### Completion checklist + +- [ ] Healthy primary receives normal traffic. +- [ ] Qualified primary failure moves traffic to backup. +- [ ] Flapping is bounded by thresholds and recovery delay. +- [ ] Backup failure cannot create an unbounded retry loop. +- [ ] Primary recovery is controlled and observable. +- [ ] Cache correctness is preserved across origin change. +- [ ] Both origins are protected against private/loop/platform destinations. +- [ ] Real failure, recovery, restart, scale, docs, and browser qualification pass. + +## Phase 14 — Managed OWASP CRS WAF + +### Outcome + +Optional managed application-signature protection using a pinned OWASP Core Rule Set without a customer rule language. + +### Scope + +- Immutable WAF-capable OpenResty cell image. +- Pinned tested ModSecurity v3 and OWASP CRS release. +- Profiles: off, monitor, balanced, and strict. +- Detection/anomaly scoring, bounded request-body inspection, and explicit response-body policy. +- WAF-capable pool/cell resource profile. +- Bounded exclusions by rule ID plus hostname, path prefix, or argument/header name. +- Expiry, administrator note, audit, and limit for exclusions. +- No raw ModSecurity directives, SecRule editor, online rule download, uploaded scripts, or plugin system. +- Monitor-first rollout, canary image/ruleset upgrade, automatic rollout pause, and last-compatible rollback. +- Rule, anomaly, action, latency, and false-positive telemetry. + +### Scale and failure checkpoint + +- Baseline and attack-pattern throughput comparison. +- CPU, memory, latency, body-size, multipart, and rule-count bounds. +- Monitor/block behavior and false-positive workflow. +- WAF crash or invalid ruleset cannot stop normal non-WAF pools. + +### Completion checklist + +- [ ] Monitor mode never blocks but records bounded sanitized events. +- [ ] Balanced and strict presets block qualified test cases. +- [ ] Exclusions are narrow, bounded, audited, and expire correctly. +- [ ] Invalid ruleset/image preserves the previous valid WAF runtime. +- [ ] WAF resource exhaustion is isolated to assigned WAF cells. +- [ ] New CRS rollout can pause and roll back safely. +- [ ] Security docs clearly separate WAF, DDoS readiness, and volumetric limits. +- [ ] API, UI, runbooks, E2E, load, and browser qualification pass. + +# Part Two — Bounded future roadmap + +Part Two begins only after Part One is running successfully and a real operator or customer requirement is documented. These phases are candidates, not promises. + +## Phase 15 — Fleet rollout automation and extended recovery + +### Candidate scope + +- Canary groups and rollout waves for gateway, agent, cell, and WAF images. +- Automatic pause on health, rejection, drift, or error thresholds. +- Automatic rollback to the last compatible image. +- Fleet compatibility reporting. +- Immutable/deletion-protected backup storage. +- Scheduled isolated restore exercises. +- Optional warm control-plane standby. + +### Boundary + +This automates already-proven manual procedures. It does not add a general orchestration platform, dynamic unbounded containers, Kubernetes requirement, or traffic dependency on the standby. + +### Admission and completion gate + +- [ ] Fleet size makes manual rollout measurably inefficient. +- [ ] Manual canary/rollback and clean restore already pass. +- [ ] Automation has bounded waves, pause, rollback, audit, and operator override. +- [ ] DNS and HTTP traffic remain independent when automation/standby is unavailable. +- [ ] Real multi-host qualification and documentation pass before release. + +## Phase 16 — Protocol, DNS, certificate, private-origin, and data extensions + +### Candidate capabilities + +Admit separately, never as one combined project: + +- HTTP/3 and QUIC. +- Secondary ACME certificate authority. +- DNSSEC signing, DS lifecycle, rollover, and recovery. +- Focused outbound private-origin connector. +- Longer-retention analytics archive/export. +- Additional proven placement policy such as region restriction or maintenance evacuation. + +### Explicitly outside product direction + +- Serverless workers. +- Object-storage product. +- General VPN or zero-trust suite. +- Arbitrary tunnels. +- Custom WAF or routing expression language. +- CAPTCHA or browser-challenge platform. +- Bot-scoring product. +- Billing/subscription engine. +- Reseller/organization hierarchy. +- Microservices, service mesh, Kafka, or Kubernetes requirement. + +### Admission gate for every capability + +- [ ] A real repeated requirement and measurable acceptance criteria exist. +- [ ] Part One cannot solve it safely. +- [ ] Typed bounded desired state and authorization are defined. +- [ ] Runtime remains independent from Laravel availability. +- [ ] Failure, disablement, rollback, compatibility, observability, and recovery are defined. +- [ ] Real-runtime, scale, documentation, and browser qualification are written before implementation is declared complete. + +# Final release contract + +CDNFoundry is release-qualified only when: + +- every admitted Part One phase has passed its completion gate; +- all current manual browser checkpoints are recorded; +- external DNS, IPv4/IPv6, public HTTPS, multi-edge, multi-cell, Anycast, cache/compression, origin-failure, WAF, outage, scale, backup, restore, upgrade, and rollback evidence is available where applicable; +- current documentation describes implemented behavior rather than plans; +- no unexecuted test is represented as passed; +- no project boundary has been weakened to complete a phase. From 8864871e9cac6245e5f607a8fa9ba626b5391580 Mon Sep 17 00:00:00 2001 From: vaheeD khoshnouD Date: Mon, 27 Jul 2026 13:10:02 +0330 Subject: [PATCH 2/7] docs: rebuild manual browser qualification --- docs/manual-browser-qualification.md | 661 ++++++++++++++++++--------- 1 file changed, 439 insertions(+), 222 deletions(-) diff --git a/docs/manual-browser-qualification.md b/docs/manual-browser-qualification.md index a1f3495..9d6ebad 100644 --- a/docs/manual-browser-qualification.md +++ b/docs/manual-browser-qualification.md @@ -1,276 +1,493 @@ --- title: Manual browser qualification -description: Owner-run browser checklist for every implemented CDNFoundry panel workflow. +description: Owner-run browser and operator checkpoints aligned with the CDNFoundry product roadmap. --- # Manual browser qualification -This is a manual, owner-run release job. Coding agents must not automate it. -Record date, operator, exact commit, topology, browser/version, desktop and -mobile viewport, operation IDs, domain revisions, certificate fingerprints, -screenshots, actual results, and failures. +## 1. Purpose and ownership -Use disposable application accounts and domains. Do not delete Compose volumes -or use customer data. +This is the owner-run browser qualification contract for CDNFoundry. -## Preparation +Coding agents: -1. Start the persistent development or approved production-like topology. +- maintain this file when implemented UI or operator behavior changes; +- do not launch or automate Chromium, Playwright, Selenium, Cypress, or another browser; +- report browser qualification as not run unless the owner supplies evidence. + +A roadmap phase is not release-complete until its active browser section passes. + +Planned roadmap phases must not invent screens, fields, or menu names before implementation. Their qualification contracts are activated and expanded into exact UI steps in this file when the corresponding UI exists. + +## 2. Evidence record + +For every run, record: + +```text +Date: +Operator: +Commit SHA: +Release/image tags: +Environment: +Control-plane URL: +Topology: +DNS clusters: +Edges and POPs: +Cell slots per edge: +Pool routing modes: +Public IPv4/IPv6: +Browser and version: +Desktop viewport: +Mobile viewport: +Test domains: +Origin fixtures: +Start time: +End time: +Result: passed / failed / blocked +``` + +For each checkpoint, record: + +- expected result; +- actual result; +- sanitized screenshot or screen recording; +- operation, task, revision, pool, cell, edge, certificate, purge, or backup identifier; +- browser console/network error when relevant; +- severity and owner for failures; +- retest evidence. + +Never include passwords, tokens, API keys, private keys, CA keys, signing keys, backup credentials, raw customer telemetry, or customer data. + +## 3. Global preparation + +1. Use a disposable or approved production-like environment without deleting persistent Compose volumes. 2. Apply Laravel and PowerDNS migrations explicitly. -3. Confirm `/api/health`, `/api/ready`, DNSdist UDP/TCP, Horizon, and scheduler. -4. Create one administrator with `cdnf:admin:create`. -5. Prepare one unused domain-user email, one delegated test domain, two edge - addresses, two origin fixtures, and two viewport sizes. -6. Sign in at `/admin`; confirm keyboard focus is visible, labels are associated, - tables scroll on a narrow viewport, and no secret appears in page source or logs. - -## Phase 1: foundation and access - -### Administrator and user - -1. Open **Customers → Users → New user**. -2. Enter name, unique email, type **User**, password, and confirmation. -3. Save. Expect one user row and no plaintext password display. -4. Edit the user and choose disable. Expect disabled status. -5. In another browser profile, attempt `/app` login. Expect denial. -6. Re-enable the user. Expect login to `/app`; no API token is recreated. -7. Confirm an administrator cannot disable, demote, or delete their own row. - -### Profile and tokens - -1. Open **Account → Profile** in each panel. -2. Change display name and save. Expect the header/account widget to update. -3. Change password with current password and matching confirmation. -4. Open **Account → API tokens**; create a named token. -5. Record that plaintext appears once; navigate away and back. -6. Expect only metadata/final six characters. Revoke it and confirm it disappears. +3. Confirm `/api/health`, `/api/ready`, DNSdist UDP/TCP, Horizon, scheduler, PostgreSQL, Redis/Valkey, ClickHouse, Vector, MMDB, registered edges, cells, and gateway health where implemented. +4. Create or identify one administrator and one disposable domain user. +5. Prepare delegated test domains, IPv4/IPv6 origin fixtures, one failing origin, and approved external DNS/HTTP vantage points. +6. Confirm the exact commit and immutable image tags being qualified. +7. Use desktop and narrow mobile viewports. +8. Confirm keyboard focus, labels, validation focus, table scrolling, empty/loading/degraded/error states, destructive confirmation, and one-time-secret boundaries throughout the run. +9. Confirm no secret appears in page source, browser storage beyond intended session/token data, browser console, rendered errors, audit details, or downloadable exports. + +# Active qualification — implemented baseline + +## Phase 1 — Foundation, access, and system identity + +### Administrator and domain user + +1. Sign in to the administrator panel. +2. Create a user with a unique disposable email and type `User`. +3. Confirm the user appears once and no plaintext password is displayed after save. +4. Disable the user and confirm login to the domain-user panel is denied. +5. Re-enable the user and confirm login succeeds. +6. Confirm an administrator cannot disable, demote, or delete their own active account. +7. Confirm a domain user cannot open administrator routes or see global resources. + +### Profile, password, and tokens + +1. Open the profile surface in both panels. +2. Change the display name and confirm shared navigation/account UI updates. +3. Change the password using the current password and matching confirmation. +4. Create a named API token. +5. Confirm plaintext token appears once only. +6. Reload and confirm only safe metadata is visible. +7. Revoke the token and confirm API use fails. ### System DNS identity -1. Open **Control plane → System DNS identity**. -2. Enter platform zone, two nameservers, IPv4 and IPv6 glue for each, proxy - hostname, SOA contact, refresh, retry, expire, minimum, and TTL values. -3. Preview. Expect normalized records and a confirmation token. -4. Change one field after preview and try the old token. Expect rejection. -5. Preview again, apply the exact token, and record the returned operation. -6. Expect pending/running then succeeded deployment state without a blocked page. +1. Open the system DNS identity surface. +2. Enter a platform zone, two nameservers, IPv4/IPv6 glue, proxy hostname, SOA mailbox, and bounded timing values. +3. Preview and record the normalized records and confirmation receipt. +4. Change one field and attempt to apply the old confirmation. Expect rejection. +5. Preview again and apply the exact payload. +6. Record the asynchronous operation and confirm pending/running/succeeded states without blocking the page. +7. Confirm failed deployment keeps the previous active identity. -### Phase 1 completion gate +### Completion record | Gate | Result | | --- | --- | -| Implementation | Present | -| Documentation | Current | -| Automated/runtime qualification | Record current run | -| Manual browser qualification | Pass only when every Phase 1 step above is recorded | +| Implementation | | +| Documentation | | +| Automated/runtime tests | | +| Scale checkpoint | | +| Failure/recovery | | +| Manual browser | | +| Release decision | | -## Phase 2: domains and authoritative DNS +## Phase 2 — Domains and authoritative DNS ### DNS cluster -1. Open **Control plane → DNS clusters → New cluster**. -2. Enter unique name, location, HTTPS API URL, API key, server ID, nameserver - list, and optional operational notes. -3. Save. Expect disabled/pending health and no API key in the table or edit form. -4. Edit and choose **Test connection**. Record the operation. -5. After success, enable the cluster and reconcile all zones. -6. Expect active checksums and no replacement of a healthy cluster's old state on a deliberately failed second target. +1. Create a DNS cluster with unique name, location, private API details, server ID, and nameserver data. +2. Confirm credentials are not displayed in table, detail, audit, or edit surfaces. +3. Test the connection and record the operation. +4. Enable the qualified cluster. +5. Reconcile zones and confirm desired/active revision and checksum visibility. +6. Introduce a controlled failure on a second target and confirm the healthy target retains valid state. ### Domain lifecycle and assignment -1. Open **Domains → New domain** and enter the delegated test domain. -2. Expect `pending_verification`, revision, nameserver guidance, and no origin. -3. Attach the domain user from the **Users** relation. -4. Sign in as that user. Expect only the assigned domain. -5. Choose **Verify nameservers**, wait for success, then **Activate**. -6. Expect active lifecycle and acknowledged DNS deployment. +1. Add a delegated test domain without entering an origin. +2. Confirm pending verification, revision, nameserver guidance, and no proxy/TLS side effect. +3. Assign the disposable domain user. +4. Sign in as that user and confirm only assigned domains are visible. +5. Verify nameservers and activate the domain. +6. Confirm lifecycle and DNS deployment acknowledgement. +7. Exercise delayed deprovision/cancel behavior only in the disposable environment and confirm tombstone/reclaim safeguards. -### DNS records +### DNS records and bulk work -1. Add A, AAAA, CNAME, MX, TXT, NS, CAA, and SRV examples with valid fields. -2. Confirm TTL minimum 30; MX requires priority; SRV requires - `_service._protocol`, priority, weight, and port. -3. Try CNAME beside another record at the same owner. Expect inline rejection. -4. Try an owner outside the zone and a non-SRV underscore. Expect rejection. -5. As a domain user, try to change apex delegation NS. Expect the action hidden or denied. -6. Bulk delete selected non-delegation records. Expect one revision change. -7. Import a small BIND zone with append, then export it. Expect deterministic text. -8. Import with replacement and confirm the final record set. +1. Add valid A, AAAA, CNAME, MX, TXT, NS, CAA, and SRV examples. +2. Confirm type-specific fields, normalized names, and TTL limits. +3. Attempt CNAME coexistence, out-of-zone owner, invalid underscore, duplicate, and malformed IPv4/IPv6 input. Expect inline rejection and no revision change. +4. Confirm a domain user cannot change protected apex delegation behavior. +5. Bulk edit/delete a bounded set and confirm one intended revision transition. +6. Import a bounded BIND zone in append mode and export deterministic text. +7. Import in replacement mode and confirm the final record set. +8. Confirm cursor pagination and useful empty/degraded states. -### Phase 2 completion gate +### External verification -| Gate | Result | -| --- | --- | -| Implementation | Present | -| Documentation | Current | -| Automated/runtime qualification | Record PHP, DNS, and scale evidence | -| Manual browser qualification | Pass only when every Phase 2 step above is recorded | +1. Use real UDP and TCP `dig` against every qualified DNS cluster. +2. Test IPv4 and IPv6 client paths. +3. Confirm SOA serial, authoritative flags, record values, and negative answers. +4. Stop the control plane and confirm existing authoritative DNS continues. -## Phase 3: Geo-DNS +### Completion record -1. Create an A record with mode **Geo-DNS**. -2. Add a default address, one continent override, and one country override. -3. Save. Expect normalized configuration and one domain revision. -4. Use **Preview** with an IP for each rule and an unknown/documentation IP. -5. Expect country before continent before default, with displayed classification. -6. Repeat with AAAA and confirm IPv6 input. -7. Try duplicate codes, no default target, too many targets, and CAA Geo-DNS. -8. Expect validation and no revision change. -9. Verify real DNS from approved external vantage points and record ECS behaviour. - -### Phase 3 completion gate +| Gate | Result | +| --- | --- | +| Implementation | | +| Documentation | | +| Automated/runtime tests | | +| Scale checkpoint | | +| Failure/recovery | | +| Manual browser/external DNS | | +| Release decision | | + +## Phase 3 — Geo-DNS + +1. Create a Geo-DNS A record with default, continent, and country targets. +2. Preview with addresses matching each level and with an unknown/documentation address. +3. Confirm country wins before continent, then default. +4. Repeat using AAAA and IPv6 preview input. +5. Attempt duplicate geography, missing default, excessive targets, invalid address family, and unsupported record type. Expect rejection and no revision change. +6. Confirm the UI labels resolver/ECS accuracy honestly. +7. Verify from approved external vantage points and record ECS or resolver-based behavior. +8. Interrupt MMDB update/provider access and confirm the last valid database remains active. + +### Completion record | Gate | Result | | --- | --- | -| Implementation | Present | -| Documentation | Current | -| Automated/runtime qualification | Record PHP and Geo-DNS runtime evidence | -| Manual browser/external vantage qualification | Pass only when every Phase 3 step is recorded | - -## Phase 4: proxy and edge - -### Pools, edges, and enrollment - -1. Create one shared and one quarantine pool with unique stable names. -2. Create two edges with name, country, continent, unique IPv4, and optional IPv6. -3. Record each UUID/bootstrap token once; reload and expect token secrecy. -4. Edit each edge's cells and enter unique public IPv4/optional IPv6. -5. Enroll the real agents. Expect registered identity, fresh heartbeat, - `listener_ready`, ready cells, version, sequence, and bounded capacity. -6. Rotate one identity. Expect the old agent to fail and a new token to appear once. -7. Restore enrollment, then drain/undrain a cell and one edge. Expect durable operations and routing exclusion/reinclusion. - -### Proxied hostname - -1. On the domain DNS relation, add a proxied A, AAAA, or CNAME. -2. Enter origin host, HTTP/HTTPS scheme, host header, verified HTTPS SNI, - connect timeout, response timeout, retry count, WebSocket flag, and optional health-check path/interval. -3. Expect DNS content to become platform managed and the explicit origin to remain visible. -4. Try loopback, link-local, metadata, multicast, edge listener, and proxy hostname origins. Expect rejection. -5. Run **Test origin**. Expect an asynchronous result with address, latency, - status, or stable failure reason. -6. Inspect edge delivery. Expect both eligible agents to acknowledge the revision. -7. Move the domain to the quarantine pool. Expect target ready before source drain. -8. Roll back to a retained revision. Expect a new higher revision, not a decremented number. - -### Phase 4 completion gate +| Implementation | | +| Documentation | | +| Automated/runtime tests | | +| Scale checkpoint | | +| Failure/recovery | | +| Manual browser/external vantage | | +| Release decision | | + +## Phase 4 — Proxy, baseline edge pools, and edge agent + +### Pools and edges + +1. Create or inspect shared, quarantine, and exceptional dedicated pools. +2. Create two edges in different locations with unique IPv4 and optional IPv6 management/default addresses. +3. Record each bootstrap token once and confirm it disappears after navigation. +4. Enroll real agents and confirm identity, version, heartbeat, active sequence, cell state, and bounded capacity. +5. Rotate one identity and confirm the old identity is rejected. +6. Restore the edge and exercise drain/undrain. +7. Confirm routing and status change without affecting the other edge. + +### Proxied hostname and origin + +1. Enable proxy for a valid hostname. +2. Configure explicit origin scheme, host, port, Host header, SNI, TLS verification, timeouts, retry bound, WebSocket option, and health path where supported. +3. Confirm platform-managed DNS content and visible safe origin metadata. +4. Attempt loopback, private/disallowed, link-local, metadata, multicast, platform, edge-service, and proxy-loop destinations. Expect rejection. +5. Run the asynchronous origin test and record resolved address, status, latency, TLS result, or stable failure reason. +6. Confirm signed edge delivery acknowledgement. +7. Move the domain to quarantine and confirm target ready before source drain. +8. Roll back to a retained revision and confirm a new higher revision. + +### Real traffic + +1. Send HTTP and HTTPS through each eligible edge using correct Host/SNI. +2. Test IPv4 and IPv6. +3. Confirm unknown Host/SNI rejection. +4. Stop Laravel/queues temporarily and confirm existing traffic continues. +5. Submit an invalid artifact in the controlled runtime test and confirm the previous state remains active. + +### Completion record | Gate | Result | | --- | --- | -| Implementation | Present | -| Documentation | Current | -| Automated/runtime qualification | Record PHP, Go, control, mTLS, and OpenResty evidence | -| Manual browser/real traffic qualification | Pass only when every Phase 4 step is recorded | +| Implementation | | +| Documentation | | +| Automated/runtime tests | | +| Scale checkpoint | | +| Failure/recovery | | +| Manual browser/real traffic | | +| Release decision | | -## Phase 5: TLS, cache, and purge +## Phase 5 — TLS, baseline cache, and purge ### Managed and custom TLS -1. With an active proxied domain, open the TLS section. -2. Expect managed mode and a visible pending/succeeded order without private data. -3. Verify public apex/wildcard HTTPS and record the fingerprint. -4. Use **Renew**, then **Reissue**; record operation IDs and final certificate state. -5. Upload a valid leaf, chain, and matching private key. Expect custom mode and metadata only. -6. Try a wrong key, wrong name, invalid chain, expired certificate, and oversized PEM. Expect rejection. -7. Remove the custom certificate. Expect managed mode and asynchronous reconciliation. -8. Set TLS disabled and confirm intended HTTPS behaviour; return to managed. - -### Cache - -1. Open **Cache settings** and set enabled, edge/browser TTLs, one allowed object - size, origin-header respect, query policy, bypass cookies, and stale grace. -2. Save and wait for edge acknowledgement. -3. Request a cacheable object twice. Expect MISS then HIT and correct browser headers. -4. Enable development mode for a short duration. Expect absolute expiry and BYPASS. -5. Disable it and expect caching to resume. -6. Purge one exact URL. Expect one task per eligible edge and a later MISS. -7. Purge everything. Expect an incremented epoch, no filesystem scan, and later MISS. -8. Create a controlled cell-delivery failure. Expect visible retry of the same task. - -### Phase 5 completion gate +1. Enable proxy for an eligible active domain and open its TLS surface. +2. Confirm managed issuance state without private material. +3. Verify public apex/wildcard HTTPS as applicable and record certificate fingerprint, names, issuer, and expiry. +4. Exercise renew and reissue operations. +5. Upload a valid leaf, chain, and matching key and confirm only safe metadata remains visible. +6. Attempt wrong key, wrong name, invalid chain, expired certificate, and oversized PEM. Expect rejection. +7. Remove the custom certificate and confirm managed reconciliation. +8. Confirm a current valid certificate continues during a controlled ACME failure. + +### Cache and purge + +1. Configure enabled state, edge/browser TTL, object size, origin-header policy, query behavior, bypass cookies, and stale grace. +2. Save and confirm edge acknowledgement. +3. Request a cacheable object twice and confirm MISS then HIT. +4. Confirm browser headers and deterministic cache key behavior. +5. Enable short development mode and confirm absolute expiry and BYPASS. +6. Disable it and confirm caching resumes. +7. Purge one exact URL and confirm bounded target tasks and a later MISS. +8. Purge all and confirm epoch increment without filesystem scan. +9. Cause one controlled delivery failure and confirm safe retry of the same durable task. + +### Completion record | Gate | Result | | --- | --- | -| Implementation | Present | -| Documentation | Current | -| Automated/runtime qualification | Record PHP, Pebble, cache, purge, and OpenResty evidence | -| Manual browser/public HTTPS qualification | Pass only when every Phase 5 step is recorded | - -## Phase 6: security and DDoS readiness - -1. Open **Security profile and limits**. -2. Select standard, protected, quarantine, then manual. Expect fixed profile - values and manual values bounded by platform ceilings. -3. Select a quarantine policy, allowed methods, and trusted proxy CIDRs; save. -4. Add IP, CIDR, country, and continent allow/block rules with priorities. -5. Import rules with append and replacement. Expect duplicate/limit validation. -6. Send matching IPv4 and IPv6 traffic. Expect rule order and reason visibility. -7. Use **Restrict domain**, **Quarantine domain**, then **Release domain**. -8. Expect target-first placement and `recovering` before normal where applicable. -9. Apply and clear an expiring emergency mode to an edge and cell. -10. Withdraw and restore a pool. Expect DNS publication to change without corrupting another pool. -11. Confirm traffic to an unrelated cell/domain remains healthy throughout. - -### Phase 6 completion gate +| Implementation | | +| Documentation | | +| Automated/runtime tests | | +| Scale checkpoint | | +| Failure/recovery | | +| Manual browser/public HTTPS | | +| Release decision | | + +## Phase 6 — Security and DDoS readiness + +1. Select standard, protected, quarantine, and manual profiles. +2. Confirm fixed profiles cannot exceed their values and manual input remains within platform ceilings. +3. Configure quarantine policy, allowed methods, and trusted proxy CIDRs. +4. Add ordered IP, CIDR, country, and continent allow/block rules. +5. Import bounded rules in append and replacement modes. +6. Send matching IPv4 and IPv6 traffic and confirm action/reason visibility. +7. Exercise restrict, quarantine, recover, and release. +8. Confirm target-first placement and safe recovery state. +9. Apply an expiring emergency action and confirm persistence across restart, expiry, and audit. +10. Withdraw and restore a pool service address from DNS. +11. Confirm an unrelated domain/cell remains healthy during controlled abuse traffic. +12. Confirm the UI and documentation do not claim protection after physical uplink saturation. + +### Completion record | Gate | Result | | --- | --- | -| Implementation | Present | -| Documentation | Current | -| Automated/runtime qualification | Record PHP and security real-runtime evidence | -| Manual browser/real attack-pattern qualification | Pass only when every Phase 6 step is recorded | - -## Phase 7: analytics and usage - -1. Generate controlled DNS and HTTP traffic, cache hits/misses, origin failure, - TLS, and security events. -2. As the domain user, open **Observe → Analytics**. -3. Check summary, timeseries, status, cache, countries, hostnames, URLs, origin, - edges, and DNS with range and unit labels. -4. Open request, DNS, error, and security logs where surfaced. Expect opaque - pagination, no query strings/secrets, and masked IPv4/IPv6. -5. Download domain usage CSV. Expect stable columns and only the assigned domain. -6. As administrator, open **Observe → Telemetry** and global views/export. -7. Stop ClickHouse in the controlled environment. Expect a visible outage, not - a broken DNS or edge service. -8. Restore ClickHouse and confirm Vector backlog behaviour and partial interval labels. - -### Phase 7 completion gate +| Implementation | | +| Documentation | | +| Automated/runtime tests | | +| Scale/isolation checkpoint | | +| Failure/recovery | | +| Manual browser/attack-pattern traffic | | +| Release decision | | + +## Phase 7 — Telemetry, analytics, and usage export + +1. Generate controlled DNS, HTTP, HTTPS, cache, origin, TLS, security, edge, and deployment events. +2. As the domain user, open analytics and confirm only assigned-domain data. +3. Verify summary, time series, units, ranges, status, cache, geography, hostname, URL, origin, edge, and DNS views that are currently implemented. +4. Inspect available raw request, DNS, error, and security logs. +5. Confirm bounded pagination, masking/redaction, and no secrets/query data beyond the documented schema. +6. Download domain usage JSON/CSV and confirm stable columns and scope. +7. As administrator, inspect global telemetry, edge/cell health, drops, and buffer state. +8. Stop ClickHouse and confirm visible analytics outage while DNS/HTTP continues. +9. Restore it and confirm bounded backlog recovery and partial-data labeling. + +### Completion record | Gate | Result | | --- | --- | -| Implementation | Present | -| Documentation | Current | -| Automated/runtime qualification | Record PHP and analytics outage/recovery evidence | -| Manual browser qualification | Pass only when every Phase 7 step is recorded | - -## Phase 8: operations and release - -1. Open the administrator dashboard. Confirm overall status, component detail, - all four queue lanes, recent operations, and no secret values. -2. Open **Operations**. Inspect pending, succeeded, and failed rows; retry one supported failure. -3. Open failed jobs through the API/operational surface. Confirm payloads are bounded and redacted. -4. Open **Platform settings**. Change a non-runtime group and a runtime group. -5. Expect typed bounds, audit history, and a global operation only for runtime-affecting policy. -6. Create a backup. Expect snapshot metadata and verification without download. -7. Start restore preflight with exact confirmation and current password. -8. On an isolated host, perform the maintenance restore and complete the recovery checklist. -9. Run one canary upgrade and rollback with prior/current compatible images. -10. Verify desktop/mobile layout, keyboard operation, error focus, tables, - destructive confirmations, CSV downloads, and one-time secret boundaries. - -### Phase 8 completion gate +| Implementation | | +| Documentation | | +| Automated/runtime tests | | +| Scale/query checkpoint | | +| Failure/recovery | | +| Manual browser | | +| Release decision | | + +## Phase 8 — Operations, recovery, and baseline release + +1. Inspect the administrator dashboard and confirm component, queue, edge, cell, DNS, TLS, backup, and recent-operation states currently implemented. +2. Inspect pending, succeeded, and failed operations and retry one supported failure. +3. Inspect failed-job information and confirm payload redaction and bounds. +4. Change one non-runtime typed setting and one runtime-affecting setting. +5. Confirm only the runtime change creates deployment/reconciliation work. +6. Create and verify an encrypted backup. +7. Perform restore preflight with exact confirmation and re-authentication. +8. Complete the documented clean-host maintenance restore in an isolated environment. +9. Run one compatible canary upgrade and rollback. +10. Confirm desktop/mobile layout, keyboard use, error focus, pagination, downloads, confirmations, and one-time secrets. +11. Record RPO, RTO, hardware, topology, throughput, saturation, and unresolved limitations. + +### Completion record | Gate | Result | | --- | --- | -| Implementation | Present | -| Documentation | Current | -| Automated/runtime qualification | Record operations, recovery, upgrade, throughput, MMDB, Compose, and image evidence | -| Manual browser/production qualification | Pass only when every Phase 8 and external release step is recorded | +| Implementation | | +| Documentation | | +| Automated/runtime tests | | +| Scale/throughput checkpoint | | +| Backup/recovery/upgrade | | +| Manual browser/production | | +| Release decision | | + +# Planned qualification contracts + +The following sections are inactive until their implementation exists. They define required outcomes without inventing menu names, pages, fields, or controls. When a phase is implemented, replace its section with exact browser steps derived from the real UI and keep every checkpoint below. + +## Phase 9 — Edge gateway and bounded cell inventory + +Required owner-visible outcomes: + +- installed cell-slot inventory, stable cell identity, assignment state, resource limits, health, active revision, and last report are inspectable; +- gateway service addresses, listeners, Host/SNI map revision, health, and last-valid state are inspectable; +- agent enrollment does not expose Docker control or create unbounded containers; +- drain, restart, invalid-map rejection, and rollback operations provide durable status and audit; +- desktop/mobile and degraded/empty/error states are usable. + +Required real-traffic evidence: + +- one public service IP pair routes HTTP Host and HTTPS SNI to the expected cells; +- unknown/mismatched traffic is rejected; +- client address preservation is correct; +- one cell restart does not interrupt unrelated cells; +- invalid gateway state preserves the previous map. + +Activation rule: write exact UI steps when Phase 9 surfaces are implemented. + +## Phase 10 — Multi-cell pools, endpoints, and stable placement + +Required owner-visible outcomes: + +- pool kind, participating edges, service endpoints, assigned cells, minimum-ready policy, capacity, and domain placement are inspectable; +- shared, reserved, dedicated, and quarantine constraints are enforced; +- slot assignment, release, drain, migration, and rollback have operation status and audit; +- only participating targets show deployment/acknowledgement. + +Required real-traffic evidence: + +- one IPv4/IPv6 pair fronts three shared cells; +- another pair fronts reserved customer cells on the same edge; +- stable domain placement preserves cache locality; +- migration is target-first and unrelated domains remain healthy; +- cell shortage and protected capacity reservations fail safely. + +Activation rule: write exact UI steps when Phase 10 surfaces are implemented. + +## Phase 11 — Geo-Unicast and Simple Anycast + +Required owner-visible outcomes: + +- routing mode, shared Anycast addresses, participating POPs, gateway/cell/revision readiness, route-advertised signal, and withdrawal state are inspectable; +- Geo controls are hidden or unavailable for Anycast pools; +- BGP neighbor credentials/configuration are absent; +- readiness and failure ordering are clear and audited. + +Required external evidence: + +- Geo-Unicast returns ready location endpoints; +- Simple Anycast returns one shared IPv4/IPv6 pair; +- multiple POPs serve the same active revision; +- failed POP withdrawal precedes exclusion; +- healthy POPs continue; +- control-plane outage does not stop valid externally routed traffic. + +Activation rule: write exact UI steps when Phase 11 surfaces are implemented. + +## Phase 12 — Cache v2 and compression + +Required owner-visible outcomes: + +- cache resource profile, persistent quota, free-space threshold, inactive period, object-size limit, query policy, cookie bypass, status TTL, stale policy, and compression profile are inspectable and bounded; +- Gzip/Brotli availability and effective profile are clear; +- disk, admission, variant, compression CPU, savings, purge, and degradation states are visible; +- emergency compression disable and retry/rollback are audited. + +Required real-traffic evidence: + +- identity, Gzip, and Brotli clients receive identical decoded content; +- cache remains canonical without accidental `Accept-Encoding` fragmentation; +- MIME, size, range, and precompressed exclusions work; +- MISS/HIT/STALE/revalidate/purge remain correct; +- disk/CPU pressure is bounded and unrelated cells remain healthy. + +Activation rule: write exact UI steps when Phase 12 surfaces are implemented. + +## Phase 13 — Simple origin resilience + +Required owner-visible outcomes: + +- primary/backup origin, health evidence, effective origin, failover/recovery state, thresholds, last transition, and audit are inspectable; +- both origins use the same safety validation; +- no weighted or percentage routing controls exist. + +Required real-traffic evidence: + +- healthy primary serves; +- qualified failure switches to backup; +- flapping is bounded; +- failed backup cannot loop retries; +- controlled primary recovery works; +- cache and unrelated domains remain correct. + +Activation rule: write exact UI steps when Phase 13 surfaces are implemented. + +## Phase 14 — Managed OWASP CRS WAF + +Required owner-visible outcomes: + +- off, monitor, balanced, and strict profiles are available only where supported; +- effective pinned engine/ruleset versions, WAF-capable pool/cell state, anomaly/action metrics, exclusions, expiry, audit, and rollout state are inspectable; +- exclusions are narrow and bounded; +- no raw ModSecurity or custom rule-language editor exists. + +Required real-traffic evidence: + +- monitor records but does not block; +- balanced/strict block qualified test cases; +- false-positive exclusion works only in its declared scope; +- invalid ruleset preserves the previous image/rules; +- WAF load is isolated from normal pools; +- canary rollout pauses and rolls back. + +Activation rule: write exact UI steps when Phase 14 surfaces are implemented. + +# Future admission + +Phases 15 and 16 remain future candidates. Do not add browser checkpoints until a capability is admitted, implemented, and has real UI behavior. + +# Final result record + +For every failed checkpoint record: -## Result record +```text +Phase: +Checkpoint: +Expected: +Actual: +Evidence: +Related IDs: +Severity: +Owner: +Decision: +Retest date: +Retest result: +``` -For each failed checkpoint, record expected result, actual result, sanitized -evidence, operation/task/revision IDs, severity, owner, and retest. The release -remains unqualified until every current checkpoint passes or an explicit -product-contract change removes it. +A release remains unqualified until every active checkpoint passes or the product contract explicitly removes it. From 8029923c383519b4f85382709d93eb62c4509b80 Mon Sep 17 00:00:00 2001 From: vaheeD khoshnouD Date: Mon, 27 Jul 2026 14:04:29 +0330 Subject: [PATCH 3/7] docs: replace roadmap with post-baseline development plan --- docs/roadmap.md | 1107 +++++++++++++++++++++++++---------------------- 1 file changed, 595 insertions(+), 512 deletions(-) diff --git a/docs/roadmap.md b/docs/roadmap.md index 2320e26..dd539b8 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -1,591 +1,674 @@ --- -title: CDNFoundry product roadmap -description: Ordered implementation and qualification contract for a simple, solid, production-grade private CDN. +title: CDNFoundry next development roadmap +description: Ordered post-baseline roadmap for the new edge, cache, origin-resilience, WAF, and fleet capabilities. --- -# CDNFoundry product roadmap - -## 1. Product goal - -CDNFoundry is a small, production-grade private CDN and authoritative DNS platform for operators that own or manage their infrastructure. - -The product favors: - -- low feature count; -- clear operations; -- bounded resource usage; -- deterministic desired state; -- failure isolation; -- last-valid-state recovery; -- real production qualification. - -The product does not attempt to reproduce Cloudflare, Fastly, Akamai, or a general cloud platform. - -A phase is complete only when its implementation, documentation, automated tests, real-runtime qualification, scale checkpoint, and owner-run browser qualification are all recorded. - -## 2. Non-negotiable boundaries - -1. One Laravel modular monolith and two Filament panels form the control plane. -2. PostgreSQL is the durable source of desired state. -3. DNS, HTTP, HTTPS, TLS selection, cache decisions, security decisions, and raw telemetry never pass through Laravel. -4. PowerDNS runtime state, edge artifacts, generated gateway maps, and analytics aggregates are derived and rebuildable. -5. External effects are asynchronous, revisioned, idempotent, coalesced, acknowledged, and last-valid-state preserving. -6. DNSdist is the only public authoritative DNS endpoint. -7. Edge traffic uses a small gateway and bounded identical OpenResty cells. -8. Cell containers are created during edge installation. The edge agent assigns and configures cells; it does not receive unrestricted Docker access. -9. No domain receives a default process, container, worker, timer, cache directory, Nginx server block, or reload. -10. Per-domain, per-pool, per-cell, queue, log, cache, import, purge, rule, artifact, and query limits are explicit. -11. DDoS readiness reduces application and noisy-neighbour blast radius but does not claim upstream volumetric scrubbing after physical capacity is saturated. -12. No microservices, Kafka, Kubernetes requirement, CQRS, event sourcing, GraphQL, custom RBAC, plugin runtime, reseller hierarchy, billing engine, custom expression language, or additional dashboard is introduced without an explicit product-contract change. -13. Production code and filenames never use roadmap phase numbers or lifecycle suffixes such as `V2`, `New`, or `Final`. -14. Development PostgreSQL and named Compose volumes remain persistent across all work. Destructive refreshes are forbidden. - -## 3. Required completion gate for every phase - -Every phase must record the following independently: +# CDNFoundry next development roadmap + +## Current boundary + +The original production platform roadmap is complete and is not repeated here. +The existing Laravel control plane, authoritative DNS, Geo-DNS, proxy, TLS, +baseline cache and purge, security controls, telemetry, analytics, backup, and +operations behavior are the starting baseline for this roadmap. + +This document contains only the **new development work** agreed after completion +of that baseline. + +Every new phase must: + +- extend the existing product without rewriting it; +- remain deployable and testable when the phase is finished; +- preserve all previously working behavior; +- stay simple to develop and operate; +- use bounded resources and explicit failure behavior; +- include tests, scale evidence, documentation, and owner-run browser qualification. + +## Product direction + +CDNFoundry remains a small, solid private CDN for operators and ISPs. It is not +trying to become Cloudflare, Fastly, Akamai, or a general cloud platform. + +The next development cycle focuses on five practical outcomes: + +1. multiple isolated OpenResty cells on one edge; +2. multiple public IPv4/IPv6 service pairs on the same edge; +3. Geo-Unicast and Simple Anycast pool routing; +4. stronger cache, compression, origin resilience, and managed WAF behavior; +5. safe operation and upgrades of the expanded edge fleet. + +## Non-negotiable boundaries + +1. Laravel and Filament remain the single management/control plane. +2. PostgreSQL remains the durable desired-state source of truth. +3. DNS, HTTP, HTTPS, TLS selection, cache decisions, WAF decisions, and raw + telemetry never pass through Laravel. +4. The edge gateway is a small data-plane router. It does not cache, terminate + customer TLS, run WAF logic, call Laravel, or query a database. +5. OpenResty cells perform TLS, cache, security, WAF when enabled, origin proxy, + and request telemetry. +6. Cell containers are created as bounded generic slots during edge installation. + The edge agent assigns and configures them; it does not receive unrestricted + Docker or containerd access. +7. A domain does not receive a default container, process, worker, timer, cache + directory, server block, or reload. +8. Pools may use many cells on one edge. A public IPv4/IPv6 pair belongs to a + pool endpoint, not necessarily to one cell. +9. Simple Anycast uses operator/provider routing. CDNFoundry does not become a + BGP controller and does not require FRR, BIRD, or direct router access. +10. External effects remain asynchronous, revisioned, idempotent, coalesced, + acknowledged, and last-valid-state preserving. +11. No custom WAF language, arbitrary ModSecurity directives, user scripts, + plugin runtime, or downloaded runtime rules are introduced. +12. No microservices, Kafka, Kubernetes requirement, CQRS, event sourcing, + GraphQL, custom RBAC, reseller hierarchy, billing engine, or second dashboard. +13. No unlimited claims. Every scale result records topology, hardware, + concurrency, dataset, saturation point, and accepted limits. +14. Development PostgreSQL and named Compose volumes remain persistent. Tests + use the repository's isolated test environment and never destructively reset + persistent development data. +15. Roadmap phase names and numbers never appear in production class names, + filenames, routes, migrations, tables, or configuration keys. + +## Completion gate for every phase + +A phase is complete only when every applicable row is recorded independently. | Gate | Required evidence | | --- | --- | -| Implementation | Durable state, API, policies, UI where required, jobs/reconciliation, metrics, audit, and rollback behavior | -| Unit and feature tests | Happy path, permissions, validation, idempotency, bounds, and stable errors | -| Runtime and E2E tests | Real DNS, HTTP, HTTPS, TLS, cache, security, telemetry, restart, and failure behavior where applicable | -| Scale checkpoint | Dataset, concurrency, hardware/topology, measured result, saturation point, and accepted limit | -| Failure and recovery | Retry, obsolete work, last-valid state, dependency outage, restart, rollback, and reconciliation | -| Documentation | User, administrator, API, architecture, operations, metrics, troubleshooting, and runbook updates | -| Browser qualification | Owner-run steps and evidence in `docs/manual-browser-qualification.md` | -| Release decision | Passed, failed, blocked, or explicitly removed from scope | - -A phase cannot be marked complete when a required test was not executed. - -## 4. Qualification rules +| Implementation | Durable state, migrations, policies, API, UI where required, jobs, reconciliation, audit, metrics, and rollback behavior | +| Unit and feature tests | Happy path, authorization, validation, bounds, idempotency, stable errors, and backward compatibility | +| Real-runtime E2E | Real HTTP/HTTPS/DNS/cache/TLS/WAF/runtime behavior; mocks alone are insufficient | +| IPv4 and IPv6 | Both families pass whenever the phase handles addressing or traffic | +| Scale checkpoint | Dataset, topology, hardware, concurrency, measured result, saturation point, and accepted limit | +| Failure and recovery | Restart, dependency outage, invalid candidate, retry, obsolete work, rollback, and last-valid state | +| Isolation | Failure or load in one domain, pool, cell, edge, or component does not unnecessarily affect unrelated traffic | +| Observability | Metrics, logs, reason codes, alerts, health, capacity, and partial/degraded state | +| Documentation | User/admin guide, API/OpenAPI, architecture, deployment, operations, troubleshooting, and runbooks | +| Manual qualification | Owner-run checkpoints in `docs/manual-browser-qualification.md` | +| Regression | Existing completed platform behavior remains functional | +| Release decision | Passed, failed, blocked, or deliberately removed from scope | + +A checkbox cannot be marked complete when the related test was not executed. +Every phase must finish as a usable production increment; no phase may leave the +active serving path dependent on unfinished work from a later phase. + +# Committed development roadmap + +## Phase 1 — Edge gateway ingress + +### Goal + +Introduce one minimal gateway on every edge so one server can bind multiple +public IPv4/IPv6 service pairs and route traffic to bounded OpenResty cells. + +### Implementation + +- Add one gateway process/container per edge. +- Bind one or more operator-configured public IPv4 and IPv6 service addresses. +- Route HTTP by destination address and validated `Host`. +- Route HTTPS by destination address and TLS SNI without terminating customer TLS. +- Forward the trusted client address to cells through a qualified PROXY protocol + or equivalent fixed internal contract. +- Reject unknown destination addresses, hosts, and SNI names early. +- Load a generated local routing map atomically. +- Keep and restore the previous valid map after invalid configuration or restart. +- Expose gateway readiness, active revision, listener state, connection totals, + routing failures, and per-endpoint health. +- Keep the gateway free of cache, certificates, origin logic, WAF rules, Lua + business logic, and control-plane network calls. -- Tests use the smallest realistic topology that proves the behavior, then a separate scale program proves the declared limits. -- Runtime behavior is not accepted from mocks alone. -- IPv4 and IPv6 are tested whenever the feature handles addresses or traffic. -- Invalid candidates never replace active DNS, gateway, cell, TLS, cache, WAF, or routing state. -- Rapid updates prove coalescing and obsolete-revision exit. -- A failure in one domain, pool, cell, edge, DNS cluster, telemetry component, or queue lane must not unnecessarily affect unrelated traffic. -- Every phase updates OpenAPI, route coverage, current documentation, examples, environment references, metrics, alerts, and runbooks when affected. -- Browser qualification is manual and owner-run. Coding agents maintain the checklist but report it as not run unless the owner records evidence. -- Non-browser E2E and real-runtime qualification is agent-owned under `tests/e2e`. - -# Part One — Simple but solid production platform - -Part One is the ordered production roadmap. Phases 1 through 8 preserve and requalify the current platform foundation. Phases 9 through 14 implement the agreed edge, Anycast, cache, origin, and managed WAF improvements. - -## Phase 1 — Foundation, access, and system identity - -### Outcome - -A recoverable Laravel/Filament control plane with exactly two user types, typed system identity, audit, idempotency, queues, scheduler, health, and safe Compose foundations. - -### Scope +### Scale checkpoint -- Administrator and assigned-domain-user access. -- Sanctum browser sessions and API tokens. -- Policy-aware route binding. -- User lifecycle, profile, password, token, and audit behavior. -- Typed platform DNS identity preview, exact confirmation, asynchronous apply, and rollback. -- Horizon queue lanes, scheduler freshness, health, readiness, and protected metrics. -- Development and production Compose definitions with explicit networks, volumes, limits, health checks, and migrations. +- At least 50,000 hostname/SNI mappings in one generated map. +- Multiple IPv4 and IPv6 service pairs on one edge. +- Measured HTTP and TLS pass-through throughput with gateway CPU, memory, + connections, latency, and saturation recorded. ### Completion checklist -- [ ] Administrator and domain-user boundaries pass API and browser tests. -- [ ] Idempotency conflict and replay behavior pass. -- [ ] Secrets appear only at their allowed one-time boundary. -- [ ] System identity preview and exact-confirmation flow pass. -- [ ] Queue and scheduler outage states are visible and bounded. -- [ ] SQLite-isolated Laravel tests fail closed when the safe test database is not active. -- [ ] Compose, OpenAPI, formatting, static, documentation, and link checks pass. -- [ ] Clean startup, graceful shutdown, restart, and last-valid settings tests pass. -- [ ] Manual browser Phase 1 is recorded. - -## Phase 2 — Domains and authoritative DNS - -### Outcome - -A complete authoritative DNS workflow with durable desired state, deterministic deployment, drift detection, and real DNS qualification. - -### Scope - -- Domain lifecycle, assignment, delegation verification, activation, delayed deprovisioning, tombstones, and reclaim safety. -- A, AAAA, CNAME, MX, TXT, NS, CAA, SRV, and reverse-zone PTR. -- Transactional bulk changes and bounded BIND import/export. -- Monotonic serials and deterministic zone rendering. -- Multiple DNS clusters with target acknowledgements. -- DNSdist-only public ingress and private PowerDNS/runtime databases. -- Zone rebuild from control-plane PostgreSQL. +- [ ] HTTP Host and HTTPS SNI route to the intended backend cell. +- [ ] IPv4 and IPv6 listeners pass real traffic. +- [ ] Unknown Host/SNI and unassigned destination addresses are rejected. +- [ ] Real client address reaches the cell through the trusted internal contract. +- [ ] Invalid or partial maps never replace the active map. +- [ ] Gateway restart preserves or reconstructs the last valid routing state. +- [ ] Gateway failure is visible and does not corrupt cell runtime state. +- [ ] Existing direct edge behavior remains available during controlled migration. +- [ ] Tests, scale evidence, metrics, alerts, docs, and manual qualification pass. + +## Phase 2 — Bounded cell inventory and edge installation + +### Goal + +Replace the fixed shared/quarantine runtime assumption with a bounded inventory +of generic OpenResty cell slots created during edge installation. + +### Implementation + +- Add an installation setting such as a bounded cell-slot count. +- Generate/start named generic slots such as `cell-01` through `cell-N`. +- Give every slot stable identity, internal ports, runtime path, cache path, + temporary path, status endpoint, and resource limits. +- Allow unused slots to remain unassigned and idle or stopped according to a + documented installation policy. +- Register the full static inventory with the edge agent. +- Remove readiness hardcoding tied to `shared-default`. +- Let the agent configure, reload, drain, restart, and report existing slots. +- Do not mount the Docker socket into the edge agent. +- Keep agent resources separate from cell resource groups. +- Support safe cell image compatibility and last-valid runtime state. ### Scale checkpoint -- At least 500,000 domains. -- At least 1,000,000 DNS records. -- At least 50,000 DNS changes per day. -- A controlled burst of at least 10,000 mutations. -- Multiple DNS clusters and mixed IPv4/IPv6 data. +- Qualify at least 8 cell slots on one edge. +- Record idle overhead and active overhead per cell. +- Prove one saturated or crashed cell does not terminate the agent, gateway, or + unrelated cells. ### Completion checklist -- [ ] Record validation, CNAME coexistence, zone boundaries, Punycode, duplicates, and TTL limits pass. -- [ ] Bulk operations are bounded, atomic, idempotent, and revisioned. -- [ ] One failed DNS target does not replace another target's valid state. -- [ ] PowerDNS runtime deletion and complete rebuild pass. -- [ ] Real UDP/TCP `dig` tests pass over IPv4 and IPv6. -- [ ] Scale and mutation-coalescing programs pass with measured evidence. -- [ ] DNS user/admin/API/operations documentation and runbooks are current. -- [ ] Manual browser Phase 2 is recorded. - -## Phase 3 — Geo-DNS - -### Outcome +- [ ] Fresh edge installation creates exactly the configured bounded slots. +- [ ] Every slot has unique identity, paths, ports, health, and resource limits. +- [ ] Agent reports assigned, unassigned, ready, degraded, drained, and stopped states. +- [ ] Agent operates without unrestricted container-engine access. +- [ ] Cell restart and image rollback preserve unrelated traffic. +- [ ] Cache/temp/log storage cannot exceed configured quotas. +- [ ] Existing edge enrollment, mTLS, rotation, and snapshot recovery still pass. +- [ ] Tests, scale evidence, docs, and manual qualification pass. + +## Phase 3 — Multi-cell pools and stable domain placement + +### Goal + +Allow one pool to use multiple cells on the same edge while preserving stable +cache locality and predictable failure isolation. + +### Implementation + +- Support pool kinds: + - `shared` for unrelated normal domains; + - `reserved` for one customer's or workload group's domains; + - `dedicated` for one exceptional domain; + - `quarantine` for attacked or unstable domains. +- Separate stable cell-slot identity from pool identity. +- Add explicit edge participation and cell assignment per pool. +- Allow multiple cells from one edge to belong to one pool. +- Add minimum-ready-cell and capacity policy per pool/edge participation. +- Place each normal domain on one stable active cell inside its pool by default. +- Allow exceptional replicated placement only as an explicit bounded mode. +- Move domains target-first: configure target, verify readiness, switch gateway, + then drain and remove source state. +- Deliver artifacts only to active and target participating cells/edges. +- Prevent one dedicated pool from accepting multiple domains. +- Preserve previous placement and gateway state after failed migration. -Bounded country, continent, and default DNS answers without runtime dependency on Laravel or external GeoIP services. - -### Scope +### Scale checkpoint -- Country, continent, then default selection. -- A and AAAA Geo-DNS records. -- Valid ECS use when present and trusted. -- Resolver-address fallback with honest accuracy documentation. -- Shared local MMDB with atomic update and last-valid preservation. -- Deterministic PowerDNS Lua compilation. +- At least 20,000 domains distributed across multiple cells without unnecessary + reshuffling after adding a domain or a new cell. +- Controlled burst of at least 10,000 placement-affecting changes with coalescing. +- Record compiler time, artifact size, gateway-map activation time, and database + query behavior. ### Completion checklist -- [ ] Country, continent, default, and unknown behavior pass. -- [ ] IPv4 and IPv6 previews and real queries pass. -- [ ] Duplicate, missing-default, invalid-type, and excessive-target inputs fail safely. -- [ ] MMDB corruption/provider outage retains the previous valid database. -- [ ] Geo changes do not rewrite unrelated zones. -- [ ] External-vantage evidence records resolver/ECS limitations. -- [ ] Documentation and browser qualification are current. - -## Phase 4 — Proxy, baseline edge pools, and edge agent +- [ ] One shared pool uses at least three cells on one edge. +- [ ] Domain placement remains stable across unrelated changes. +- [ ] Reserved, dedicated, and quarantine constraints are enforced. +- [ ] Failed target readiness leaves source placement active. +- [ ] Successful movement drains and removes the old assignment safely. +- [ ] Artifacts are not sent to non-participating edges or cells. +- [ ] Pool readiness counts all required cells rather than one arbitrary cell. +- [ ] Unrelated domain/cache traffic remains healthy during movement. +- [ ] Tests, scale evidence, docs, and manual qualification pass. + +## Phase 4 — Pool service endpoints and Geo-Unicast + +### Goal + +Support multiple public service IP pairs on one edge, with each pair serving one +pool endpoint backed by one or more cells. + +### Implementation + +- Add a pool endpoint per participating edge with: + - public IPv4; + - optional public IPv6; + - listener/gateway state; + - participating cells; + - routing mode `geo_unicast`; + - enabled, drained, withdrawn, and readiness state. +- Allow one edge to expose, for example: + - one IPv4/IPv6 pair for three shared cells; + - another pair for one reserved customer pool; + - another pair for quarantine. +- Publish only ready, non-withdrawn pool endpoints into system-managed DNS. +- Preserve country, continent, and global fallback behavior. +- Keep management addresses separate from service endpoints. +- Prevent duplicate/conflicting address ownership. +- Reconcile gateway bindings, pool readiness, and DNS publication as one + revisioned workflow with last-valid state. -### Outcome - -Safe reverse proxying through signed, data-driven OpenResty runtime state with stable baseline pool placement. - -### Scope +### Scale checkpoint -- One explicit origin per proxied hostname. -- Origin safety, forwarding-header normalization, timeouts, retries, WebSocket behavior, and health tests. -- Shared, quarantine, and exceptional dedicated pools. -- Edge registration, one-time bootstrap, mTLS identity, revocation, and rotation. -- Signed deltas and bounded full snapshots. -- Atomic activation, acknowledgement, previous-valid retention, and offline serving. -- Target-first placement and source drain. -- Generic OpenResty configuration without per-domain reload. +- Multiple pools and service pairs on each of at least two edges. +- Measured DNS publication and gateway reconciliation after endpoint health + changes without rewriting every domain. ### Completion checklist -- [ ] Unsafe origins and proxy loops are rejected at save and connect time. -- [ ] Agent enrollment, rotation, revocation, and replay limits pass. -- [ ] Corrupt, unsigned, oversized, incompatible, and obsolete artifacts are rejected. -- [ ] A fresh edge restores from a bounded full snapshot. -- [ ] Control-plane, queue, and network interruption do not stop existing traffic. -- [ ] One cell/domain failure does not stop an unrelated cell/domain. -- [ ] Real HTTP/HTTPS, IPv4/IPv6, restart, drain, migration, and rollback tests pass. -- [ ] Documentation and browser qualification are current. - -## Phase 5 — TLS, baseline cache, and purge - -### Outcome +- [ ] Three shared cells serve one IPv4/IPv6 pair on one edge. +- [ ] A reserved customer pool serves a different pair on the same edge. +- [ ] DNS returns the correct ready endpoint set for each pool. +- [ ] Withdrawal of one endpoint does not change unrelated pools. +- [ ] IPv4-only, IPv6-only, and dual-stack endpoints behave correctly. +- [ ] Address conflicts and invalid bindings fail before activation. +- [ ] Gateway, DNS, placement, and cell state converge after restart/reconcile. +- [ ] Tests, scale evidence, docs, and manual qualification pass. -Recoverable managed/custom TLS and deterministic basic caching with durable bounded purge delivery. +## Phase 5 — Simple Anycast pools -### Scope +### Goal -- Managed ACME DNS-01 only for eligible proxied domains. -- Renewal spreading, retries, cleanup, certificate reuse, and expiry alerts. -- Validated encrypted custom certificate upload. -- Cache enablement, TTLs, object limits, origin-header behavior, query inclusion, cookie bypass, development mode, and stale-if-error. -- Epoch-based full purge and exact URL purge. -- Durable per-edge purge tasks with retry and status. +Allow selected pools to use one shared IPv4/IPv6 pair across multiple POPs while +keeping route announcement outside CDNFoundry. -### Completion checklist - -- [ ] Managed and custom TLS lifecycle passes with no private-key exposure. -- [ ] Wrong key, chain, name, expiry, and oversized PEM are rejected. -- [ ] Existing valid certificates continue during ACME failure. -- [ ] Cache MISS/HIT/BYPASS/STALE behavior matches the deterministic key. -- [ ] URL and full purge reach every eligible target and retry safely. -- [ ] Restart and failed-delivery tests preserve serving and task state. -- [ ] Public HTTPS, documentation, and browser qualification are current. - -## Phase 6 — Security and DDoS readiness +### Implementation -### Outcome +- Add routing mode `simple_anycast` to eligible pools. +- Store one pool-level IPv4 and optional IPv6 service pair. +- Attach explicit participating POPs/edges and readiness requirements. +- Bind the same service pair on every participating edge gateway. +- Publish the same Anycast pair for domains assigned to the pool. +- Expose a clear readiness/withdrawal signal for the operator or upstream routing + system. +- Document that the network/provider owns BGP advertisement and withdrawal. +- Do not add FRR, BIRD, router credentials, arbitrary routing commands, or a BGP + control plane to CDNFoundry. +- Preserve Geo-Unicast pools on the same fleet. -Bounded early rejection, application protection, quarantine, and emergency controls without pretending to provide upstream scrubbing. +### Scale and network checkpoint -### Scope - -- Ordered IP, CIDR, country, and continent allow/block rules. -- Standard, protected, quarantine, and bounded manual profiles. -- Client/domain request, connection, TLS, body, header, timeout, origin, and cache-admission ceilings. -- Trusted proxy handling. -- Restrict, quarantine, recover, and release states. -- Expiring emergency actions for domains, cells, edges, and pools. -- Pool service-address withdrawal from DNS. -- Security telemetry with stable reason codes. +- At least two real or approved lab POPs using the same IPv4/IPv6 pair. +- Traffic observations from multiple external vantage points. +- Controlled POP loss and restoration with route behavior recorded by the + operator/provider. +- Gateway and cell capacity limits recorded independently per POP. ### Completion checklist -- [ ] Rule order, permissions, imports, limits, IPv4, and IPv6 pass. -- [ ] Unknown Host/SNI and malformed traffic are rejected before expensive work. -- [ ] Quarantine migration is target-first and recoverable. -- [ ] Emergency actions persist across restart and expire safely. -- [ ] One attacked domain cannot exhaust unrelated cell resource budgets in qualification. -- [ ] Physical-uplink saturation limitations remain explicit. -- [ ] Security docs, runbooks, metrics, alerts, and browser qualification are current. - -## Phase 7 — Telemetry, analytics, and usage export - -### Outcome - -Direct bounded telemetry and accurate operational analytics independent from serving. - -### Scope - -- OpenResty and DNS structured telemetry through Vector directly to ClickHouse. -- Bounded disk buffers, retries, drops, retention, and redaction. -- Request, DNS, cache, origin, TLS, security, deployment, edge, cell, and pool fields. -- Domain-scoped and administrator-global analytics. -- Bounded raw-log access. -- Idempotent PostgreSQL usage rollups and stable JSON/CSV exports. -- Partial/outage labeling. +- [ ] One Anycast pool uses the same service pair on multiple POPs. +- [ ] Geo-Unicast and Anycast pools coexist on the same edges. +- [ ] CDNFoundry exposes honest ready/degraded/withdrawn state. +- [ ] POP failure does not corrupt another POP's local serving state. +- [ ] External route withdrawal/restoration is qualified and documented. +- [ ] UI/docs never claim CDNFoundry itself announces BGP routes. +- [ ] Physical-uplink and upstream-scrubbing limitations remain explicit. +- [ ] Tests, network evidence, docs, and manual qualification pass. + +## Phase 6 — Cache v2 storage and cache-key policy + +### Goal + +Turn the existing basic cache into a persistent, bounded, production-strength +cell cache without adding per-domain cache directories or a distributed cache. + +### Implementation + +- Use persistent per-cell cache volumes with explicit maximum size, inactive + duration, temporary-storage quota, and minimum-free-space policy. +- Add a small set of pool resource profiles such as `small`, `standard`, `large`, + and `streaming`. +- Keep domain placement stable so a domain normally reaches one cell/cache on an + edge. +- Add bounded cache-key query policies: + - include all parameters; + - ignore all parameters; + - include selected names; + - ignore selected names. +- Preserve deterministic Host, path, scheme, query, and cache-epoch behavior. +- Add bounded TTL policy for approved status codes such as 200/206, redirects, + and optional short negative caching. +- Add cache admission protection: + - per-domain admission rate; + - cache-key and query-variant limits; + - maximum cacheable object size; + - optional minimum cacheable object size; + - low-disk bypass; + - range and high-cardinality protections. +- Add explicit stale modes: + - off; + - stale-if-error; + - stale-while-revalidate; + - cache-only emergency; + - stale-only emergency. +- Preserve epoch full purge and exact URL purge. ### Scale checkpoint -- At least 20,000 domains with active analytics. -- Bounded query ranges, filters, result sizes, and execution limits. -- Controlled ClickHouse outage and backlog recovery. +- Cache-hit and cache-miss load at declared object-size distributions. +- Cache quota pressure, low-disk behavior, restart persistence, purge fan-out, + and high-cardinality abuse. +- Record throughput, latency, disk IOPS, disk usage, memory, CPU, and hit ratio. ### Completion checklist -- [ ] Raw traffic never passes through Laravel queues or PostgreSQL. -- [ ] Secrets, request bodies, tokens, cookies, and certificate material are absent. -- [ ] Generated traffic matches request, byte, cache, origin, security, and DNS totals. -- [ ] ClickHouse/Vector outage does not affect serving. -- [ ] Backlog recovery does not starve live traffic. -- [ ] Usage rebuild and export are stable and idempotent. -- [ ] Documentation and browser qualification are current. - -## Phase 8 — Operations, recovery, and baseline release qualification - -### Outcome +- [ ] Cache survives routine cell restart and remains rebuildable after loss. +- [ ] Pool profiles enforce disk/temp/object/admission ceilings. +- [ ] Query policies produce deterministic keys without unbounded variants. +- [ ] TTL and status-code behavior match configured policy. +- [ ] Stale modes behave correctly during origin failure. +- [ ] Low disk and cache abuse bypass safely without filling the host. +- [ ] URL/full purge remains durable and bounded across participating cells. +- [ ] One domain cannot evict or fill unrelated cell caches beyond accepted limits. +- [ ] Tests, load evidence, docs, and manual qualification pass. + +## Phase 7 — Gzip, Brotli, and compressed delivery + +### Goal + +Reduce customer bandwidth with safe response compression integrated with Cache +v2 and bounded by cell/pool resources. + +### Implementation + +- Store one canonical uncompressed cache object by default. +- Request identity encoding from origins where required for deterministic cache + behavior. +- Enable Gzip as the default broadly compatible compression method. +- Add optional Brotli through one immutable, pinned, tested edge image/module. +- Expose simple profiles only: + - `off`; + - `standard`; + - `maximum_savings` for reserved/dedicated pools. +- Use a tested MIME-type allowlist and minimum response size. +- Avoid recompressing JPEG, PNG, WebP, AVIF, video, archives, and other already + compressed formats. +- Handle `Accept-Encoding`, `Vary`, ETag/revalidation, HEAD, 304, stale, purge, + and identity fallback correctly. +- Disable or bound on-the-fly compression for range traffic and large responses. +- Add per-cell compression concurrency, CPU-pressure fallback, and emergency + disable behavior. +- Emit encoding, origin bytes, uncompressed bytes, served bytes, compression + ratio, profile, and fallback telemetry. -A deployable, observable, recoverable baseline platform with measured release evidence. - -### Scope - -- Component health, queue health, drift, capacity, alerts, and operations. -- Bounded global reconciliation. -- Failed-job and failed-operation inspection/retry. -- Encrypted off-host backup and authenticated restore preflight. -- Clean-host restore. -- Expand/contract migrations, canary upgrade, and rollback. -- Clock synchronization and drift monitoring. -- Capacity planning and failure runbooks. - -### Completion checklist - -- [ ] Existing DNS and edge traffic continues during control-plane and ClickHouse outage. -- [ ] Encrypted backup and clean replacement-host restore pass. -- [ ] PowerDNS and edge state rebuild pass. -- [ ] Queue loss is repaired by reconciliation. -- [ ] Canary upgrade and rollback pass without database restore. -- [ ] RPO, RTO, topology, hardware, and measured throughput are recorded. -- [ ] Production installation, upgrade, backup, recovery, monitoring, and capacity documentation are current. -- [ ] Baseline release browser qualification is recorded. - -## Phase 9 — Edge gateway and bounded cell inventory - -### Outcome - -Each edge has one minimal public gateway and a bounded inventory of pre-created generic OpenResty cell slots. - -### Scope - -- One edge gateway container/process group. -- Fixed configurable cell-slot count created during edge installation. -- Stable cell identities independent from pool names. -- Gateway listeners on one or more service IPv4/IPv6 pairs. -- Host routing for HTTP and SNI preread routing for HTTPS. -- Trusted client-address preservation to cells. -- Unknown destination, Host, and SNI rejection. -- Signed/revisioned gateway maps with atomic activation and last-valid rollback. -- Agent discovery and reporting of all installed cell slots. -- No Docker socket in the normal edge agent. - -### Scale and failure checkpoint +### Scale checkpoint -- Maximum declared service IP pairs, host mappings, cells, and connections per edge. -- Gateway restart and invalid-map rejection. -- One cell restart without gateway or unrelated-cell interruption. -- Gateway CPU/memory reservation under full cell load. +- Mixed identity/Gzip/Brotli clients against cache HIT and MISS traffic. +- Compressible and non-compressible object distributions. +- CPU saturation, concurrency limit, fallback, and large-response behavior. +- Record bandwidth saved, latency, throughput, and CPU cost per profile. ### Completion checklist -- [ ] Installation creates exactly the configured bounded slots. -- [ ] Agent registers inventory without dynamically creating containers. -- [ ] Gateway routes HTTP Host and TLS SNI to the assigned cell. -- [ ] Real client IP reaches the trusted cell runtime. -- [ ] Unknown or mismatched destination/Host/SNI is rejected. -- [ ] Invalid gateway state preserves the previous map. -- [ ] Gateway and cell health, saturation, revisions, and metrics are visible. -- [ ] Compose, API, UI, docs, runbooks, E2E, scale, and browser qualification are current. +- [ ] Identity, Gzip, and Brotli responses decode to identical content. +- [ ] One canonical cached object serves different client encodings correctly. +- [ ] Already-compressed and range responses follow safe policy. +- [ ] `Vary`, ETag, revalidation, stale, and purge behavior remain correct. +- [ ] Shared pools cannot select unsafe compression levels. +- [ ] CPU pressure falls back or disables compression without stopping traffic. +- [ ] Compression telemetry and bandwidth-savings analytics are accurate. +- [ ] Tests, load evidence, docs, and manual qualification pass. -## Phase 10 — Multi-cell pools, endpoints, and stable domain placement +## Phase 8 — Primary and backup origin failover -### Outcome +### Goal -A pool can use multiple cells on each participating edge and one service IP pair can front those cells. +Remove the single-origin availability weakness with one simple active-passive +backup origin per proxied hostname. -### Scope +### Implementation -- Pool kinds: shared, reserved, dedicated, and quarantine. -- Explicit participating edges. -- One or more cell slots per pool per edge. -- Pool service endpoints with IPv4/IPv6 ownership separate from individual cells. -- Stable domain-to-cell placement inside a pool. -- Optional explicitly bounded replicated placement for exceptional high-capacity domains. -- Different service IP pairs for different pools on the same edge. -- Target-first domain and cell migration. -- Selective artifact/certificate delivery only to participating cells and migration targets. -- Minimum-ready-cell policy per pool and edge. -- Slot drain, release, reassignment, and capacity reservation. +- Support one primary and one optional backup origin. +- Reuse the same strict origin-address, TLS, header, timeout, and loop validation. +- Use bounded health checks and request-path failure evidence. +- Add failure threshold, recovery threshold, hold-down, and failback delay. +- Keep active origin state local to cells and last-valid during control-plane loss. +- Fail over without a Laravel request-path call. +- Use stale/cache-only behavior before unnecessary origin retry storms. +- Expose active origin, transition reason, timestamps, and health without secrets. +- Do not add weighted balancing, traffic percentages, geographic origin steering, + service discovery, or arbitrary origin pools. ### Scale checkpoint -- Declared maximum pools, endpoints, cells, domains, and mappings per edge. -- Rebalancing does not reshuffle unrelated domains. -- Adding/removing a cell does not require full-fleet artifact delivery. -- Shared, reserved, dedicated, and quarantine isolation load tests. +- Controlled primary failure and recovery under concurrent cache HIT/MISS traffic. +- Record failover time, failback time, origin connection pressure, errors, and + unrelated-domain impact. ### Completion checklist -- [ ] One IP pair fronts three shared cells successfully. -- [ ] A second IP pair fronts reserved customer cells on the same edge. -- [ ] Stable domain placement preserves cache locality. -- [ ] Dedicated pools enforce the single-domain contract. -- [ ] Reserved pools accept only explicitly assigned domains. -- [ ] Cell shortage and minimum shared/quarantine reservations fail safely. -- [ ] Only participating cells receive domain certificates and artifacts. -- [ ] Migration, drain, rollback, scale, docs, UI, API, and browser qualification pass. - -## Phase 11 — Geo-Unicast and Simple Anycast routing modes - -### Outcome - -Pools support either normal Geo-Unicast endpoints or externally routed Simple Anycast without introducing BGP control into Laravel. - -### Scope - -- `geo_unicast` and `simple_anycast` pool routing modes. -- Geo-Unicast publishes ready per-edge pool endpoints. -- Simple Anycast uses one shared IPv4 and one shared IPv6 across participating POPs. -- Anycast disables GeoDNS edge selection for the pool. -- CDNFoundry manages desired membership, configuration, TLS, purge, security, revision, and readiness. -- Provider/network tooling manages BGP, FRR, BIRD, route announcement, and withdrawal. -- Agent reports service-address presence, gateway readiness, cell readiness, active revision, and external route-advertised signal. -- A POP is ready only after required runtime state is active. -- A failed POP is excluded only after route withdrawal is confirmed. -- Shared Anycast service addresses are forbidden as origins. +- [ ] Healthy primary receives normal origin traffic. +- [ ] Qualified primary failure moves traffic to backup within policy. +- [ ] Recovery uses hysteresis and does not flap. +- [ ] Both-origin failure follows stale/cache-only/maintenance policy. +- [ ] Invalid backup configuration never replaces valid primary state. +- [ ] Control-plane outage does not remove local failover behavior. +- [ ] One failing origin cannot exhaust unrelated origin budgets. +- [ ] Tests, load evidence, docs, and manual qualification pass. + +## Phase 9 — Managed OWASP CRS WAF + +### Goal + +Add optional managed application-signature protection without building a custom +WAF language or exposing raw ModSecurity configuration. + +### Implementation + +- Use a pinned ModSecurity v3 and OWASP Core Rule Set release. +- Build one immutable WAF-capable OpenResty cell image/profile. +- Support simple domain/pool profiles: + - `off`; + - `monitor`; + - `balanced`; + - `strict`. +- Map profiles to tested anomaly thresholds, paranoia levels, body-inspection + limits, and blocking behavior. +- Prefer WAF-capable reserved, dedicated, or quarantine cells where isolation is + required; do not force WAF overhead onto every normal pool. +- Support only bounded exclusions by approved dimensions such as rule ID, + hostname, path prefix, argument/header name, reason, owner, and expiry. +- Never expose arbitrary `SecRule`, arbitrary directives, customer rule uploads, + runtime downloads, or custom expression languages. +- Add detection/block reason, rule ID, anomaly score, processing time, body-limit, + exclusion, and profile telemetry with redaction. +- Roll new CRS/image versions through monitor-only canaries before blocking. +- Preserve the previous WAF image/ruleset and active traffic state on failure. + +### Scale and security checkpoint + +- Safe laboratory test corpus for common SQL injection, XSS, traversal, protocol, + and evasion patterns. +- False-positive corpus for representative applications. +- Cache HIT/MISS traffic with WAF enabled and disabled. +- Record latency, CPU, memory, request-body cost, throughput, detection accuracy, + false positives, and accepted limits. ### Completion checklist -- [ ] Geo-Unicast and Anycast modes cannot be ambiguously combined. -- [ ] Every Anycast POP activates the same required domain revision before ready. -- [ ] DNS returns the shared Anycast addresses without country/continent edge selection. -- [ ] Route withdrawal signal prevents unsafe POP exclusion ordering. -- [ ] One POP failure leaves healthy POPs serving. -- [ ] Control-plane outage does not stop externally announced valid POPs. -- [ ] BGP credentials and neighbor configuration are absent from CDNFoundry. -- [ ] External multi-POP, IPv4/IPv6, failure, scale, docs, and browser qualification pass. - -## Phase 12 — Cache v2 and response compression - -### Outcome - -A persistent, bounded, observable cache with correct variants, origin protection, Gzip, and optional Brotli. - -### Scope - -- Persistent per-cell cache storage with explicit maximum size, inactive period, minimum free disk, temporary-storage quota, and eviction visibility. -- Small, standard, large, and streaming pool cache resource profiles. -- Canonical uncompressed cached objects. -- Gzip enabled by default for approved compressible MIME types. -- Optional Brotli from an immutable tested edge image. -- Off, standard, and maximum-savings compression profiles. -- Minimum/maximum compressible response size. -- CPU/concurrency ceilings and emergency compression disable. -- No compression for already-compressed, range/video, or excluded content. -- Correct `Vary: Accept-Encoding`. -- Query policy: all, none, include-list, or ignore-list. -- Bounded cookie bypass and cache variants. -- Status-code TTL policy for explicitly supported responses. -- Cache lock/request collapsing, admission limits, stale-if-error, and stale-while-revalidate. -- Exact URL and full epoch purge preserved. -- Compression and bandwidth-savings telemetry. - -### Scale and failure checkpoint - -- Cache HIT load with identity, Gzip, and Brotli clients. -- CPU saturation and fallback behavior. -- Disk-full, minimum-free-space, restart, corrupt-cache, and eviction behavior. -- High-cardinality URL/query abuse. -- Large-object and range-request qualification. - -### Completion checklist - -- [ ] Identical decoded content is served for identity, Gzip, and Brotli. -- [ ] Cache key does not fragment unintentionally by `Accept-Encoding`. -- [ ] MIME, size, range, and precompressed exclusions work. -- [ ] CPU pressure remains bounded and can disable Brotli/compression safely. -- [ ] Query and cookie variants remain within configured limits. -- [ ] Disk pressure cannot exhaust the host or unrelated cells. -- [ ] MISS/HIT/STALE/revalidate/purge behavior remains correct with compression. -- [ ] Metrics report origin bytes, uncompressed bytes, served bytes, encoding, and savings. -- [ ] UI, API, docs, troubleshooting, load tests, and browser qualification pass. - -## Phase 13 — Simple origin resilience - -### Outcome +- [ ] Off, monitor, balanced, and strict profiles behave as documented. +- [ ] Monitor records detections without blocking. +- [ ] Blocking profiles return stable reasons and preserve privacy. +- [ ] Bounded exclusions work, expire, audit, and cannot become arbitrary rules. +- [ ] Oversized or malformed bodies fail according to policy without exhausting cells. +- [ ] WAF failure/canary regression does not replace the previous valid runtime. +- [ ] Unrelated non-WAF pools remain healthy during WAF load/failure. +- [ ] Tests, security evidence, docs, and manual qualification pass. + +## Phase 10 — Expanded telemetry, analytics, and capacity control + +### Goal + +Make the new gateway, endpoint, pool, cell, cache, compression, failover, Anycast, +and WAF behavior observable and capacity-manageable. + +### Implementation + +- Add bounded telemetry for: + - gateway listeners, maps, routes, connections, errors, and revisions; + - service endpoints and Anycast readiness; + - pool participation, minimum readiness, and placement transitions; + - cell CPU, memory, connections, cache disk/temp, admission, and saturation; + - compression encoding, ratios, bytes saved, concurrency, and fallback; + - origin health, active origin, failover/failback, and circuit state; + - WAF profile, anomaly score, rule category, processing time, and action. +- Extend ClickHouse schemas and aggregates with bounded retention and query limits. +- Update administrator and domain analytics only where the data is useful. +- Add Prometheus alerts for stale maps, endpoint mismatch, cell exhaustion, cache + disk pressure, compression CPU pressure, origin failover, WAF errors, and + Anycast readiness disagreement. +- Keep telemetry best-effort and outside serving decisions. -A proxied hostname can use one primary and one backup origin with active-passive failover only. - -### Scope +### Scale checkpoint -- Primary and optional backup origin. -- Same safety validation for both origins. -- Bounded active and passive health evidence. -- Explicit failover and recovery thresholds. -- Last-known-good origin state. -- No weighted traffic, percentage splitting, Geo origin steering, discovery, or arbitrary origin pools. -- Origin selection remains local to the cell and independent from Laravel availability. -- Failover/recovery telemetry and audit. +- At least 20,000 active proxied domains across multiple pools, cells, endpoints, + and edges. +- High-cardinality fields remain bounded or aggregated. +- ClickHouse/Vector outage and backlog recovery under live traffic. ### Completion checklist -- [ ] Healthy primary receives normal traffic. -- [ ] Qualified primary failure moves traffic to backup. -- [ ] Flapping is bounded by thresholds and recovery delay. -- [ ] Backup failure cannot create an unbounded retry loop. -- [ ] Primary recovery is controlled and observable. -- [ ] Cache correctness is preserved across origin change. -- [ ] Both origins are protected against private/loop/platform destinations. -- [ ] Real failure, recovery, restart, scale, docs, and browser qualification pass. +- [ ] Every new serving component has healthy/degraded/unavailable state. +- [ ] Metrics and logs identify the responsible pool, cell, edge, and revision. +- [ ] Domain users cannot see unrelated pool/customer data. +- [ ] Raw logs remain redacted, bounded, and directly delivered to ClickHouse. +- [ ] Telemetry outage never blocks gateway/cell traffic. +- [ ] Queries remain bounded and responsive at the qualification dataset. +- [ ] Alerts and runbooks identify actionable recovery steps. +- [ ] Tests, scale evidence, docs, and manual qualification pass. -## Phase 14 — Managed OWASP CRS WAF +## Phase 11 — Bounded fleet rollout automation -### Outcome +### Goal -Optional managed application-signature protection using a pinned OWASP Core Rule Set without a customer rule language. +Automate proven manual upgrades when fleet size makes per-edge rollout +inefficient, without adding general remote execution or dynamic containers. -### Scope +### Implementation -- Immutable WAF-capable OpenResty cell image. -- Pinned tested ModSecurity v3 and OWASP CRS release. -- Profiles: off, monitor, balanced, and strict. -- Detection/anomaly scoring, bounded request-body inspection, and explicit response-body policy. -- WAF-capable pool/cell resource profile. -- Bounded exclusions by rule ID plus hostname, path prefix, or argument/header name. -- Expiry, administrator note, audit, and limit for exclusions. -- No raw ModSecurity directives, SecRule editor, online rule download, uploaded scripts, or plugin system. -- Monitor-first rollout, canary image/ruleset upgrade, automatic rollout pause, and last-compatible rollback. -- Rule, anomaly, action, latency, and false-positive telemetry. +- Manage immutable versions for gateway, edge agent, normal cell, and WAF cell. +- Define compatibility ranges and a bounded mixed-version window. +- Support explicit canary edges/POPs and rollout waves. +- Pause automatically on health, error, readiness, revision, or capacity thresholds. +- Roll back to the last compatible image/configuration. +- Preserve fixed cell-slot topology; rollout automation does not create arbitrary + containers or run arbitrary commands. +- Expose desired/current version, wave, progress, failure, pause, and rollback. +- Audit every rollout decision and retain operator confirmation for destructive + or fleet-wide actions. -### Scale and failure checkpoint +### Scale checkpoint -- Baseline and attack-pattern throughput comparison. -- CPU, memory, latency, body-size, multipart, and rule-count bounds. -- Monitor/block behavior and false-positive workflow. -- WAF crash or invalid ruleset cannot stop normal non-WAF pools. +- Multi-edge, multi-POP rollout with normal and WAF cells. +- Mixed-version serving, controlled canary failure, automatic pause, and rollback. +- Record rollout time, unavailable capacity, errors, and operator recovery steps. ### Completion checklist -- [ ] Monitor mode never blocks but records bounded sanitized events. -- [ ] Balanced and strict presets block qualified test cases. -- [ ] Exclusions are narrow, bounded, audited, and expire correctly. -- [ ] Invalid ruleset/image preserves the previous valid WAF runtime. -- [ ] WAF resource exhaustion is isolated to assigned WAF cells. -- [ ] New CRS rollout can pause and roll back safely. -- [ ] Security docs clearly separate WAF, DDoS readiness, and volumetric limits. -- [ ] API, UI, runbooks, E2E, load, and browser qualification pass. - -# Part Two — Bounded future roadmap - -Part Two begins only after Part One is running successfully and a real operator or customer requirement is documented. These phases are candidates, not promises. - -## Phase 15 — Fleet rollout automation and extended recovery - -### Candidate scope - -- Canary groups and rollout waves for gateway, agent, cell, and WAF images. -- Automatic pause on health, rejection, drift, or error thresholds. -- Automatic rollback to the last compatible image. -- Fleet compatibility reporting. -- Immutable/deletion-protected backup storage. -- Scheduled isolated restore exercises. -- Optional warm control-plane standby. - -### Boundary - -This automates already-proven manual procedures. It does not add a general orchestration platform, dynamic unbounded containers, Kubernetes requirement, or traffic dependency on the standby. - -### Admission and completion gate +- [ ] Canary completes before later waves start. +- [ ] Failed canary pauses rollout automatically. +- [ ] Rollback restores the prior compatible runtime without database restore. +- [ ] Existing traffic continues through a bounded mixed-version window. +- [ ] No arbitrary command execution or unbounded container creation exists. +- [ ] Version drift and incompatible agents/cells are visible. +- [ ] Tests, recovery evidence, docs, and manual qualification pass. + +## Phase 12 — Final production qualification for the new architecture + +### Goal + +Prove the complete post-baseline architecture works as one simple, solid, +recoverable product. + +### Required topology + +- At least two POPs/edges. +- At least eight bounded cell slots per edge. +- One shared pool using at least three cells per edge. +- One reserved customer pool using a separate IPv4/IPv6 pair. +- One quarantine pool. +- Geo-Unicast service endpoints. +- One Simple Anycast pool across both POPs where the operator can provide the + required routing environment. +- Persistent Cache v2, Gzip, Brotli, primary/backup origins, and one managed WAF + pool/profile. +- Real IPv4 and IPv6 clients/origins where available. + +### Final qualification + +- Complete new-edge installation and registration from clean hosts. +- Create and activate all pool types and endpoint modes. +- Serve real HTTP/HTTPS through multiple service pairs on the same edge. +- Verify stable domain placement, movement, drain, quarantine, and rollback. +- Verify Geo-Unicast and Simple Anycast behavior from external vantage points. +- Exercise Cache v2, persistent restart, purge, stale, quota pressure, Gzip, and Brotli. +- Fail the primary origin and prove controlled backup failover/failback. +- Exercise WAF monitor/block/exclusion/canary behavior with safe test traffic. +- Stop Laravel, queues, Redis/Valkey, and ClickHouse while existing traffic continues. +- Restart gateway, agent, cells, Vector, ClickHouse, DNSdist, and PowerDNS according + to their runbooks. +- Apply invalid gateway/cell/WAF artifacts and verify previous valid state remains. +- Saturate one cell within the approved lab and verify unrelated cells/pools continue. +- Perform fleet canary upgrade and rollback. +- Restore control-plane data on a clean replacement host and reconcile derived state. +- Record measured limits, hardware, topology, RPO, RTO, throughput, latency, + saturation, known limitations, and owner browser evidence. -- [ ] Fleet size makes manual rollout measurably inefficient. -- [ ] Manual canary/rollback and clean restore already pass. -- [ ] Automation has bounded waves, pause, rollback, audit, and operator override. -- [ ] DNS and HTTP traffic remain independent when automation/standby is unavailable. -- [ ] Real multi-host qualification and documentation pass before release. - -## Phase 16 — Protocol, DNS, certificate, private-origin, and data extensions - -### Candidate capabilities - -Admit separately, never as one combined project: - -- HTTP/3 and QUIC. -- Secondary ACME certificate authority. -- DNSSEC signing, DS lifecycle, rollover, and recovery. -- Focused outbound private-origin connector. -- Longer-retention analytics archive/export. -- Additional proven placement policy such as region restriction or maintenance evacuation. - -### Explicitly outside product direction - -- Serverless workers. -- Object-storage product. -- General VPN or zero-trust suite. -- Arbitrary tunnels. -- Custom WAF or routing expression language. -- CAPTCHA or browser-challenge platform. -- Bot-scoring product. -- Billing/subscription engine. -- Reseller/organization hierarchy. -- Microservices, service mesh, Kafka, or Kubernetes requirement. - -### Admission gate for every capability - -- [ ] A real repeated requirement and measurable acceptance criteria exist. -- [ ] Part One cannot solve it safely. -- [ ] Typed bounded desired state and authorization are defined. -- [ ] Runtime remains independent from Laravel availability. -- [ ] Failure, disablement, rollback, compatibility, observability, and recovery are defined. -- [ ] Real-runtime, scale, documentation, and browser qualification are written before implementation is declared complete. - -# Final release contract - -CDNFoundry is release-qualified only when: +### Completion checklist -- every admitted Part One phase has passed its completion gate; -- all current manual browser checkpoints are recorded; -- external DNS, IPv4/IPv6, public HTTPS, multi-edge, multi-cell, Anycast, cache/compression, origin-failure, WAF, outage, scale, backup, restore, upgrade, and rollback evidence is available where applicable; -- current documentation describes implemented behavior rather than plans; -- no unexecuted test is represented as passed; -- no project boundary has been weakened to complete a phase. +- [ ] Every phase completion gate is passed and linked to evidence. +- [ ] Existing completed baseline features pass the regression smoke suite. +- [ ] All current API/OpenAPI, UI, deployment, architecture, security, operations, + troubleshooting, and runbook documentation is updated. +- [ ] All tests clearly report passed, failed, blocked, and not executed results. +- [ ] No unresolved critical/high failure remains. +- [ ] The owner records the final manual/browser/real-traffic qualification. +- [ ] The release notes state measured capabilities and limitations without + unsupported scale, Anycast, WAF, or DDoS claims. + +# Future candidates — not part of the committed phases + +These capabilities remain outside the committed roadmap until repeated customer +or operator demand justifies a separate scope and qualification contract: + +- secondary ACME certificate authority; +- DNSSEC signing and rollover lifecycle; +- HTTP/3 and QUIC; +- private outbound origin connector for non-public origins; +- immutable/deletion-protected backup storage and warm control-plane standby; +- long-retention analytics archive/export; +- replicated placement for exceptional high-volume domains; +- additional placement policies that preserve the bounded pool/cell model; +- origin shield or hierarchical cache. + +A candidate is admitted only when: + +- a real requirement exists; +- the existing product cannot solve it safely; +- operational and failure costs are understood; +- it does not move traffic through Laravel; +- it does not require rewriting the control plane or generic edge runtime; +- it has bounded state, rollback, observability, tests, docs, and a release gate; +- disabling it leaves the committed product functional. + +# Explicitly out of scope + +- Weighted origin balancing and percentage traffic splitting +- General BGP/router management +- Volumetric DDoS scrubbing guarantees +- CAPTCHA, browser challenges, or bot-scoring platforms +- Customer-written WAF rules or edge scripts +- Serverless workers or plugin marketplaces +- Object-storage product features +- Per-domain containers/processes by default +- Kubernetes as a deployment requirement +- Billing, payment, reseller, organization, team, or custom-role systems + +## Final rule + +> Keep the completed platform intact. Add one bounded production capability per +> phase. Keep the gateway simple, keep cells isolated, preserve last-valid state, +> test real traffic and failure behavior, record scale honestly, update every +> affected document, and never move DNS or HTTP traffic through Laravel. From d176e6cf0fe061a10b9df701998231be70ae5b53 Mon Sep 17 00:00:00 2001 From: vaheeD khoshnouD Date: Mon, 27 Jul 2026 14:55:14 +0330 Subject: [PATCH 4/7] docs: fix roadmap structure and anchors --- docs/roadmap.md | 1018 +++++++++++++++++++---------------------------- 1 file changed, 410 insertions(+), 608 deletions(-) diff --git a/docs/roadmap.md b/docs/roadmap.md index dd539b8..09f039c 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -1,674 +1,476 @@ --- title: CDNFoundry next development roadmap -description: Ordered post-baseline roadmap for the new edge, cache, origin-resilience, WAF, and fleet capabilities. +description: Post-baseline roadmap for the new edge, cache, origin resilience, WAF, and fleet capabilities. --- # CDNFoundry next development roadmap -## Current boundary - -The original production platform roadmap is complete and is not repeated here. -The existing Laravel control plane, authoritative DNS, Geo-DNS, proxy, TLS, -baseline cache and purge, security controls, telemetry, analytics, backup, and -operations behavior are the starting baseline for this roadmap. - -This document contains only the **new development work** agreed after completion -of that baseline. - -Every new phase must: - -- extend the existing product without rewriting it; -- remain deployable and testable when the phase is finished; -- preserve all previously working behavior; -- stay simple to develop and operate; -- use bounded resources and explicit failure behavior; -- include tests, scale evidence, documentation, and owner-run browser qualification. - -## Product direction - -CDNFoundry remains a small, solid private CDN for operators and ISPs. It is not -trying to become Cloudflare, Fastly, Akamai, or a general cloud platform. - -The next development cycle focuses on five practical outcomes: - -1. multiple isolated OpenResty cells on one edge; -2. multiple public IPv4/IPv6 service pairs on the same edge; -3. Geo-Unicast and Simple Anycast pool routing; -4. stronger cache, compression, origin resilience, and managed WAF behavior; -5. safe operation and upgrades of the expanded edge fleet. - -## Non-negotiable boundaries - -1. Laravel and Filament remain the single management/control plane. -2. PostgreSQL remains the durable desired-state source of truth. -3. DNS, HTTP, HTTPS, TLS selection, cache decisions, WAF decisions, and raw - telemetry never pass through Laravel. -4. The edge gateway is a small data-plane router. It does not cache, terminate - customer TLS, run WAF logic, call Laravel, or query a database. -5. OpenResty cells perform TLS, cache, security, WAF when enabled, origin proxy, - and request telemetry. -6. Cell containers are created as bounded generic slots during edge installation. - The edge agent assigns and configures them; it does not receive unrestricted - Docker or containerd access. -7. A domain does not receive a default container, process, worker, timer, cache - directory, server block, or reload. -8. Pools may use many cells on one edge. A public IPv4/IPv6 pair belongs to a - pool endpoint, not necessarily to one cell. -9. Simple Anycast uses operator/provider routing. CDNFoundry does not become a - BGP controller and does not require FRR, BIRD, or direct router access. -10. External effects remain asynchronous, revisioned, idempotent, coalesced, - acknowledged, and last-valid-state preserving. -11. No custom WAF language, arbitrary ModSecurity directives, user scripts, - plugin runtime, or downloaded runtime rules are introduced. -12. No microservices, Kafka, Kubernetes requirement, CQRS, event sourcing, - GraphQL, custom RBAC, reseller hierarchy, billing engine, or second dashboard. -13. No unlimited claims. Every scale result records topology, hardware, - concurrency, dataset, saturation point, and accepted limits. -14. Development PostgreSQL and named Compose volumes remain persistent. Tests - use the repository's isolated test environment and never destructively reset - persistent development data. -15. Roadmap phase names and numbers never appear in production class names, - filenames, routes, migrations, tables, or configuration keys. - -## Completion gate for every phase - -A phase is complete only when every applicable row is recorded independently. +## Starting point + +The original production roadmap is complete. The existing control plane, +authoritative DNS, Geo-DNS, proxy, TLS, baseline cache and purge, security, +telemetry, analytics, backup, and operations are the regression baseline. + +This roadmap contains only the new work discussed after that baseline. + +CDNFoundry remains a simple but solid private CDN. It must stay understandable, +bounded, production-safe, and easy to operate. It is not intended to become a +general cloud platform or a Cloudflare replacement. + +## Project boundaries + +- Laravel and Filament remain the single control plane. +- PostgreSQL remains the desired-state source of truth. +- DNS and HTTP traffic never pass through Laravel. +- The edge gateway only binds service addresses and routes by destination IP, + Host, and TLS SNI. +- OpenResty cells own TLS, cache, compression, security, WAF, origin proxying, + and request telemetry. +- Cell slots are bounded and created during edge installation. +- The edge agent never receives unrestricted Docker access. +- No default per-domain process, container, worker, timer, cache directory, + server block, or reload is allowed. +- External effects remain asynchronous, revisioned, idempotent, coalesced, + acknowledged, and last-valid-state preserving. +- No microservices, Kafka, Kubernetes requirement, CQRS, event sourcing, + GraphQL, custom RBAC, plugin runtime, reseller hierarchy, billing engine, + custom WAF language, or second dashboard. +- Simple Anycast uses operator/provider routing. CDNFoundry does not become a + BGP controller. +- DDoS readiness never claims upstream volumetric scrubbing after physical + capacity is saturated. +- Roadmap phase numbers and lifecycle suffixes must not appear in production + code or filenames. + +## Required completion gate + +Every phase must independently record: | Gate | Required evidence | | --- | --- | -| Implementation | Durable state, migrations, policies, API, UI where required, jobs, reconciliation, audit, metrics, and rollback behavior | -| Unit and feature tests | Happy path, authorization, validation, bounds, idempotency, stable errors, and backward compatibility | -| Real-runtime E2E | Real HTTP/HTTPS/DNS/cache/TLS/WAF/runtime behavior; mocks alone are insufficient | -| IPv4 and IPv6 | Both families pass whenever the phase handles addressing or traffic | -| Scale checkpoint | Dataset, topology, hardware, concurrency, measured result, saturation point, and accepted limit | -| Failure and recovery | Restart, dependency outage, invalid candidate, retry, obsolete work, rollback, and last-valid state | +| Implementation | State, migrations, authorization, API, UI where needed, jobs, reconciliation, metrics, audit, and rollback | +| Unit and feature tests | Happy path, permissions, validation, bounds, idempotency, and stable errors | +| Real-runtime E2E | Real DNS, HTTP, HTTPS, TLS, cache, compression, WAF, restart, and failure behavior where applicable | +| IPv4 and IPv6 | Both families pass whenever the phase handles addresses or traffic | +| Scale | Dataset, topology, hardware, concurrency, result, saturation point, and accepted limit | +| Failure and recovery | Retry, obsolete work, invalid candidate, dependency outage, restart, rollback, and last-valid state | | Isolation | Failure or load in one domain, pool, cell, edge, or component does not unnecessarily affect unrelated traffic | -| Observability | Metrics, logs, reason codes, alerts, health, capacity, and partial/degraded state | -| Documentation | User/admin guide, API/OpenAPI, architecture, deployment, operations, troubleshooting, and runbooks | -| Manual qualification | Owner-run checkpoints in `docs/manual-browser-qualification.md` | -| Regression | Existing completed platform behavior remains functional | -| Release decision | Passed, failed, blocked, or deliberately removed from scope | +| Observability | Health, metrics, logs, alerts, capacity, degraded states, and stable reason codes | +| Documentation | User, administrator, API/OpenAPI, architecture, deployment, operations, troubleshooting, and runbooks | +| Manual qualification | Owner-run evidence from `docs/manual-browser-qualification.md` | +| Regression | The completed baseline continues to work | +| Release decision | Passed, failed, blocked, or removed from scope | -A checkbox cannot be marked complete when the related test was not executed. -Every phase must finish as a usable production increment; no phase may leave the -active serving path dependent on unfinished work from a later phase. +A phase is not complete when an applicable test was not executed. Every phase +must finish as a usable production increment and must not depend on unfinished +work from a later phase. -# Committed development roadmap +# Phase 1 — Edge gateway ingress -## Phase 1 — Edge gateway ingress - -### Goal - -Introduce one minimal gateway on every edge so one server can bind multiple +**Goal:** introduce one minimal gateway per edge so one server can bind several public IPv4/IPv6 service pairs and route traffic to bounded OpenResty cells. -### Implementation +**Implementation:** -- Add one gateway process/container per edge. -- Bind one or more operator-configured public IPv4 and IPv6 service addresses. -- Route HTTP by destination address and validated `Host`. +- Bind operator-configured IPv4 and IPv6 service addresses. +- Route HTTP by destination address and validated Host. - Route HTTPS by destination address and TLS SNI without terminating customer TLS. -- Forward the trusted client address to cells through a qualified PROXY protocol - or equivalent fixed internal contract. -- Reject unknown destination addresses, hosts, and SNI names early. -- Load a generated local routing map atomically. -- Keep and restore the previous valid map after invalid configuration or restart. -- Expose gateway readiness, active revision, listener state, connection totals, - routing failures, and per-endpoint health. -- Keep the gateway free of cache, certificates, origin logic, WAF rules, Lua - business logic, and control-plane network calls. - -### Scale checkpoint - -- At least 50,000 hostname/SNI mappings in one generated map. -- Multiple IPv4 and IPv6 service pairs on one edge. -- Measured HTTP and TLS pass-through throughput with gateway CPU, memory, - connections, latency, and saturation recorded. - -### Completion checklist - -- [ ] HTTP Host and HTTPS SNI route to the intended backend cell. -- [ ] IPv4 and IPv6 listeners pass real traffic. -- [ ] Unknown Host/SNI and unassigned destination addresses are rejected. -- [ ] Real client address reaches the cell through the trusted internal contract. -- [ ] Invalid or partial maps never replace the active map. -- [ ] Gateway restart preserves or reconstructs the last valid routing state. -- [ ] Gateway failure is visible and does not corrupt cell runtime state. -- [ ] Existing direct edge behavior remains available during controlled migration. -- [ ] Tests, scale evidence, metrics, alerts, docs, and manual qualification pass. - -## Phase 2 — Bounded cell inventory and edge installation - -### Goal - -Replace the fixed shared/quarantine runtime assumption with a bounded inventory -of generic OpenResty cell slots created during edge installation. - -### Implementation - -- Add an installation setting such as a bounded cell-slot count. -- Generate/start named generic slots such as `cell-01` through `cell-N`. -- Give every slot stable identity, internal ports, runtime path, cache path, - temporary path, status endpoint, and resource limits. -- Allow unused slots to remain unassigned and idle or stopped according to a - documented installation policy. -- Register the full static inventory with the edge agent. -- Remove readiness hardcoding tied to `shared-default`. +- Forward trusted client identity to cells through a fixed internal contract. +- Reject unknown addresses, Host values, and SNI names before expensive work. +- Activate generated routing maps atomically and retain the previous valid map. +- Expose listener, revision, route, connection, error, and readiness metrics. +- Keep cache, TLS certificates, WAF, origin logic, and control-plane calls out of + the gateway. + +**Scale target:** at least 50,000 Host/SNI mappings and multiple dual-stack +service pairs on one edge, with throughput, latency, CPU, memory, and saturation +recorded. + +**Completion checklist:** + +- [ ] Real HTTP Host and HTTPS SNI traffic reaches the intended cell. +- [ ] IPv4 and IPv6 pass. +- [ ] Unknown and invalid traffic is rejected. +- [ ] Invalid maps never replace active maps. +- [ ] Restart restores or rebuilds the last valid map. +- [ ] Existing baseline edge traffic remains functional during migration. +- [ ] Tests, scale, documentation, and manual qualification pass. + +# Phase 2 — Bounded cell inventory + +**Goal:** replace the fixed shared/quarantine assumption with a bounded set of +generic OpenResty cell slots created during edge installation. + +**Implementation:** + +- Configure a bounded slot count such as `cell-01` through `cell-N`. +- Give every slot stable identity, ports, runtime path, cache path, temporary + path, status endpoint, and resource limits. +- Register assigned, unassigned, ready, degraded, drained, and stopped states. +- Remove readiness logic tied to one hardcoded shared cell. - Let the agent configure, reload, drain, restart, and report existing slots. -- Do not mount the Docker socket into the edge agent. -- Keep agent resources separate from cell resource groups. -- Support safe cell image compatibility and last-valid runtime state. +- Keep agent resources separate from cell resources. +- Do not mount the container-engine socket into the agent. -### Scale checkpoint +**Scale target:** qualify at least eight slots on one edge and record idle and +active overhead per slot. -- Qualify at least 8 cell slots on one edge. -- Record idle overhead and active overhead per cell. -- Prove one saturated or crashed cell does not terminate the agent, gateway, or - unrelated cells. +**Completion checklist:** -### Completion checklist +- [ ] Fresh installation creates exactly the configured slots. +- [ ] Every slot has unique identity, paths, ports, health, and limits. +- [ ] One crashed or saturated cell does not stop the gateway, agent, or other cells. +- [ ] Cell restart and rollback preserve unrelated traffic. +- [ ] Cache, temporary, and log storage remain bounded. +- [ ] Existing enrollment, mTLS, and snapshot recovery still pass. +- [ ] Tests, scale, documentation, and manual qualification pass. -- [ ] Fresh edge installation creates exactly the configured bounded slots. -- [ ] Every slot has unique identity, paths, ports, health, and resource limits. -- [ ] Agent reports assigned, unassigned, ready, degraded, drained, and stopped states. -- [ ] Agent operates without unrestricted container-engine access. -- [ ] Cell restart and image rollback preserve unrelated traffic. -- [ ] Cache/temp/log storage cannot exceed configured quotas. -- [ ] Existing edge enrollment, mTLS, rotation, and snapshot recovery still pass. -- [ ] Tests, scale evidence, docs, and manual qualification pass. +# Phase 3 — Multi-cell pools and stable placement -## Phase 3 — Multi-cell pools and stable domain placement +**Goal:** allow one pool to use several cells on the same edge while preserving +cache locality and predictable isolation. -### Goal +**Implementation:** -Allow one pool to use multiple cells on the same edge while preserving stable -cache locality and predictable failure isolation. +- Support shared, reserved, dedicated, and quarantine pool kinds. +- Separate stable cell identity from pool identity. +- Add explicit edge participation and cell assignment per pool. +- Allow several cells from one edge in one pool. +- Add minimum-ready-cell and capacity policies. +- Place a normal domain on one stable cell per edge by default. +- Keep replicated placement exceptional and bounded. +- Move domains target-first, switch gateway routing, then drain the source. +- Deliver artifacts only to participating and migration-target cells. -### Implementation +**Scale target:** at least 20,000 domains across several cells and a controlled +burst of 10,000 placement-affecting changes without unnecessary reshuffling. -- Support pool kinds: - - `shared` for unrelated normal domains; - - `reserved` for one customer's or workload group's domains; - - `dedicated` for one exceptional domain; - - `quarantine` for attacked or unstable domains. -- Separate stable cell-slot identity from pool identity. -- Add explicit edge participation and cell assignment per pool. -- Allow multiple cells from one edge to belong to one pool. -- Add minimum-ready-cell and capacity policy per pool/edge participation. -- Place each normal domain on one stable active cell inside its pool by default. -- Allow exceptional replicated placement only as an explicit bounded mode. -- Move domains target-first: configure target, verify readiness, switch gateway, - then drain and remove source state. -- Deliver artifacts only to active and target participating cells/edges. -- Prevent one dedicated pool from accepting multiple domains. -- Preserve previous placement and gateway state after failed migration. - -### Scale checkpoint - -- At least 20,000 domains distributed across multiple cells without unnecessary - reshuffling after adding a domain or a new cell. -- Controlled burst of at least 10,000 placement-affecting changes with coalescing. -- Record compiler time, artifact size, gateway-map activation time, and database - query behavior. - -### Completion checklist +**Completion checklist:** - [ ] One shared pool uses at least three cells on one edge. -- [ ] Domain placement remains stable across unrelated changes. -- [ ] Reserved, dedicated, and quarantine constraints are enforced. -- [ ] Failed target readiness leaves source placement active. -- [ ] Successful movement drains and removes the old assignment safely. -- [ ] Artifacts are not sent to non-participating edges or cells. -- [ ] Pool readiness counts all required cells rather than one arbitrary cell. -- [ ] Unrelated domain/cache traffic remains healthy during movement. -- [ ] Tests, scale evidence, docs, and manual qualification pass. - -## Phase 4 — Pool service endpoints and Geo-Unicast - -### Goal - -Support multiple public service IP pairs on one edge, with each pair serving one -pool endpoint backed by one or more cells. - -### Implementation - -- Add a pool endpoint per participating edge with: - - public IPv4; - - optional public IPv6; - - listener/gateway state; - - participating cells; - - routing mode `geo_unicast`; - - enabled, drained, withdrawn, and readiness state. -- Allow one edge to expose, for example: - - one IPv4/IPv6 pair for three shared cells; - - another pair for one reserved customer pool; - - another pair for quarantine. -- Publish only ready, non-withdrawn pool endpoints into system-managed DNS. -- Preserve country, continent, and global fallback behavior. +- [ ] Placement remains stable across unrelated changes. +- [ ] Reserved, dedicated, and quarantine rules are enforced. +- [ ] Failed target readiness keeps the source active. +- [ ] Successful movement removes old state only after the target is serving. +- [ ] Non-participating cells receive no artifacts. +- [ ] Pool readiness evaluates all required cells. +- [ ] Tests, scale, documentation, and manual qualification pass. + +# Phase 4 — Pool service endpoints and Geo-Unicast + +**Goal:** support several public service IP pairs on one edge, with each pair +owned by one pool endpoint backed by one or more cells. + +**Implementation:** + +- Add per-edge pool endpoints with IPv4, optional IPv6, gateway state, + participating cells, and readiness. +- Allow separate service pairs for shared, reserved, dedicated, and quarantine pools. - Keep management addresses separate from service endpoints. -- Prevent duplicate/conflicting address ownership. -- Reconcile gateway bindings, pool readiness, and DNS publication as one - revisioned workflow with last-valid state. - -### Scale checkpoint - -- Multiple pools and service pairs on each of at least two edges. -- Measured DNS publication and gateway reconciliation after endpoint health - changes without rewriting every domain. - -### Completion checklist - -- [ ] Three shared cells serve one IPv4/IPv6 pair on one edge. -- [ ] A reserved customer pool serves a different pair on the same edge. -- [ ] DNS returns the correct ready endpoint set for each pool. -- [ ] Withdrawal of one endpoint does not change unrelated pools. -- [ ] IPv4-only, IPv6-only, and dual-stack endpoints behave correctly. -- [ ] Address conflicts and invalid bindings fail before activation. -- [ ] Gateway, DNS, placement, and cell state converge after restart/reconcile. -- [ ] Tests, scale evidence, docs, and manual qualification pass. - -## Phase 5 — Simple Anycast pools - -### Goal - -Allow selected pools to use one shared IPv4/IPv6 pair across multiple POPs while -keeping route announcement outside CDNFoundry. - -### Implementation - -- Add routing mode `simple_anycast` to eligible pools. -- Store one pool-level IPv4 and optional IPv6 service pair. -- Attach explicit participating POPs/edges and readiness requirements. -- Bind the same service pair on every participating edge gateway. -- Publish the same Anycast pair for domains assigned to the pool. -- Expose a clear readiness/withdrawal signal for the operator or upstream routing - system. -- Document that the network/provider owns BGP advertisement and withdrawal. -- Do not add FRR, BIRD, router credentials, arbitrary routing commands, or a BGP - control plane to CDNFoundry. -- Preserve Geo-Unicast pools on the same fleet. - -### Scale and network checkpoint - -- At least two real or approved lab POPs using the same IPv4/IPv6 pair. -- Traffic observations from multiple external vantage points. -- Controlled POP loss and restoration with route behavior recorded by the - operator/provider. -- Gateway and cell capacity limits recorded independently per POP. - -### Completion checklist - -- [ ] One Anycast pool uses the same service pair on multiple POPs. -- [ ] Geo-Unicast and Anycast pools coexist on the same edges. -- [ ] CDNFoundry exposes honest ready/degraded/withdrawn state. -- [ ] POP failure does not corrupt another POP's local serving state. -- [ ] External route withdrawal/restoration is qualified and documented. -- [ ] UI/docs never claim CDNFoundry itself announces BGP routes. -- [ ] Physical-uplink and upstream-scrubbing limitations remain explicit. -- [ ] Tests, network evidence, docs, and manual qualification pass. - -## Phase 6 — Cache v2 storage and cache-key policy - -### Goal - -Turn the existing basic cache into a persistent, bounded, production-strength -cell cache without adding per-domain cache directories or a distributed cache. - -### Implementation - -- Use persistent per-cell cache volumes with explicit maximum size, inactive - duration, temporary-storage quota, and minimum-free-space policy. -- Add a small set of pool resource profiles such as `small`, `standard`, `large`, - and `streaming`. -- Keep domain placement stable so a domain normally reaches one cell/cache on an - edge. -- Add bounded cache-key query policies: - - include all parameters; - - ignore all parameters; - - include selected names; - - ignore selected names. -- Preserve deterministic Host, path, scheme, query, and cache-epoch behavior. -- Add bounded TTL policy for approved status codes such as 200/206, redirects, - and optional short negative caching. -- Add cache admission protection: - - per-domain admission rate; - - cache-key and query-variant limits; - - maximum cacheable object size; - - optional minimum cacheable object size; - - low-disk bypass; - - range and high-cardinality protections. -- Add explicit stale modes: - - off; - - stale-if-error; - - stale-while-revalidate; - - cache-only emergency; - - stale-only emergency. -- Preserve epoch full purge and exact URL purge. - -### Scale checkpoint - -- Cache-hit and cache-miss load at declared object-size distributions. -- Cache quota pressure, low-disk behavior, restart persistence, purge fan-out, - and high-cardinality abuse. -- Record throughput, latency, disk IOPS, disk usage, memory, CPU, and hit ratio. - -### Completion checklist - -- [ ] Cache survives routine cell restart and remains rebuildable after loss. -- [ ] Pool profiles enforce disk/temp/object/admission ceilings. -- [ ] Query policies produce deterministic keys without unbounded variants. -- [ ] TTL and status-code behavior match configured policy. -- [ ] Stale modes behave correctly during origin failure. -- [ ] Low disk and cache abuse bypass safely without filling the host. -- [ ] URL/full purge remains durable and bounded across participating cells. -- [ ] One domain cannot evict or fill unrelated cell caches beyond accepted limits. -- [ ] Tests, load evidence, docs, and manual qualification pass. - -## Phase 7 — Gzip, Brotli, and compressed delivery - -### Goal - -Reduce customer bandwidth with safe response compression integrated with Cache -v2 and bounded by cell/pool resources. - -### Implementation +- Publish only ready, non-withdrawn endpoints through system-managed DNS. +- Preserve country, continent, and global Geo-DNS fallback behavior. +- Prevent duplicate or conflicting address ownership. +- Reconcile gateway bindings, pool readiness, and DNS publication through one + revisioned workflow. + +**Scale target:** several pools and service pairs on each of at least two edges, +with endpoint-health changes measured without rewriting every domain. + +**Completion checklist:** + +- [ ] Three shared cells serve one dual-stack pair. +- [ ] A reserved pool serves a different pair on the same edge. +- [ ] DNS publishes the correct ready endpoints. +- [ ] Withdrawal affects only the intended pool. +- [ ] IPv4-only, IPv6-only, and dual-stack endpoints pass. +- [ ] Conflicts fail before activation. +- [ ] Restart and reconciliation converge gateway, DNS, pool, and cell state. +- [ ] Tests, scale, documentation, and manual qualification pass. + +# Phase 5 — Simple Anycast pools + +**Goal:** allow selected pools to use one shared IPv4/IPv6 pair across several +POPs while route advertisement remains owned by the network operator/provider. + +**Implementation:** + +- Add `simple_anycast` routing mode beside Geo-Unicast. +- Store one pool-level IPv4 and optional IPv6 pair. +- Attach explicit POPs/edges and readiness requirements. +- Bind the same pair on participating gateways. +- Publish the shared pair for assigned domains. +- Expose clear ready, degraded, and withdrawn signals. +- Do not add FRR, BIRD, router credentials, arbitrary commands, or BGP control. + +**Scale target:** at least two approved POPs using the same dual-stack service +pair, tested from multiple external vantage points with controlled POP loss. + +**Completion checklist:** + +- [ ] One Anycast pool serves from multiple POPs. +- [ ] Geo-Unicast and Anycast coexist on the same fleet. +- [ ] POP failure does not corrupt another POP's local state. +- [ ] External route withdrawal and restoration are recorded. +- [ ] UI and docs clearly state that CDNFoundry does not announce BGP routes. +- [ ] Uplink and upstream-scrubbing limitations remain explicit. +- [ ] Tests, network evidence, documentation, and manual qualification pass. + +# Phase 6 — Cache v2 + +**Goal:** turn the baseline cache into a persistent, bounded, production-strength +cell cache without a distributed cache or per-domain directories. + +**Implementation:** + +- Use persistent per-cell cache volumes with explicit size, inactive time, + temporary quota, and minimum free space. +- Add small, standard, large, and streaming pool profiles. +- Keep stable domain-to-cell placement. +- Add query policies: include all, ignore all, include selected, ignore selected. +- Keep deterministic scheme, Host, path, query, and epoch behavior. +- Add bounded TTL policies for approved status codes. +- Add admission, object-size, low-disk, range, and variant protections. +- Add stale-if-error, stale-while-revalidate, cache-only, and stale-only modes. +- Preserve exact URL purge and epoch full purge. + +**Scale target:** mixed HIT/MISS traffic, quota pressure, restart persistence, +purge fan-out, and high-cardinality abuse with throughput, latency, IOPS, CPU, +memory, disk, and hit ratio recorded. + +**Completion checklist:** + +- [ ] Cache survives routine restart and remains rebuildable after loss. +- [ ] Profiles enforce disk, temporary, object, and admission ceilings. +- [ ] Query policies do not create unbounded variants. +- [ ] TTL and stale behavior match policy. +- [ ] Low disk and abuse bypass safely. +- [ ] Purge remains durable across participating cells. +- [ ] One domain cannot exhaust unrelated cache resources beyond accepted limits. +- [ ] Tests, load evidence, documentation, and manual qualification pass. + +# Phase 7 — Gzip and Brotli compression + +**Goal:** reduce delivered bandwidth through safe compression integrated with +Cache v2 and bounded by pool and cell resources. + +**Implementation:** - Store one canonical uncompressed cache object by default. -- Request identity encoding from origins where required for deterministic cache - behavior. -- Enable Gzip as the default broadly compatible compression method. +- Request identity encoding from origins where required. +- Enable Gzip as the normal default. - Add optional Brotli through one immutable, pinned, tested edge image/module. -- Expose simple profiles only: - - `off`; - - `standard`; - - `maximum_savings` for reserved/dedicated pools. -- Use a tested MIME-type allowlist and minimum response size. -- Avoid recompressing JPEG, PNG, WebP, AVIF, video, archives, and other already - compressed formats. -- Handle `Accept-Encoding`, `Vary`, ETag/revalidation, HEAD, 304, stale, purge, - and identity fallback correctly. -- Disable or bound on-the-fly compression for range traffic and large responses. -- Add per-cell compression concurrency, CPU-pressure fallback, and emergency - disable behavior. -- Emit encoding, origin bytes, uncompressed bytes, served bytes, compression - ratio, profile, and fallback telemetry. - -### Scale checkpoint - -- Mixed identity/Gzip/Brotli clients against cache HIT and MISS traffic. -- Compressible and non-compressible object distributions. -- CPU saturation, concurrency limit, fallback, and large-response behavior. -- Record bandwidth saved, latency, throughput, and CPU cost per profile. - -### Completion checklist - -- [ ] Identity, Gzip, and Brotli responses decode to identical content. -- [ ] One canonical cached object serves different client encodings correctly. -- [ ] Already-compressed and range responses follow safe policy. -- [ ] `Vary`, ETag, revalidation, stale, and purge behavior remain correct. -- [ ] Shared pools cannot select unsafe compression levels. -- [ ] CPU pressure falls back or disables compression without stopping traffic. -- [ ] Compression telemetry and bandwidth-savings analytics are accurate. -- [ ] Tests, load evidence, docs, and manual qualification pass. - -## Phase 8 — Primary and backup origin failover - -### Goal - -Remove the single-origin availability weakness with one simple active-passive -backup origin per proxied hostname. - -### Implementation +- Expose only off, standard, and maximum-savings profiles. +- Use a tested MIME allowlist and minimum response size. +- Avoid recompressing images, video, archives, and other compressed formats. +- Handle Accept-Encoding, Vary, ETag, HEAD, 304, stale, purge, and fallback. +- Bound range traffic, large responses, concurrency, and CPU usage. +- Add emergency disable and CPU-pressure fallback. +- Record encoding, bytes, ratio, profile, and fallback telemetry. + +**Scale target:** mixed identity, Gzip, and Brotli clients against HIT and MISS +traffic, with bandwidth saved, throughput, latency, and CPU cost recorded. + +**Completion checklist:** + +- [ ] Identity, Gzip, and Brotli decode to identical content. +- [ ] One canonical object serves different encodings correctly. +- [ ] Compressed and range content follows safe policy. +- [ ] Vary, ETag, revalidation, stale, and purge remain correct. +- [ ] Shared pools cannot select unsafe levels. +- [ ] CPU pressure falls back without stopping traffic. +- [ ] Compression analytics are accurate. +- [ ] Tests, load evidence, documentation, and manual qualification pass. + +# Phase 8 — Primary and backup origin failover + +**Goal:** add one simple active-passive backup origin per proxied hostname. + +**Implementation:** - Support one primary and one optional backup origin. -- Reuse the same strict origin-address, TLS, header, timeout, and loop validation. -- Use bounded health checks and request-path failure evidence. +- Reuse strict origin, TLS, timeout, header, and loop validation. +- Add bounded health checks and request-path failure evidence. - Add failure threshold, recovery threshold, hold-down, and failback delay. -- Keep active origin state local to cells and last-valid during control-plane loss. -- Fail over without a Laravel request-path call. -- Use stale/cache-only behavior before unnecessary origin retry storms. -- Expose active origin, transition reason, timestamps, and health without secrets. -- Do not add weighted balancing, traffic percentages, geographic origin steering, - service discovery, or arbitrary origin pools. - -### Scale checkpoint - -- Controlled primary failure and recovery under concurrent cache HIT/MISS traffic. -- Record failover time, failback time, origin connection pressure, errors, and - unrelated-domain impact. - -### Completion checklist - -- [ ] Healthy primary receives normal origin traffic. -- [ ] Qualified primary failure moves traffic to backup within policy. -- [ ] Recovery uses hysteresis and does not flap. -- [ ] Both-origin failure follows stale/cache-only/maintenance policy. -- [ ] Invalid backup configuration never replaces valid primary state. -- [ ] Control-plane outage does not remove local failover behavior. +- Keep active-origin state local to cells during control-plane loss. +- Fail over without calling Laravel in the request path. +- Prefer stale or cache-only behavior before retry storms. +- Expose active origin and transition reason without secrets. +- Do not add weighted, percentage, geographic, or arbitrary origin pools. + +**Scale target:** controlled failover and recovery under concurrent HIT/MISS +traffic, with transition time, origin pressure, errors, and isolation recorded. + +**Completion checklist:** + +- [ ] Healthy primary receives normal traffic. +- [ ] Qualified failure moves traffic to backup. +- [ ] Recovery does not flap. +- [ ] Both-origin failure follows stale or maintenance policy. +- [ ] Invalid backup state never replaces valid primary state. +- [ ] Control-plane outage does not remove local failover. - [ ] One failing origin cannot exhaust unrelated origin budgets. -- [ ] Tests, load evidence, docs, and manual qualification pass. - -## Phase 9 — Managed OWASP CRS WAF - -### Goal - -Add optional managed application-signature protection without building a custom -WAF language or exposing raw ModSecurity configuration. - -### Implementation - -- Use a pinned ModSecurity v3 and OWASP Core Rule Set release. -- Build one immutable WAF-capable OpenResty cell image/profile. -- Support simple domain/pool profiles: - - `off`; - - `monitor`; - - `balanced`; - - `strict`. -- Map profiles to tested anomaly thresholds, paranoia levels, body-inspection - limits, and blocking behavior. -- Prefer WAF-capable reserved, dedicated, or quarantine cells where isolation is - required; do not force WAF overhead onto every normal pool. -- Support only bounded exclusions by approved dimensions such as rule ID, - hostname, path prefix, argument/header name, reason, owner, and expiry. -- Never expose arbitrary `SecRule`, arbitrary directives, customer rule uploads, - runtime downloads, or custom expression languages. -- Add detection/block reason, rule ID, anomaly score, processing time, body-limit, - exclusion, and profile telemetry with redaction. -- Roll new CRS/image versions through monitor-only canaries before blocking. -- Preserve the previous WAF image/ruleset and active traffic state on failure. - -### Scale and security checkpoint - -- Safe laboratory test corpus for common SQL injection, XSS, traversal, protocol, - and evasion patterns. -- False-positive corpus for representative applications. -- Cache HIT/MISS traffic with WAF enabled and disabled. -- Record latency, CPU, memory, request-body cost, throughput, detection accuracy, - false positives, and accepted limits. - -### Completion checklist - -- [ ] Off, monitor, balanced, and strict profiles behave as documented. -- [ ] Monitor records detections without blocking. -- [ ] Blocking profiles return stable reasons and preserve privacy. -- [ ] Bounded exclusions work, expire, audit, and cannot become arbitrary rules. -- [ ] Oversized or malformed bodies fail according to policy without exhausting cells. -- [ ] WAF failure/canary regression does not replace the previous valid runtime. -- [ ] Unrelated non-WAF pools remain healthy during WAF load/failure. -- [ ] Tests, security evidence, docs, and manual qualification pass. - -## Phase 10 — Expanded telemetry, analytics, and capacity control - -### Goal - -Make the new gateway, endpoint, pool, cell, cache, compression, failover, Anycast, -and WAF behavior observable and capacity-manageable. - -### Implementation - -- Add bounded telemetry for: - - gateway listeners, maps, routes, connections, errors, and revisions; - - service endpoints and Anycast readiness; - - pool participation, minimum readiness, and placement transitions; - - cell CPU, memory, connections, cache disk/temp, admission, and saturation; - - compression encoding, ratios, bytes saved, concurrency, and fallback; - - origin health, active origin, failover/failback, and circuit state; - - WAF profile, anomaly score, rule category, processing time, and action. -- Extend ClickHouse schemas and aggregates with bounded retention and query limits. -- Update administrator and domain analytics only where the data is useful. -- Add Prometheus alerts for stale maps, endpoint mismatch, cell exhaustion, cache - disk pressure, compression CPU pressure, origin failover, WAF errors, and - Anycast readiness disagreement. +- [ ] Tests, load evidence, documentation, and manual qualification pass. + +# Phase 9 — Managed OWASP CRS WAF + +**Goal:** add optional managed application-signature protection without exposing +a custom WAF language or raw ModSecurity configuration. + +**Implementation:** + +- Pin ModSecurity v3 and OWASP Core Rule Set releases. +- Build an immutable WAF-capable OpenResty cell image/profile. +- Support off, monitor, balanced, and strict profiles. +- Map profiles to tested thresholds, paranoia levels, body limits, and blocking. +- Prefer reserved, dedicated, or quarantine WAF-capable cells where isolation is needed. +- Allow only bounded exclusions by approved dimensions, reason, owner, and expiry. +- Reject arbitrary SecRule directives, customer rule uploads, runtime downloads, + and custom expression languages. +- Add rule, score, action, processing-time, body-limit, and exclusion telemetry. +- Roll new CRS/image versions through monitor-only canaries. +- Preserve the previous image and ruleset after failure. + +**Scale target:** safe attack and false-positive corpora plus HIT/MISS load, with +latency, CPU, memory, throughput, detection, false positives, and accepted limits recorded. + +**Completion checklist:** + +- [ ] Off, monitor, balanced, and strict behave as documented. +- [ ] Monitor detects without blocking. +- [ ] Blocking uses stable, privacy-safe reasons. +- [ ] Exclusions are bounded, audited, and expiring. +- [ ] Oversized or malformed bodies remain bounded. +- [ ] Failed canaries keep the previous valid WAF runtime. +- [ ] Non-WAF pools remain healthy during WAF load or failure. +- [ ] Tests, security evidence, documentation, and manual qualification pass. + +# Phase 10 — Observability and capacity control + +**Goal:** make gateway, endpoint, pool, cell, cache, compression, failover, +Anycast, and WAF behavior operationally visible and capacity-manageable. + +**Implementation:** + +- Add bounded gateway, endpoint, pool, placement, cell, cache, compression, + origin, and WAF telemetry. +- Extend ClickHouse schemas and aggregates with bounded retention and queries. +- Add useful administrator and domain analytics without leaking unrelated data. +- Add alerts for stale maps, endpoint mismatch, cell exhaustion, cache pressure, + compression pressure, origin failover, WAF errors, and Anycast disagreement. - Keep telemetry best-effort and outside serving decisions. -### Scale checkpoint - -- At least 20,000 active proxied domains across multiple pools, cells, endpoints, - and edges. -- High-cardinality fields remain bounded or aggregated. -- ClickHouse/Vector outage and backlog recovery under live traffic. +**Scale target:** at least 20,000 active proxied domains across several pools, +cells, endpoints, and edges, including ClickHouse/Vector outage and recovery. -### Completion checklist +**Completion checklist:** -- [ ] Every new serving component has healthy/degraded/unavailable state. -- [ ] Metrics and logs identify the responsible pool, cell, edge, and revision. -- [ ] Domain users cannot see unrelated pool/customer data. -- [ ] Raw logs remain redacted, bounded, and directly delivered to ClickHouse. -- [ ] Telemetry outage never blocks gateway/cell traffic. -- [ ] Queries remain bounded and responsive at the qualification dataset. -- [ ] Alerts and runbooks identify actionable recovery steps. -- [ ] Tests, scale evidence, docs, and manual qualification pass. +- [ ] Every new component has healthy, degraded, and unavailable states. +- [ ] Metrics identify pool, cell, edge, and revision. +- [ ] Domain users cannot see unrelated data. +- [ ] Raw logs remain bounded and redacted. +- [ ] Telemetry outage never blocks serving. +- [ ] Queries remain bounded at the qualification dataset. +- [ ] Alerts link to actionable runbooks. +- [ ] Tests, scale evidence, documentation, and manual qualification pass. -## Phase 11 — Bounded fleet rollout automation +# Phase 11 — Bounded fleet rollout automation -### Goal +**Goal:** automate proven edge upgrades without introducing general remote +execution or dynamic containers. -Automate proven manual upgrades when fleet size makes per-edge rollout -inefficient, without adding general remote execution or dynamic containers. +**Implementation:** -### Implementation - -- Manage immutable versions for gateway, edge agent, normal cell, and WAF cell. +- Manage immutable gateway, agent, normal-cell, and WAF-cell versions. - Define compatibility ranges and a bounded mixed-version window. -- Support explicit canary edges/POPs and rollout waves. -- Pause automatically on health, error, readiness, revision, or capacity thresholds. -- Roll back to the last compatible image/configuration. -- Preserve fixed cell-slot topology; rollout automation does not create arbitrary - containers or run arbitrary commands. -- Expose desired/current version, wave, progress, failure, pause, and rollback. -- Audit every rollout decision and retain operator confirmation for destructive - or fleet-wide actions. - -### Scale checkpoint - -- Multi-edge, multi-POP rollout with normal and WAF cells. -- Mixed-version serving, controlled canary failure, automatic pause, and rollback. -- Record rollout time, unavailable capacity, errors, and operator recovery steps. +- Support canary edges/POPs and rollout waves. +- Pause on health, error, readiness, revision, or capacity thresholds. +- Roll back to the last compatible image and configuration. +- Preserve the fixed slot topology. +- Expose desired/current version, wave, progress, pause, failure, and rollback. +- Audit every rollout decision. -### Completion checklist +**Scale target:** multi-edge, multi-POP rollout with mixed normal and WAF cells, +including failed canary, automatic pause, and rollback. -- [ ] Canary completes before later waves start. -- [ ] Failed canary pauses rollout automatically. -- [ ] Rollback restores the prior compatible runtime without database restore. -- [ ] Existing traffic continues through a bounded mixed-version window. -- [ ] No arbitrary command execution or unbounded container creation exists. -- [ ] Version drift and incompatible agents/cells are visible. -- [ ] Tests, recovery evidence, docs, and manual qualification pass. +**Completion checklist:** -## Phase 12 — Final production qualification for the new architecture +- [ ] Canary completes before later waves. +- [ ] Failed canary pauses automatically. +- [ ] Rollback restores the previous compatible runtime. +- [ ] Traffic continues during the mixed-version window. +- [ ] No arbitrary command execution or dynamic unbounded containers exist. +- [ ] Version drift and incompatibility are visible. +- [ ] Tests, recovery evidence, documentation, and manual qualification pass. -### Goal +# Phase 12 — Final production qualification -Prove the complete post-baseline architecture works as one simple, solid, -recoverable product. +**Goal:** prove the complete post-baseline architecture as one deployable, +recoverable, simple, and solid product. -### Required topology +**Required topology:** - At least two POPs/edges. - At least eight bounded cell slots per edge. - One shared pool using at least three cells per edge. -- One reserved customer pool using a separate IPv4/IPv6 pair. +- One reserved pool using a separate IPv4/IPv6 pair. - One quarantine pool. -- Geo-Unicast service endpoints. -- One Simple Anycast pool across both POPs where the operator can provide the - required routing environment. -- Persistent Cache v2, Gzip, Brotli, primary/backup origins, and one managed WAF - pool/profile. -- Real IPv4 and IPv6 clients/origins where available. - -### Final qualification - -- Complete new-edge installation and registration from clean hosts. -- Create and activate all pool types and endpoint modes. -- Serve real HTTP/HTTPS through multiple service pairs on the same edge. -- Verify stable domain placement, movement, drain, quarantine, and rollback. -- Verify Geo-Unicast and Simple Anycast behavior from external vantage points. -- Exercise Cache v2, persistent restart, purge, stale, quota pressure, Gzip, and Brotli. -- Fail the primary origin and prove controlled backup failover/failback. -- Exercise WAF monitor/block/exclusion/canary behavior with safe test traffic. -- Stop Laravel, queues, Redis/Valkey, and ClickHouse while existing traffic continues. -- Restart gateway, agent, cells, Vector, ClickHouse, DNSdist, and PowerDNS according - to their runbooks. -- Apply invalid gateway/cell/WAF artifacts and verify previous valid state remains. -- Saturate one cell within the approved lab and verify unrelated cells/pools continue. -- Perform fleet canary upgrade and rollback. -- Restore control-plane data on a clean replacement host and reconcile derived state. -- Record measured limits, hardware, topology, RPO, RTO, throughput, latency, - saturation, known limitations, and owner browser evidence. - -### Completion checklist - -- [ ] Every phase completion gate is passed and linked to evidence. -- [ ] Existing completed baseline features pass the regression smoke suite. -- [ ] All current API/OpenAPI, UI, deployment, architecture, security, operations, - troubleshooting, and runbook documentation is updated. -- [ ] All tests clearly report passed, failed, blocked, and not executed results. -- [ ] No unresolved critical/high failure remains. -- [ ] The owner records the final manual/browser/real-traffic qualification. -- [ ] The release notes state measured capabilities and limitations without - unsupported scale, Anycast, WAF, or DDoS claims. - -# Future candidates — not part of the committed phases - -These capabilities remain outside the committed roadmap until repeated customer -or operator demand justifies a separate scope and qualification contract: - -- secondary ACME certificate authority; -- DNSSEC signing and rollover lifecycle; +- Geo-Unicast endpoints. +- One Simple Anycast pool where an approved routing environment exists. +- Persistent Cache v2, Gzip, Brotli, backup origin, and managed WAF. +- Real IPv4 and IPv6 traffic where available. + +**Final checklist:** + +- [ ] Clean edge installation and registration pass. +- [ ] All pool kinds and endpoint modes pass. +- [ ] Multiple service pairs work on one edge. +- [ ] Placement, movement, drain, quarantine, and rollback pass. +- [ ] Geo-Unicast and Anycast external checks pass. +- [ ] Cache persistence, purge, stale, pressure, Gzip, and Brotli pass. +- [ ] Origin failover and failback pass. +- [ ] WAF monitor, block, exclusion, canary, and rollback pass. +- [ ] Existing traffic continues through controlled control-plane and telemetry outages. +- [ ] Invalid gateway, cell, and WAF candidates preserve previous valid state. +- [ ] One saturated cell does not stop unrelated pools. +- [ ] Fleet canary upgrade and rollback pass. +- [ ] Clean-host control-plane restore and derived-state reconciliation pass. +- [ ] All affected API, UI, architecture, deployment, operations, security, + troubleshooting, and runbook documentation is current. +- [ ] Every completed test is linked and every unexecuted test is marked clearly. +- [ ] Owner-run browser and real-traffic qualification is recorded. +- [ ] Release notes state measured capabilities and limitations honestly. + +# Future candidates + +The following remain outside the committed roadmap until real demand justifies a +separate bounded implementation and qualification contract: + +- secondary ACME authority; +- DNSSEC; - HTTP/3 and QUIC; -- private outbound origin connector for non-public origins; -- immutable/deletion-protected backup storage and warm control-plane standby; -- long-retention analytics archive/export; +- private origin connector; +- immutable backup storage and warm control-plane standby; +- long-retention analytics archive; - replicated placement for exceptional high-volume domains; -- additional placement policies that preserve the bounded pool/cell model; +- additional bounded placement policies; - origin shield or hierarchical cache. -A candidate is admitted only when: - -- a real requirement exists; -- the existing product cannot solve it safely; -- operational and failure costs are understood; -- it does not move traffic through Laravel; -- it does not require rewriting the control plane or generic edge runtime; -- it has bounded state, rollback, observability, tests, docs, and a release gate; -- disabling it leaves the committed product functional. - # Explicitly out of scope -- Weighted origin balancing and percentage traffic splitting -- General BGP/router management -- Volumetric DDoS scrubbing guarantees -- CAPTCHA, browser challenges, or bot-scoring platforms -- Customer-written WAF rules or edge scripts -- Serverless workers or plugin marketplaces -- Object-storage product features -- Per-domain containers/processes by default -- Kubernetes as a deployment requirement -- Billing, payment, reseller, organization, team, or custom-role systems +- weighted or percentage origin balancing; +- general BGP/router management; +- volumetric DDoS scrubbing guarantees; +- CAPTCHA, browser challenge, or bot-scoring platforms; +- customer-written WAF rules or edge scripts; +- serverless workers or plugin marketplaces; +- object-storage product features; +- per-domain containers or processes by default; +- Kubernetes as a deployment requirement; +- billing, reseller, organization, team, or custom-role systems. ## Final rule -> Keep the completed platform intact. Add one bounded production capability per -> phase. Keep the gateway simple, keep cells isolated, preserve last-valid state, -> test real traffic and failure behavior, record scale honestly, update every -> affected document, and never move DNS or HTTP traffic through Laravel. +Keep the completed platform intact. Add one bounded production capability per +phase. Keep the gateway simple, keep cells isolated, preserve last-valid state, +test real traffic and failure behavior, record scale honestly, update every +affected document, and never move DNS or HTTP traffic through Laravel. From 78c71430eec4022b566fae655d357c275ada6751 Mon Sep 17 00:00:00 2001 From: vaheeD khoshnouD Date: Mon, 27 Jul 2026 15:00:44 +0330 Subject: [PATCH 5/7] docs: align browser qualification with next roadmap --- docs/manual-browser-qualification.md | 746 +++++++++++---------------- 1 file changed, 290 insertions(+), 456 deletions(-) diff --git a/docs/manual-browser-qualification.md b/docs/manual-browser-qualification.md index 9d6ebad..2fe3905 100644 --- a/docs/manual-browser-qualification.md +++ b/docs/manual-browser-qualification.md @@ -1,27 +1,26 @@ --- title: Manual browser qualification -description: Owner-run browser and operator checkpoints aligned with the CDNFoundry product roadmap. +description: Owner-run browser and operator checkpoints for the post-baseline CDNFoundry roadmap. --- # Manual browser qualification -## 1. Purpose and ownership +## Purpose -This is the owner-run browser qualification contract for CDNFoundry. +This document mirrors `docs/roadmap.md` and covers only the new post-baseline +phases. The completed original platform remains covered by regression checks and +is not repeated as roadmap work here. -Coding agents: +Browser qualification is owner-run. Coding agents maintain this checklist but +must report it as not executed unless the owner records the evidence. -- maintain this file when implemented UI or operator behavior changes; -- do not launch or automate Chromium, Playwright, Selenium, Cypress, or another browser; -- report browser qualification as not run unless the owner supplies evidence. +Do not invent UI controls before they exist. During implementation, replace each +phase's activation contract with exact menu names, fields, buttons, URLs, and +expected results from the real UI. -A roadmap phase is not release-complete until its active browser section passes. +## Evidence template -Planned roadmap phases must not invent screens, fields, or menu names before implementation. Their qualification contracts are activated and expanded into exact UI steps in this file when the corresponding UI exists. - -## 2. Evidence record - -For every run, record: +Record the following for every qualification run: ```text Date: @@ -30,464 +29,299 @@ Commit SHA: Release/image tags: Environment: Control-plane URL: -Topology: -DNS clusters: -Edges and POPs: -Cell slots per edge: -Pool routing modes: -Public IPv4/IPv6: -Browser and version: +Browser/version: Desktop viewport: Mobile viewport: +Edges/POPs: +Cell slots per edge: +Pools and routing modes: +Public IPv4/IPv6 endpoints: Test domains: Origin fixtures: Start time: End time: -Result: passed / failed / blocked +Overall result: passed / failed / blocked ``` -For each checkpoint, record: +For every checkpoint, record: - expected result; - actual result; - sanitized screenshot or screen recording; -- operation, task, revision, pool, cell, edge, certificate, purge, or backup identifier; +- relevant operation, revision, edge, pool, cell, endpoint, certificate, purge, + origin, WAF, or rollout identifier; - browser console/network error when relevant; -- severity and owner for failures; +- failure severity and owner; - retest evidence. -Never include passwords, tokens, API keys, private keys, CA keys, signing keys, backup credentials, raw customer telemetry, or customer data. - -## 3. Global preparation - -1. Use a disposable or approved production-like environment without deleting persistent Compose volumes. -2. Apply Laravel and PowerDNS migrations explicitly. -3. Confirm `/api/health`, `/api/ready`, DNSdist UDP/TCP, Horizon, scheduler, PostgreSQL, Redis/Valkey, ClickHouse, Vector, MMDB, registered edges, cells, and gateway health where implemented. -4. Create or identify one administrator and one disposable domain user. -5. Prepare delegated test domains, IPv4/IPv6 origin fixtures, one failing origin, and approved external DNS/HTTP vantage points. -6. Confirm the exact commit and immutable image tags being qualified. -7. Use desktop and narrow mobile viewports. -8. Confirm keyboard focus, labels, validation focus, table scrolling, empty/loading/degraded/error states, destructive confirmation, and one-time-secret boundaries throughout the run. -9. Confirm no secret appears in page source, browser storage beyond intended session/token data, browser console, rendered errors, audit details, or downloadable exports. - -# Active qualification — implemented baseline - -## Phase 1 — Foundation, access, and system identity - -### Administrator and domain user - -1. Sign in to the administrator panel. -2. Create a user with a unique disposable email and type `User`. -3. Confirm the user appears once and no plaintext password is displayed after save. -4. Disable the user and confirm login to the domain-user panel is denied. -5. Re-enable the user and confirm login succeeds. -6. Confirm an administrator cannot disable, demote, or delete their own active account. -7. Confirm a domain user cannot open administrator routes or see global resources. - -### Profile, password, and tokens - -1. Open the profile surface in both panels. -2. Change the display name and confirm shared navigation/account UI updates. -3. Change the password using the current password and matching confirmation. -4. Create a named API token. -5. Confirm plaintext token appears once only. -6. Reload and confirm only safe metadata is visible. -7. Revoke the token and confirm API use fails. - -### System DNS identity - -1. Open the system DNS identity surface. -2. Enter a platform zone, two nameservers, IPv4/IPv6 glue, proxy hostname, SOA mailbox, and bounded timing values. -3. Preview and record the normalized records and confirmation receipt. -4. Change one field and attempt to apply the old confirmation. Expect rejection. -5. Preview again and apply the exact payload. -6. Record the asynchronous operation and confirm pending/running/succeeded states without blocking the page. -7. Confirm failed deployment keeps the previous active identity. - -### Completion record - -| Gate | Result | -| --- | --- | -| Implementation | | -| Documentation | | -| Automated/runtime tests | | -| Scale checkpoint | | -| Failure/recovery | | -| Manual browser | | -| Release decision | | - -## Phase 2 — Domains and authoritative DNS - -### DNS cluster - -1. Create a DNS cluster with unique name, location, private API details, server ID, and nameserver data. -2. Confirm credentials are not displayed in table, detail, audit, or edit surfaces. -3. Test the connection and record the operation. -4. Enable the qualified cluster. -5. Reconcile zones and confirm desired/active revision and checksum visibility. -6. Introduce a controlled failure on a second target and confirm the healthy target retains valid state. - -### Domain lifecycle and assignment - -1. Add a delegated test domain without entering an origin. -2. Confirm pending verification, revision, nameserver guidance, and no proxy/TLS side effect. -3. Assign the disposable domain user. -4. Sign in as that user and confirm only assigned domains are visible. -5. Verify nameservers and activate the domain. -6. Confirm lifecycle and DNS deployment acknowledgement. -7. Exercise delayed deprovision/cancel behavior only in the disposable environment and confirm tombstone/reclaim safeguards. - -### DNS records and bulk work - -1. Add valid A, AAAA, CNAME, MX, TXT, NS, CAA, and SRV examples. -2. Confirm type-specific fields, normalized names, and TTL limits. -3. Attempt CNAME coexistence, out-of-zone owner, invalid underscore, duplicate, and malformed IPv4/IPv6 input. Expect inline rejection and no revision change. -4. Confirm a domain user cannot change protected apex delegation behavior. -5. Bulk edit/delete a bounded set and confirm one intended revision transition. -6. Import a bounded BIND zone in append mode and export deterministic text. -7. Import in replacement mode and confirm the final record set. -8. Confirm cursor pagination and useful empty/degraded states. - -### External verification - -1. Use real UDP and TCP `dig` against every qualified DNS cluster. -2. Test IPv4 and IPv6 client paths. -3. Confirm SOA serial, authoritative flags, record values, and negative answers. -4. Stop the control plane and confirm existing authoritative DNS continues. - -### Completion record - -| Gate | Result | -| --- | --- | -| Implementation | | -| Documentation | | -| Automated/runtime tests | | -| Scale checkpoint | | -| Failure/recovery | | -| Manual browser/external DNS | | -| Release decision | | - -## Phase 3 — Geo-DNS - -1. Create a Geo-DNS A record with default, continent, and country targets. -2. Preview with addresses matching each level and with an unknown/documentation address. -3. Confirm country wins before continent, then default. -4. Repeat using AAAA and IPv6 preview input. -5. Attempt duplicate geography, missing default, excessive targets, invalid address family, and unsupported record type. Expect rejection and no revision change. -6. Confirm the UI labels resolver/ECS accuracy honestly. -7. Verify from approved external vantage points and record ECS or resolver-based behavior. -8. Interrupt MMDB update/provider access and confirm the last valid database remains active. - -### Completion record - -| Gate | Result | -| --- | --- | -| Implementation | | -| Documentation | | -| Automated/runtime tests | | -| Scale checkpoint | | -| Failure/recovery | | -| Manual browser/external vantage | | -| Release decision | | - -## Phase 4 — Proxy, baseline edge pools, and edge agent - -### Pools and edges - -1. Create or inspect shared, quarantine, and exceptional dedicated pools. -2. Create two edges in different locations with unique IPv4 and optional IPv6 management/default addresses. -3. Record each bootstrap token once and confirm it disappears after navigation. -4. Enroll real agents and confirm identity, version, heartbeat, active sequence, cell state, and bounded capacity. -5. Rotate one identity and confirm the old identity is rejected. -6. Restore the edge and exercise drain/undrain. -7. Confirm routing and status change without affecting the other edge. - -### Proxied hostname and origin - -1. Enable proxy for a valid hostname. -2. Configure explicit origin scheme, host, port, Host header, SNI, TLS verification, timeouts, retry bound, WebSocket option, and health path where supported. -3. Confirm platform-managed DNS content and visible safe origin metadata. -4. Attempt loopback, private/disallowed, link-local, metadata, multicast, platform, edge-service, and proxy-loop destinations. Expect rejection. -5. Run the asynchronous origin test and record resolved address, status, latency, TLS result, or stable failure reason. -6. Confirm signed edge delivery acknowledgement. -7. Move the domain to quarantine and confirm target ready before source drain. -8. Roll back to a retained revision and confirm a new higher revision. - -### Real traffic - -1. Send HTTP and HTTPS through each eligible edge using correct Host/SNI. -2. Test IPv4 and IPv6. -3. Confirm unknown Host/SNI rejection. -4. Stop Laravel/queues temporarily and confirm existing traffic continues. -5. Submit an invalid artifact in the controlled runtime test and confirm the previous state remains active. - -### Completion record - -| Gate | Result | -| --- | --- | -| Implementation | | -| Documentation | | -| Automated/runtime tests | | -| Scale checkpoint | | -| Failure/recovery | | -| Manual browser/real traffic | | -| Release decision | | - -## Phase 5 — TLS, baseline cache, and purge - -### Managed and custom TLS - -1. Enable proxy for an eligible active domain and open its TLS surface. -2. Confirm managed issuance state without private material. -3. Verify public apex/wildcard HTTPS as applicable and record certificate fingerprint, names, issuer, and expiry. -4. Exercise renew and reissue operations. -5. Upload a valid leaf, chain, and matching key and confirm only safe metadata remains visible. -6. Attempt wrong key, wrong name, invalid chain, expired certificate, and oversized PEM. Expect rejection. -7. Remove the custom certificate and confirm managed reconciliation. -8. Confirm a current valid certificate continues during a controlled ACME failure. - -### Cache and purge - -1. Configure enabled state, edge/browser TTL, object size, origin-header policy, query behavior, bypass cookies, and stale grace. -2. Save and confirm edge acknowledgement. -3. Request a cacheable object twice and confirm MISS then HIT. -4. Confirm browser headers and deterministic cache key behavior. -5. Enable short development mode and confirm absolute expiry and BYPASS. -6. Disable it and confirm caching resumes. -7. Purge one exact URL and confirm bounded target tasks and a later MISS. -8. Purge all and confirm epoch increment without filesystem scan. -9. Cause one controlled delivery failure and confirm safe retry of the same durable task. - -### Completion record - -| Gate | Result | -| --- | --- | -| Implementation | | -| Documentation | | -| Automated/runtime tests | | -| Scale checkpoint | | -| Failure/recovery | | -| Manual browser/public HTTPS | | -| Release decision | | - -## Phase 6 — Security and DDoS readiness - -1. Select standard, protected, quarantine, and manual profiles. -2. Confirm fixed profiles cannot exceed their values and manual input remains within platform ceilings. -3. Configure quarantine policy, allowed methods, and trusted proxy CIDRs. -4. Add ordered IP, CIDR, country, and continent allow/block rules. -5. Import bounded rules in append and replacement modes. -6. Send matching IPv4 and IPv6 traffic and confirm action/reason visibility. -7. Exercise restrict, quarantine, recover, and release. -8. Confirm target-first placement and safe recovery state. -9. Apply an expiring emergency action and confirm persistence across restart, expiry, and audit. -10. Withdraw and restore a pool service address from DNS. -11. Confirm an unrelated domain/cell remains healthy during controlled abuse traffic. -12. Confirm the UI and documentation do not claim protection after physical uplink saturation. - -### Completion record - -| Gate | Result | -| --- | --- | -| Implementation | | -| Documentation | | -| Automated/runtime tests | | -| Scale/isolation checkpoint | | -| Failure/recovery | | -| Manual browser/attack-pattern traffic | | -| Release decision | | - -## Phase 7 — Telemetry, analytics, and usage export - -1. Generate controlled DNS, HTTP, HTTPS, cache, origin, TLS, security, edge, and deployment events. -2. As the domain user, open analytics and confirm only assigned-domain data. -3. Verify summary, time series, units, ranges, status, cache, geography, hostname, URL, origin, edge, and DNS views that are currently implemented. -4. Inspect available raw request, DNS, error, and security logs. -5. Confirm bounded pagination, masking/redaction, and no secrets/query data beyond the documented schema. -6. Download domain usage JSON/CSV and confirm stable columns and scope. -7. As administrator, inspect global telemetry, edge/cell health, drops, and buffer state. -8. Stop ClickHouse and confirm visible analytics outage while DNS/HTTP continues. -9. Restore it and confirm bounded backlog recovery and partial-data labeling. - -### Completion record - -| Gate | Result | -| --- | --- | -| Implementation | | -| Documentation | | -| Automated/runtime tests | | -| Scale/query checkpoint | | -| Failure/recovery | | -| Manual browser | | -| Release decision | | - -## Phase 8 — Operations, recovery, and baseline release - -1. Inspect the administrator dashboard and confirm component, queue, edge, cell, DNS, TLS, backup, and recent-operation states currently implemented. -2. Inspect pending, succeeded, and failed operations and retry one supported failure. -3. Inspect failed-job information and confirm payload redaction and bounds. -4. Change one non-runtime typed setting and one runtime-affecting setting. -5. Confirm only the runtime change creates deployment/reconciliation work. -6. Create and verify an encrypted backup. -7. Perform restore preflight with exact confirmation and re-authentication. -8. Complete the documented clean-host maintenance restore in an isolated environment. -9. Run one compatible canary upgrade and rollback. -10. Confirm desktop/mobile layout, keyboard use, error focus, pagination, downloads, confirmations, and one-time secrets. -11. Record RPO, RTO, hardware, topology, throughput, saturation, and unresolved limitations. - -### Completion record - -| Gate | Result | -| --- | --- | -| Implementation | | -| Documentation | | -| Automated/runtime tests | | -| Scale/throughput checkpoint | | -| Backup/recovery/upgrade | | -| Manual browser/production | | -| Release decision | | - -# Planned qualification contracts - -The following sections are inactive until their implementation exists. They define required outcomes without inventing menu names, pages, fields, or controls. When a phase is implemented, replace its section with exact browser steps derived from the real UI and keep every checkpoint below. - -## Phase 9 — Edge gateway and bounded cell inventory - -Required owner-visible outcomes: - -- installed cell-slot inventory, stable cell identity, assignment state, resource limits, health, active revision, and last report are inspectable; -- gateway service addresses, listeners, Host/SNI map revision, health, and last-valid state are inspectable; -- agent enrollment does not expose Docker control or create unbounded containers; -- drain, restart, invalid-map rejection, and rollback operations provide durable status and audit; -- desktop/mobile and degraded/empty/error states are usable. - -Required real-traffic evidence: - -- one public service IP pair routes HTTP Host and HTTPS SNI to the expected cells; -- unknown/mismatched traffic is rejected; -- client address preservation is correct; -- one cell restart does not interrupt unrelated cells; -- invalid gateway state preserves the previous map. - -Activation rule: write exact UI steps when Phase 9 surfaces are implemented. - -## Phase 10 — Multi-cell pools, endpoints, and stable placement - -Required owner-visible outcomes: - -- pool kind, participating edges, service endpoints, assigned cells, minimum-ready policy, capacity, and domain placement are inspectable; -- shared, reserved, dedicated, and quarantine constraints are enforced; -- slot assignment, release, drain, migration, and rollback have operation status and audit; -- only participating targets show deployment/acknowledgement. - -Required real-traffic evidence: - -- one IPv4/IPv6 pair fronts three shared cells; -- another pair fronts reserved customer cells on the same edge; -- stable domain placement preserves cache locality; -- migration is target-first and unrelated domains remain healthy; -- cell shortage and protected capacity reservations fail safely. - -Activation rule: write exact UI steps when Phase 10 surfaces are implemented. - -## Phase 11 — Geo-Unicast and Simple Anycast - -Required owner-visible outcomes: - -- routing mode, shared Anycast addresses, participating POPs, gateway/cell/revision readiness, route-advertised signal, and withdrawal state are inspectable; -- Geo controls are hidden or unavailable for Anycast pools; -- BGP neighbor credentials/configuration are absent; -- readiness and failure ordering are clear and audited. - -Required external evidence: - -- Geo-Unicast returns ready location endpoints; -- Simple Anycast returns one shared IPv4/IPv6 pair; -- multiple POPs serve the same active revision; -- failed POP withdrawal precedes exclusion; -- healthy POPs continue; -- control-plane outage does not stop valid externally routed traffic. - -Activation rule: write exact UI steps when Phase 11 surfaces are implemented. - -## Phase 12 — Cache v2 and compression - -Required owner-visible outcomes: - -- cache resource profile, persistent quota, free-space threshold, inactive period, object-size limit, query policy, cookie bypass, status TTL, stale policy, and compression profile are inspectable and bounded; -- Gzip/Brotli availability and effective profile are clear; -- disk, admission, variant, compression CPU, savings, purge, and degradation states are visible; -- emergency compression disable and retry/rollback are audited. - -Required real-traffic evidence: - -- identity, Gzip, and Brotli clients receive identical decoded content; -- cache remains canonical without accidental `Accept-Encoding` fragmentation; -- MIME, size, range, and precompressed exclusions work; -- MISS/HIT/STALE/revalidate/purge remain correct; -- disk/CPU pressure is bounded and unrelated cells remain healthy. - -Activation rule: write exact UI steps when Phase 12 surfaces are implemented. - -## Phase 13 — Simple origin resilience - -Required owner-visible outcomes: - -- primary/backup origin, health evidence, effective origin, failover/recovery state, thresholds, last transition, and audit are inspectable; -- both origins use the same safety validation; -- no weighted or percentage routing controls exist. - -Required real-traffic evidence: - -- healthy primary serves; -- qualified failure switches to backup; -- flapping is bounded; -- failed backup cannot loop retries; -- controlled primary recovery works; -- cache and unrelated domains remain correct. - -Activation rule: write exact UI steps when Phase 13 surfaces are implemented. - -## Phase 14 — Managed OWASP CRS WAF - -Required owner-visible outcomes: - -- off, monitor, balanced, and strict profiles are available only where supported; -- effective pinned engine/ruleset versions, WAF-capable pool/cell state, anomaly/action metrics, exclusions, expiry, audit, and rollout state are inspectable; -- exclusions are narrow and bounded; -- no raw ModSecurity or custom rule-language editor exists. - -Required real-traffic evidence: - -- monitor records but does not block; -- balanced/strict block qualified test cases; -- false-positive exclusion works only in its declared scope; -- invalid ruleset preserves the previous image/rules; -- WAF load is isolated from normal pools; -- canary rollout pauses and rolls back. - -Activation rule: write exact UI steps when Phase 14 surfaces are implemented. - -# Future admission - -Phases 15 and 16 remain future candidates. Do not add browser checkpoints until a capability is admitted, implemented, and has real UI behavior. - -# Final result record - -For every failed checkpoint record: +Never record passwords, tokens, API keys, private keys, signing keys, backup +credentials, raw customer telemetry, or customer data. + +## Global preparation + +1. Use a disposable or approved production-like environment. +2. Preserve persistent Compose volumes and existing development data. +3. Confirm the exact commit and immutable image tags under qualification. +4. Confirm implemented health/readiness endpoints are healthy. +5. Prepare one administrator and one disposable domain user. +6. Prepare delegated test domains and IPv4/IPv6 origin fixtures. +7. Prepare one failing origin and approved external DNS/HTTP vantage points. +8. Use desktop and narrow mobile viewports. +9. Check keyboard navigation, focus, labels, validation focus, responsive tables, + loading, empty, degraded, error, destructive-confirmation, and one-time-secret states. +10. Confirm no secret appears in page source, browser storage beyond intended + session data, console output, rendered errors, audit details, or exports. +11. Run the completed baseline regression smoke suite before releasing any new phase. + +# Phase 1 qualification — Edge gateway ingress + +Activate this section when gateway endpoint and routing management exist in the UI. + +- [ ] Create or edit an edge gateway configuration. +- [ ] Add valid IPv4 and IPv6 service addresses. +- [ ] Confirm duplicate, malformed, and conflicting addresses are rejected. +- [ ] Inspect generated Host/SNI routing state without exposing secrets. +- [ ] Apply a valid routing revision and observe pending, active, and acknowledged states. +- [ ] Attempt an invalid revision and confirm the previous valid map remains active. +- [ ] Send real HTTP Host traffic to each configured service address. +- [ ] Send real HTTPS SNI traffic to each configured service address. +- [ ] Confirm unknown Host, unknown SNI, and unassigned destination traffic are rejected. +- [ ] Restart the gateway and confirm active routing is restored. +- [ ] Verify gateway health, listener, revision, connection, and error visibility. +- [ ] Confirm existing baseline edge traffic remains functional. + +**Phase 1 result:** passed / failed / blocked / not activated + +# Phase 2 qualification — Bounded cell inventory + +Activate this section when bounded cell-slot installation and inventory exist. + +- [ ] Configure a bounded slot count for a disposable edge. +- [ ] Install or reconcile the edge and confirm exactly that number of slots exists. +- [ ] Confirm every slot has unique identity, ports, paths, health, and resource limits. +- [ ] Inspect assigned, unassigned, ready, degraded, drained, and stopped states. +- [ ] Assign and unassign a slot through the supported workflow. +- [ ] Drain and restore one slot. +- [ ] Restart one slot and confirm unrelated slots continue serving. +- [ ] Trigger a controlled unhealthy slot and confirm gateway and agent remain healthy. +- [ ] Confirm the agent UI/API exposes no unrestricted container command surface. +- [ ] Verify cache, temporary, and log quota visibility. +- [ ] Confirm enrollment, mTLS identity, and snapshot recovery regression checks pass. + +**Phase 2 result:** passed / failed / blocked / not activated + +# Phase 3 qualification — Multi-cell pools and placement + +Activate this section when pool membership and placement management exist. + +- [ ] Create or inspect shared, reserved, dedicated, and quarantine pool kinds. +- [ ] Add at least three cells from one edge to one shared pool. +- [ ] Configure minimum-ready-cell and capacity policy. +- [ ] Assign several test domains and confirm stable placement. +- [ ] Add an unrelated domain and confirm existing placements do not reshuffle unnecessarily. +- [ ] Confirm a dedicated pool rejects a second domain. +- [ ] Move a domain to another cell and observe target preparation before route switch. +- [ ] Force target preparation failure and confirm source placement remains active. +- [ ] Complete a successful move and confirm source state is removed only after cutover. +- [ ] Move a domain into and out of quarantine. +- [ ] Confirm non-participating cells do not receive domain artifacts. +- [ ] Verify unrelated domains and caches remain healthy during movement. + +**Phase 3 result:** passed / failed / blocked / not activated + +# Phase 4 qualification — Pool endpoints and Geo-Unicast + +Activate this section when pool endpoint management exists. + +- [ ] Add a dual-stack service endpoint to a shared pool on one edge. +- [ ] Add a different service pair to a reserved pool on the same edge. +- [ ] Confirm management and service addresses remain distinct. +- [ ] Confirm address conflicts are rejected before activation. +- [ ] Enable the endpoints and observe gateway binding and readiness. +- [ ] Verify system-managed DNS publishes only ready endpoints. +- [ ] Test country, continent, and fallback Geo-Unicast answers. +- [ ] Withdraw one endpoint and confirm only its pool changes. +- [ ] Restore the endpoint and confirm DNS and gateway state converge. +- [ ] Test IPv4-only, IPv6-only, and dual-stack endpoint behavior. +- [ ] Restart/reconcile gateway, DNS, and cells and confirm consistent state. + +**Phase 4 result:** passed / failed / blocked / not activated + +# Phase 5 qualification — Simple Anycast + +Activate this section only when an approved multi-POP routing environment exists. + +- [ ] Create or edit an eligible pool with Simple Anycast routing mode. +- [ ] Configure one shared IPv4 and optional IPv6 service pair. +- [ ] Attach at least two participating POPs/edges. +- [ ] Confirm the same pair is bound locally on each participating edge. +- [ ] Confirm assigned domains publish the shared pair. +- [ ] Verify ready, degraded, and withdrawn states are clear. +- [ ] Test traffic from approved external vantage points. +- [ ] Remove one POP through the external routing process and record convergence. +- [ ] Confirm another POP continues serving valid local state. +- [ ] Restore the POP and record route and serving recovery. +- [ ] Confirm the UI states that the operator/provider owns BGP advertisement. +- [ ] Confirm no router credentials or arbitrary routing command surface exists. + +**Phase 5 result:** passed / failed / blocked / not activated + +# Phase 6 qualification — Cache v2 + +Activate this section when Cache v2 settings and status exist. + +- [ ] Select each available bounded pool cache profile. +- [ ] Configure cache size, temporary quota, inactive time, object limit, and free-space policy. +- [ ] Exercise include-all, ignore-all, include-selected, and ignore-selected query policies. +- [ ] Confirm deterministic cache-key behavior with repeated requests. +- [ ] Verify approved status-code TTL policies. +- [ ] Confirm MISS, HIT, BYPASS, STALE, and revalidation states where applicable. +- [ ] Exercise stale-if-error and stale-while-revalidate. +- [ ] Exercise cache-only and stale-only emergency modes. +- [ ] Restart a cell and confirm persistent cache behavior. +- [ ] Execute exact URL purge and full epoch purge. +- [ ] Trigger low-disk and high-variant protection in the approved lab. +- [ ] Confirm one domain cannot exhaust unrelated cache resources beyond declared limits. +- [ ] Verify disk, temporary storage, admissions, evictions, and hit-ratio visibility. + +**Phase 6 result:** passed / failed / blocked / not activated + +# Phase 7 qualification — Gzip and Brotli + +Activate this section when compression profiles and telemetry exist. + +- [ ] Test off, standard, and maximum-savings profiles where permitted. +- [ ] Request identical content with identity, Gzip, and Brotli support. +- [ ] Confirm decoded content is identical. +- [ ] Confirm one canonical cache object serves different encodings correctly. +- [ ] Verify MIME allowlist and minimum-size behavior. +- [ ] Confirm images, video, archives, and other compressed formats are not recompressed. +- [ ] Test HEAD, 304, ETag, Vary, stale, and purge behavior. +- [ ] Test range and large-response handling. +- [ ] Trigger compression concurrency or CPU-pressure limits in the approved lab. +- [ ] Confirm safe fallback or emergency disable without traffic interruption. +- [ ] Verify encoding, ratio, origin bytes, served bytes, savings, and fallback telemetry. +- [ ] Confirm shared pools cannot select unsafe compression settings. + +**Phase 7 result:** passed / failed / blocked / not activated + +# Phase 8 qualification — Origin failover + +Activate this section when backup-origin configuration exists. + +- [ ] Configure one valid primary and one valid backup origin. +- [ ] Confirm unsafe, looping, private/disallowed, and malformed origins are rejected. +- [ ] Verify normal traffic uses the primary. +- [ ] Trigger a qualified primary failure and observe transition to backup. +- [ ] Confirm transition reason and timestamps are visible without secrets. +- [ ] Restore primary health and verify hold-down and delayed failback. +- [ ] Confirm repeated health changes do not cause flapping. +- [ ] Fail both origins and verify stale, cache-only, or maintenance behavior. +- [ ] Stop the control plane and confirm local failover behavior remains available. +- [ ] Attempt invalid backup changes and confirm active valid state remains. +- [ ] Confirm one failing origin does not exhaust unrelated origin budgets. + +**Phase 8 result:** passed / failed / blocked / not activated + +# Phase 9 qualification — Managed OWASP CRS WAF + +Activate this section when managed WAF profiles exist. + +- [ ] Apply off, monitor, balanced, and strict profiles to approved test domains/pools. +- [ ] Confirm monitor mode records safe test detections without blocking. +- [ ] Confirm blocking profiles reject approved test attack patterns with stable reasons. +- [ ] Verify privacy-safe rule ID, category, score, action, and timing visibility. +- [ ] Add an approved bounded exclusion with reason, owner, and expiry. +- [ ] Confirm the exclusion affects only its intended scope. +- [ ] Confirm expired exclusions stop applying and remain audited. +- [ ] Verify arbitrary rules, raw directives, uploads, and runtime downloads are unavailable. +- [ ] Test oversized and malformed body limits in the approved lab. +- [ ] Run a monitor-only canary of a new WAF image/ruleset. +- [ ] Force canary failure and confirm the previous valid WAF runtime remains active. +- [ ] Confirm unrelated non-WAF pools remain healthy during WAF load or failure. + +**Phase 9 result:** passed / failed / blocked / not activated + +# Phase 10 qualification — Observability and capacity + +Activate this section when the new operational views and analytics exist. + +- [ ] Inspect gateway listener, route, map, connection, error, and revision status. +- [ ] Inspect service endpoint, Anycast, pool readiness, and placement status. +- [ ] Inspect cell CPU, memory, connections, cache disk, temporary space, and saturation. +- [ ] Inspect compression bytes, ratios, concurrency, and fallback. +- [ ] Inspect origin health, active origin, failover, and failback state. +- [ ] Inspect WAF profile, action, score, rule category, and processing cost. +- [ ] Confirm domain users see only assigned-domain data. +- [ ] Confirm raw logs and exports remain bounded and redacted. +- [ ] Stop ClickHouse/Vector and confirm serving continues with visible degraded analytics. +- [ ] Restore telemetry and verify bounded backlog recovery. +- [ ] Trigger representative alerts and confirm each links to an actionable runbook. +- [ ] Confirm queries remain responsive at the qualification dataset. + +**Phase 10 result:** passed / failed / blocked / not activated + +# Phase 11 qualification — Fleet rollout automation + +Activate this section when rollout management exists. + +- [ ] Inspect desired and current gateway, agent, normal-cell, and WAF-cell versions. +- [ ] Configure one canary edge/POP and later rollout waves. +- [ ] Start a canary rollout and observe progress and health gates. +- [ ] Confirm later waves do not start before canary success. +- [ ] Force a controlled canary failure and confirm automatic pause. +- [ ] Confirm the failure and pause reason are visible. +- [ ] Roll back and verify the previous compatible runtime is restored. +- [ ] Resume a successful rollout and verify bounded mixed-version serving. +- [ ] Confirm version drift and incompatibility are visible. +- [ ] Confirm no arbitrary command execution or unbounded container creation is exposed. +- [ ] Verify every rollout, pause, resume, and rollback is audited. + +**Phase 11 result:** passed / failed / blocked / not activated + +# Phase 12 qualification — Final production release + +Activate this section after Phases 1 through 11 are implemented and individually qualified. + +- [ ] Record at least two POPs/edges and at least eight cell slots per edge. +- [ ] Record one multi-cell shared pool, one reserved pool, and one quarantine pool. +- [ ] Verify several service pairs on the same edge. +- [ ] Verify Geo-Unicast and approved Simple Anycast traffic externally. +- [ ] Verify stable placement, migration, drain, quarantine, rollback, and recovery. +- [ ] Verify Cache v2 persistence, purge, stale behavior, and pressure controls. +- [ ] Verify Gzip and Brotli behavior and savings. +- [ ] Verify primary/backup origin failover and failback. +- [ ] Verify managed WAF monitor, block, exclusion, canary, and rollback. +- [ ] Stop selected control-plane and telemetry dependencies and confirm existing serving continues. +- [ ] Apply invalid gateway, cell, and WAF candidates and confirm last-valid state. +- [ ] Saturate one approved lab cell and confirm unrelated pools continue. +- [ ] Perform fleet canary upgrade and rollback. +- [ ] Perform clean-host control-plane restore and derived-state reconciliation. +- [ ] Run the completed baseline regression smoke suite. +- [ ] Confirm all relevant documentation and runbooks match the final product. +- [ ] Record measured limits, topology, hardware, RPO, RTO, throughput, latency, + saturation points, known limitations, and unresolved risks. +- [ ] Confirm no unresolved critical or high-severity failure remains. + +**Phase 12 result:** passed / failed / blocked / not activated + +## Release record ```text -Phase: -Checkpoint: -Expected: -Actual: -Evidence: -Related IDs: -Severity: -Owner: -Decision: -Retest date: -Retest result: +Roadmap phase: +Implementation: passed / failed / blocked +Unit and feature tests: passed / failed / blocked / not applicable +Real-runtime E2E: passed / failed / blocked / not applicable +IPv4 and IPv6: passed / failed / blocked / not applicable +Scale: passed / failed / blocked / not applicable +Failure and recovery: passed / failed / blocked / not applicable +Isolation: passed / failed / blocked / not applicable +Observability: passed / failed / blocked / not applicable +Documentation: passed / failed / blocked +Manual qualification: passed / failed / blocked +Baseline regression: passed / failed / blocked +Release decision: release / do not release +Evidence links: +Known limitations: +Owner approval: ``` - -A release remains unqualified until every active checkpoint passes or the product contract explicitly removes it. From a08ce46bf76b3778ac5840ebdb19ec02eb2f28ad Mon Sep 17 00:00:00 2001 From: vaheeD khoshnouD Date: Mon, 27 Jul 2026 15:03:33 +0330 Subject: [PATCH 6/7] ci: allow roadmap phase headings --- docs/.markdownlint.json | 3 +++ 1 file changed, 3 insertions(+) create mode 100644 docs/.markdownlint.json diff --git a/docs/.markdownlint.json b/docs/.markdownlint.json new file mode 100644 index 0000000..e4739a6 --- /dev/null +++ b/docs/.markdownlint.json @@ -0,0 +1,3 @@ +{ + "MD025": false +} From b74c19afeb4aa1e63a4f3bec3ad07938d74db733 Mon Sep 17 00:00:00 2001 From: vaheeD khoshnouD Date: Mon, 27 Jul 2026 15:07:02 +0330 Subject: [PATCH 7/7] ci: remove nested markdownlint override --- docs/.markdownlint.json | 3 --- 1 file changed, 3 deletions(-) delete mode 100644 docs/.markdownlint.json diff --git a/docs/.markdownlint.json b/docs/.markdownlint.json deleted file mode 100644 index e4739a6..0000000 --- a/docs/.markdownlint.json +++ /dev/null @@ -1,3 +0,0 @@ -{ - "MD025": false -}