Autonomous Store Provisioning Platform (Level-5 Auto Pilot)

A Kubernetes-native control plane designed for provisioning, scaling, and lifecycle-managing multi-tenant e-commerce stores on demand.

The platform features a Level-5 Auto Pilot AI Orchestrator powered by a self-hosted Medha vLLM inference engine, providing natural-language merchant intake and closed-loop (MAPE-K) autonomous Day-2 operations—built with a strict $0 budget constraint using 100% open-source software.

---

Technical Highlights

• Level-5 Operator Maturity: Fully automated lifecycle management (create, update, delete with finalizers, backup/restore), instrumented observability, and closed-loop AI remediation.
• Single-Writer Safety Model: The LLM has zero Kubernetes API credentials. The model emits typed JSON proposals; the deterministic Python Kopf Operator validates and disposes.
• Guided Decoding Engine: Serves 4-bit Gemma models via Medha vLLM using server-side response_format JSON schema enforcement, Pydantic validation, and capped repair loops.
• Day-2 Autopilot (MAPE-K Loop): Monitor → Analyze → Plan → Execute → Knowledge feedback loop driven by Prometheus metrics and Alertmanager signals, restricted to a constrained action catalog.
• Multi-Tenant Defense-in-Depth: Namespace-per-store isolation with ResourceQuota, LimitRange, default-deny NetworkPolicy, and per-tenant Secret generation.
• $0 Spend / Self-Hosted Stack: Completely independent of paid cloud APIs (OpenAI/Anthropic) and SaaS monitoring—runs on Medha vLLM, Prometheus, Grafana, Loki, and Redis.

---

Demo & Walkthroughs

Dashboard Interface

Provisioned Storefront

Video Walkthroughs

• 📽️ Architecture & Control Plane Overview
• 🛒 Store Provisioning & Order Execution
• 🧹 Lifecycle Teardown & Finalizer Cleanup
• 🔒 Security, Guardrails & Production Readiness
• 🎥 Dashboard Walkthrough

---

Architecture at a Glance

                 KUBERNETES-NATIVE STORE PROVISIONING PLATFORM
                    Current: AI-Assisted Store Provisioning

 Onboarding / Platform Admin
       │
       ├────────────── Structured Store Form ──────────────────────────────┐
       │                                                                   │
       └────────────── Merchant Brief ──────────────────────┐              │
                                                            │              │
                                                            ▼              ▼
┌──────────────────────────────────────────────────────────────────────────┐
│ Dashboard + FastAPI Platform API                                         │
│ Validation · Redis idempotency · rate/store caps · draft preview          │
└──────────────────────────────┬───────────────────────────────────────────┘
                               │
                ┌──────────────┴─────────────────┐
                │                                │
                │ Structured request             │ AI-assisted intake
                │                                ▼
                │              ┌──────────────────────────────────────────┐
                │              │ LangGraph + self-hosted Medha vLLM       │
                │              │ Gemma-4 26B AWQ 4-bit                    │
                │              │ sanitize → extract → strict JSON schema  │
                │              │ validate → preview → merchant confirm    │
                │              │                                          │
                │              │ LLM has no Kubernetes credentials.     │
                │              └─────────────────┬────────────────────────┘
                └────────────────────────────────┘
                                                 │ validated Store spec
                                                 ▼
┌──────────────────────────────────────────────────────────────────────────┐
│ Kubernetes API Server                                                    │
│ Store CR: desired configuration                                          │
│ Status: Provisioning · Ready · Failed · Stalled                          │
└──────────────────────────────┬───────────────────────────────────────────┘
                               │ watch and reconcile
                               ▼
┌──────────────────────────────────────────────────────────────────────────┐
│ Python Kopf Store Operator                                                │
│ Reconciles tenant infrastructure: namespace · quotas · NetworkPolicy     │
│ Secret · Helm upgrade --install · retry · backup · finalizer cleanup      │
│ Restore is explicit and requires a fresh request ID.                      │
└──────────────────────────────┬───────────────────────────────────────────┘
                               ▼
┌──────────────────────────────────────────────────────────────────────────┐
│ Isolated Namespace: store-<storeId>                                     │
│ WooCommerce / WordPress · MariaDB · PVCs · Service · Ingress             │
│ ResourceQuota · LimitRange · default-deny NetworkPolicy                  │
└──────────────────────────────┬───────────────────────────────────────────┘
                               │ metrics · logs · events · Store status
                               ▼
┌──────────────────────────────────────────────────────────────────────────┐
│ Self-Hosted Observability                                                 │
│ Prometheus · Grafana · Alertmanager · Loki + log collector               │
│ Dashboards, alerts, structured logs, backup/restore visibility           │
└──────────────────────────────────────────────────────────────────────────┘

---

Operator Capability Model (Level 1 → Level 5)

The platform implements the industry-standard Operator Capability Model maturity ladder:

Level	Capability	Status	Implementation Details
L1	Basic Install	✅ Done	Automated Store CR reconciliation into per-store namespaces with quota caps, NetworkPolicies, and Helm chart releases.
L2	Seamless Upgrades	✅ Done	Declarative spec.version and spec.tier updates applied via idempotent helm upgrade --install with zero data loss.
L3	Full Lifecycle	✅ Done	Finalizer teardown logic, automated MariaDB mysqldump CronJobs, dedicated backup PVCs, and explicit restore request handlers.
L4	Deep Insights	✅ Done	Prometheus /metrics endpoints (ports 9090/9100), Grafana dashboards, Loki log aggregation, and Alertmanager rules.
L5	Auto Pilot	✅ Done	Medha-driven intake graph, Day-2 MAPE-K loop, AutopilotPolicy enforcer, and RemediationAction audit/rollback.

---

System Architecture & Components

1. Control Plane & Operator

• Platform API (apps/api): Built with FastAPI. Provides RESTful endpoints for store lifecycle management, brief ingestion, rate limiting, and draft previews.
• Kopf Store Operator (controller/operator.py): Asynchronous Kubernetes operator managing Store CR lifecycle (stores.platform.storeprovisioner.com). Enforces idempotency via helm upgrade --install, concurrency limits (MAX_CONCURRENT_PROVISIONS), and graceful teardown with finalizers.

2. AI Surfaces & Guided Decoding

• AI Merchant Intake (apps/api/intake.py): Converts unstructured briefs into validated store specifications using a LangGraph state machine (extract → validate → repair). Uses vLLM server-side guided decoding (response_format: {type: json_schema}) to guarantee output schema validity.
• Day-2 Autopilot (MAPE-K Loop): Autonomous closed loop analyzing telemetry signals (CPU, memory, HTTP 5xx, latency, OOMKills) and proposing actions from a Constrained Action Catalog (scale_replicas, adjust_hpa_target, bump_quota, restart_workload, rollback_release, tune_resources). Governed by AutopilotPolicy (off, suggest, dry-run, auto) and audited in RemediationAction CRs.

3. Multi-Tenant Security & Guardrails

• Namespace-per-Store Isolation: Each store runs in store-<storeId>.
• Resource Limits: Enforces per-namespace ResourceQuota and LimitRange based on store tier (small, medium, large).
• Network Boundaries: Default-deny ingress/egress NetworkPolicy per store namespace. Medha LLM egress is strictly restricted to API control plane pods.

4. Observability & Operations

• Prometheus & Alertmanager: Collects operator metrics, API performance, and container statistics. Fires alerts on crash loops, storage pressure, and provisioning stalls.
• Loki & Promtail: Structured JSON log aggregation with strict label cardinality constraints.
• Automated Backup & Restore: Nightly MariaDB dumps to separate backup PVCs with single-use idempotent restore API processing.

---

Tech Stack

Category	Technology
Control Plane	Python 3.11, FastAPI, Kopf Operator Framework, Pydantic v2, Asyncio
Orchestration & Packaging	Kubernetes (CRDs), Helm 3, Docker, Kind / k3s
AI / LLM Stack	Medha vLLM (Gemma 4-bit), LangGraph, OpenAI Client API, Redis
Observability	Prometheus, Alertmanager, Grafana, Loki, Promtail, prometheus-adapter
Workloads	WooCommerce, WordPress, MariaDB, HPA, KEDA, VPA

---

Source Code Structure

├── apps/
│   ├── api/                 # FastAPI control plane, intake endpoints, Redis rate limits
│   └── dashboard/           # React / Vite merchant & admin dashboard UI
├── controller/
│   ├── operator.py          # Python Kopf operator (reconciliation & Kubernetes API writer)
│   └── Dockerfile           # Operator container build
├── charts/
│   ├── platform/            # Core control plane chart (API, Controller, CRDs, Ingress)
│   └── woocommerce/         # Bitnami WordPress / WooCommerce chart wrapper + bundled storefront theme
├── docs/
│   ├── ai-orchestrator.md   # Comprehensive Level-5 Auto Pilot & AI Orchestrator design
│   ├── ai-intake.md         # Merchant brief intake pipeline & guided decoding spec
│   ├── storefront.md        # Merchant storefront presets, branding boundary & seeding
│   ├── observability.md     # Metrics catalog, Grafana dashboards & Alertmanager rules
│   ├── operations.md        # Operations runbook, backup/restore drills & troubleshooting
│   └── system-design-tradeoffs.md # Architectural decisions & engineering tradeoffs
└── tests/                   # Pytest suite (operator reconciliation, API, intake, Lua caps)

---

Quick Start (Local Setup)

Prerequisites

• Docker Engine
• kubectl
• Helm 3
• Kind (kubernetes-in-docker)

One-Command Setup

Run the automated setup script to launch a local cluster and deploy the platform:

./setup.sh

Manual Helm Deployment

Alternatively, deploy using the local Helm values profile:

helm upgrade --install store-platform charts/platform \
  --namespace store-platform --create-namespace \
  -f charts/platform/values-local.yaml

Local Access Endpoints

• Dashboard UI: http://dashboard.127.0.0.1.nip.io
• Platform API Docs (Swagger): http://api.127.0.0.1.nip.io/docs

---

Usage Examples

1. Create a Store via API

curl -X POST http://api.127.0.0.1.nip.io/stores \
  -H 'Content-Type: application/json' \
  -d '{"engine":"woocommerce","storeId":"fashion-store","tier":"medium"}'

Optionally pick a storefront preset. The store comes up as a branded merchant site with a seeded catalogue and working Shop / Cart / Checkout / My Account routes, not the stock WordPress demo:

curl -X POST http://api.127.0.0.1.nip.io/stores \
  -H 'Content-Type: application/json' \
  -d '{"storeId":"fresh-mart","storefront":{"preset":"grocery","displayName":"Fresh Mart"}}'

preset is one of grocery (default), fashion, pharmacy. Only the preset and two plain-text strings are accepted — palette, theme, CSS, images and URLs are platform-controlled. See docs/storefront.md.

2. Submit a Natural-Language Merchant Brief (AI Intake)

curl -X POST http://api.127.0.0.1.nip.io/intake/briefs \
  -H 'Content-Type: application/json' \
  -d '{"brief": "I sell handmade leather goods, expecting around 1000 visitors daily. Need a medium tier store with WordPress 6.4."}'

Response returns a preview and a draftId. Confirming the draft provisions the store through the standard API path:

curl -X POST http://api.127.0.0.1.nip.io/intake/briefs/<DRAFT_ID>/confirm

3. Inspect Store Status & Events

curl http://api.127.0.0.1.nip.io/stores/fashion-store | jq
curl http://api.127.0.0.1.nip.io/stores/fashion-store/events | jq

4. Delete a Store

curl -X DELETE http://api.127.0.0.1.nip.io/stores/fashion-store

---

Verification & Testing

Run the automated Python test suite covering operator reconciliation, API guardrails, intake schemas, and Redis rate limits:

./scripts/test_in_image.sh

Run MariaDB automated backup and point-in-time recovery verification drills:

./scripts/backup_restore_drill.sh

---

Deep-Dive Documentation

For detailed architectural specifications, operational runbooks, and design tradeoffs:

• 📘 AI Orchestrator & Autopilot Design
• 🤖 AI Merchant Intake & Guided Decoding Spec
• 📊 Observability Stack & Metrics Catalog
• 🛠️ Operations Runbook & Backup Drills
• ⚖️ System Design Tradeoffs & Decisions