diff --git a/README.md b/README.md index ca35827..34c767f 100644 --- a/README.md +++ b/README.md @@ -1,3 +1,182 @@ +# The Digital Forge + +The Digital Forge is a multi-agent software generation system for turning a natural-language request into a tested Python implementation. It coordinates four specialized agents, executes generated code in an isolated sandbox, records repair attempts, and exposes both live run traces and precomputed benchmark evidence through a Next.js dashboard. + +The current revamp branch is `mjp/revamp-digital-forge`. + +## What It Does + +- Accepts a user request for a small Python feature or service. +- Builds a technical brief and development plan. +- Generates application code and tests. +- Runs the generated artifacts in a network-isolated sandbox. +- Classifies failures as application, test, infrastructure, or contract issues. +- Repairs failed candidates within a bounded attempt budget. +- Records generated artifacts, execution output, retrieval evidence, and the final report. +- Displays immutable benchmark reports without spending model tokens in the public dashboard. + +## Architecture + +```text +Next.js frontend + | + v +FastAPI backend + | + v +CrewAI orchestration +Janus -> Athena -> Hephaestus -> Argus + | | + v v + ChromaDB retrieval SandboxRunner + | | + v v + Docker Modal + local/CI hosted demo + | + v +Structured run and benchmark artifacts +``` + +Agent roles: + +- `Janus`: turns the request into a technical brief and final report. +- `Athena`: creates the typed development and testing plan. +- `Hephaestus`: writes and repairs application code. +- `Argus`: writes and repairs the generated test suite. + +The benchmark evaluator is independent from the agents. Agent-generated tests support development, but hidden benchmark tests determine correctness. + +## Current Benchmark Evidence + +The active benchmark suite is version `1.1.0` with 20 tasks: 10 easy and 10 medium. It measures the algorithmic and workflow-code path, not the separate RAG evaluation suite. + +| Run | Model | Result | Easy | Medium | Report | +| --- | --- | ---: | ---: | ---: | --- | +| Digital Forge | `digital-forge:gpt-4o-mini` | `16/20` | `9/10` | `7/10` | `benchmark-results/46e08fb1861141cf82fad76ef676ce5b/report.json` | +| Zero-shot baseline | `gpt-4o-mini` | `14/20` | `7/10` | `7/10` | `benchmark-results/3da7bfb192c84a47aedc04a1d20be0f7/report.json` | + +The dashboard loads these tracked report artifacts through `GET /benchmarks`. Full benchmark execution happens locally or in CI, not in the public demo. + +See `docs/BENCHMARK_GUARDED_RUN_2026_07_23.md` for the guarded run details. + +## Repository Layout + +```text +backend/ FastAPI app, run manager, pipeline integration +benchmark/ Benchmark catalog, runners, evaluator, hidden cases +benchmark-results/ Tracked benchmark report JSON files used by the dashboard +frontend/ Next.js App Router frontend +rag/ Versioned ChromaDB documentation index +tests/ Backend, pipeline, sandbox, benchmark, and API tests +docs/ Architecture, decisions, status, and benchmark notes +``` + +## Requirements + +- Python 3.10 or newer +- Node.js compatible with the pinned frontend dependencies +- Docker for local sandbox execution and benchmark reproduction +- OpenAI API key for live agent runs and benchmark generation + +Create a local `.env` file when running model-backed paths: + +```bash +OPENAI_API_KEY=... +OPENAI_MODEL_NAME=gpt-4o-mini +``` + +The `.env` file is ignored by Git. + +## Local Setup + +Backend: + +```bash +python -m venv .venv +. .venv/bin/activate +pip install -r requirements.txt +python -m backend.main +``` + +The backend defaults to `http://localhost:8000`. + +Frontend: + +```bash +cd frontend +npm install +npm run dev +``` + +The frontend defaults to `http://localhost:3000` and calls the backend at `http://localhost:8000`. Override that with `NEXT_PUBLIC_BACKEND_URL` if needed. + +## Running Benchmarks + +Zero-shot baseline: + +```bash +.venv/bin/python -m benchmark.baseline \ + --model gpt-4o-mini \ + --sandbox docker \ + --max-consecutive-failures 3 \ + --finish-remaining-threshold 3 +``` + +Digital Forge: + +```bash +.venv/bin/python -m benchmark.digital_forge \ + --model gpt-4o-mini \ + --sandbox docker \ + --max-consecutive-failures 3 \ + --finish-remaining-threshold 3 +``` + +Each completed task writes a checkpoint before the aggregate report is finalized. A full successful run writes `benchmark-results//report.json`. Guarded interrupted runs write `interrupted.json`. + +## Verification + +Python checks: + +```bash +.venv/bin/python -m pytest -q +.venv/bin/python -m ruff check backend benchmark rag tests +.venv/bin/python -m ruff format --check backend benchmark rag tests +.venv/bin/python -m mypy backend benchmark rag tests +``` + +Frontend checks: + +```bash +cd frontend +npm run lint +npm run typecheck +npm run build +``` + +Targeted benchmark/frontend checks used during the latest revamp: + +```bash +.venv/bin/python -m pytest tests/test_benchmarks.py tests/test_baseline.py tests/test_digital_forge_benchmark.py -q +cd frontend && npm run lint && npm run typecheck && npm run build +``` + +## Current Limits + +- Run snapshots are process-local and disappear when the backend restarts. +- Cancellation is cooperative and stops at workflow boundaries. +- Public deployment rate limits, spend controls, durable run storage, and one-run concurrency enforcement remain deployment work. +- Per-agent token and cost telemetry is not yet captured by the benchmark runner. +- The RAG layer is evaluated separately from the active 20-task algorithm benchmark. + +## Project Docs + +- `docs/STATUS.md`: current implementation status, verification, risks, and handoff notes. +- `docs/ARCHITECTURE.md`: system architecture and deployment model. +- `docs/DECISIONS.md`: accepted design decisions. +- `docs/WEEK_PLAN.md`: phased revamp plan. +- `docs/BENCHMARK_GUARDED_RUN_2026_07_23.md`: latest guarded benchmark evidence. # 🛠️ The Digital Forge: An Autonomous AI Software Development Crew **The Digital Forge** is a fully autonomous software development pipeline powered by a crew of specialized AI agents. Give it a natural language request, and the crew will work together to understand the requirements, write the code, create tests, and even debug their own work to deliver a final, validated solution.