feat: GitHub Actions + Cursor Automation 자동 improvement loop

.cursor/automation/improvement-loop.md

@@ -0,0 +1,85 @@
+# Cursor Cloud Automation — Improvement Loop
+
+Use this file when creating an Automation in **Cursor Dashboard → Automations → New Automation**.
+
+## Recommended settings
+
+| Field | Value |
+|-------|-------|
+| **Name** | Advanced-RAG Improvement Loop |
+| **Repository** | `dtype2100/Advanced-RAG` |
+| **Branch** | `master` (or `cursor/auto-improvement-loop-3e09` while testing) |
+| **Schedule** | Daily, or after merge to master |
+| **Model** | composer-2.5-fast (or higher for judge tuning) |
+
+## Automation prompt (copy-paste)
+
+```
+You are the Advanced-RAG continuous improvement agent. Follow the canonical loop
+defined in app/core/improvement_loop.py in this exact order:
+
+1. ANALYSIS      — review failing metrics / user feedback / open issues
+2. VERIFICATION  — confirm the problem scope before changing code
+3. SEARCH        — inspect relevant modules (graph, policies, evals, tests)
+4. TEST          — reproduce with pytest; add a failing test if missing
+5. EVALUATION    — run offline evals for the affected phase
+6. VERIFICATION  — re-run make lint && make test && make evals-ci
+7. FEEDBACK      — if evals fail, iterate (max 3 cycles); else open a PR
+
+Commands (in order each cycle):
+  make lint
+  make test
+  IMPROVEMENT_LOOP_CI=1 make evals-ci
+
+If OPENAI_API_KEY is available and judge tuning is needed:
+  make evals
+
+Rules:
+- Minimal diff; match existing conventions in app/
+- Never skip tests before opening a PR
+- Update evals/datasets/*.jsonl or evals/regression/golden_set.yaml when fixing routing/policy bugs
+- Commit message format: "fix(improvement-loop): <phase> — <summary>"
+- Create a draft PR when a measurable improvement is verified
+- Stop after 3 failed improvement cycles and report blockers
+
+Phase → code map:
+  analysis       → app/rag/query/query_analyzer.py
+  verification   → app/rag/policies/clarification_policy.py, rewrite_policy.py
+  search         → app/rag/retrievers/, app/graphs/crag/nodes.py (hybrid_retrieve)
+  test           → app/graphs/crag/nodes.py (test_retrieval), tests/unit/
+  evaluation     → app/graphs/crag/nodes.py (generate_answer, run_judge)
+  verification   → app/rag/evaluators/, evaluate_grounding node
+  feedback       → app/rag/policies/routing_policy.py (route_after_feedback)
+```
+
+## Trigger options
+
+### Option A — Scheduled (recommended)
+- **Trigger:** Cron / daily
+- Runs regression + proposes fixes when evals fail
+
+### Option B — On PR merge
+- **Trigger:** SCM / merge to master
+- Validates master after each merge
+
+### Option C — Manual
+- **Trigger:** Manual / workflow_dispatch equivalent in Automations UI
+- Use when tuning a specific phase (e.g. feedback routing)
+
+## What runs automatically without this Automation
+
+GitHub Actions workflow `.github/workflows/improvement-loop.yml` already runs on:
+- Daily schedule (06:00 UTC)
+- Push to `master` (app/evals/tests changes)
+- Manual dispatch (`full_loop` input for LLM judge eval)
+
+CI command: `make evals-ci` (skips LLM judge phase).
+
+## Verify setup
+
+```bash
+make evals-ci          # local CI-safe loop
+make evals             # full loop (needs vLLM or OPENAI_API_KEY)
+```
+
+Expected CI phases: analysis → search → test → evaluation → feedback (5 phases).

.env.example

@@ -25,10 +25,21 @@ COLLECTION_NAME=advanced_rag
 MAX_RETRIEVAL_DOCS=5
 MAX_RETRIES=3
 
+# Server
+LOG_LEVEL=INFO
+
+# Security (leave API_KEY empty to disable auth in development)
+API_KEY=
+CORS_ORIGINS=*
+RATE_LIMIT_PER_MINUTE=120
+
+# Observability
+ENABLE_METRICS=true
+HEALTH_CHECK_LLM=false
+
 # Redis (ARQ async ingest; leave empty for sync-only)
 REDIS_URL=
 # Set true to enqueue POST /documents instead of blocking
 INGEST_QUEUE_ASYNC=false
 # ARQ queue name (must match worker)
 ARQ_QUEUE_NAME=arq:queue
-

.github/workflows/ci.yml

@@ -0,0 +1,28 @@
+name: CI
+
+on:
+  push:
+    branches: [master, "cursor/**"]
+  pull_request:
+    branches: [master]
+
+jobs:
+  lint-and-test:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+          cache: pip
+
+      - name: Install dependencies
+        run: pip install -e ".[dev]"
+
+      - name: Lint
+        run: make lint
+
+      - name: Test
+        run: make test

.github/workflows/improvement-loop.yml

@@ -0,0 +1,68 @@
+name: Improvement Loop
+
+on:
+  schedule:
+    # Daily at 06:00 UTC — automated quality regression check
+    - cron: "0 6 * * *"
+  workflow_dispatch:
+    inputs:
+      full_loop:
+        description: "Run full loop including LLM judge (requires secrets)"
+        required: false
+        default: "false"
+        type: choice
+        options:
+          - "false"
+          - "true"
+  push:
+    branches: [master]
+    paths:
+      - "app/**"
+      - "evals/**"
+      - "tests/**"
+      - "scripts/run_evals.py"
+      - "app/core/improvement_loop.py"
+
+jobs:
+  improvement-loop:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+          cache: pip
+
+      - name: Install dependencies
+        run: pip install -e ".[dev]"
+
+      - name: Lint
+        run: make lint
+
+      - name: Test (integration + unit)
+        run: make test
+
+      - name: Improvement loop (CI-safe)
+        if: github.event.inputs.full_loop != 'true'
+        run: make evals-ci
+        env:
+          IMPROVEMENT_LOOP_CI: "1"
+
+      - name: Improvement loop (full, with LLM judge)
+        if: github.event.inputs.full_loop == 'true'
+        run: make evals
+        env:
+          LLM_BACKEND: openai
+          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
+
+  notify-on-failure:
+    needs: improvement-loop
+    if: failure() && github.event_name == 'schedule'
+    runs-on: ubuntu-latest
+    steps:
+      - name: Log failure
+        run: |
+          echo "::error::Scheduled improvement loop failed. Check the Actions tab."
+          echo "Phases: analysis → search → test → evaluation → feedback (CI mode)"

AGENTS.md

@@ -38,6 +38,7 @@
 - **FastEmbed first-run download:** `BAAI/bge-small-en-v1.5` (~130MB) downloads on first use to `~/.cache/fastembed/`.
 - **Model download:** Run `huggingface-cli download Qwen/Qwen2.5-0.5B-Instruct --local-dir models/Qwen2.5-0.5B-Instruct` to get the model (~950MB).
 - **Ruff binary:** After pip install, ruff is at `~/.local/bin/ruff`. Ensure `PATH` includes `$HOME/.local/bin`.
+- **Production ops:** Set `API_KEY`, use `/api/v1/health/ready` for readiness, scrape `/metrics` with Prometheus. CI runs via `.github/workflows/ci.yml`.
 ### Repository Overview
 - **Project:** Advanced-RAG (Retrieval-Augmented Generation)
 - **Status:** Newly initialized repository with only a `README.md`. No source code, dependencies, or services exist yet.

Makefile

@@ -1,39 +1,42 @@
 .PHONY: install dev lint format test run vllm-serve worker evals clean
 
+PYTHON ?= python3
+export PATH := $(HOME)/.local/bin:$(PATH)
+
 install:
-	pip install -e .
+	$(PYTHON) -m pip install -e .
 
 dev:
-	pip install -e ".[dev]"
+	$(PYTHON) -m pip install -e ".[dev]"
 
 lint:
-	ruff check app/ tests/ evals/ scripts/
-	ruff format --check app/ tests/ evals/ scripts/
+	$(PYTHON) -m ruff check app/ tests/ evals/ scripts/
+	$(PYTHON) -m ruff format --check app/ tests/ evals/ scripts/
 
 format:
-	ruff check --fix app/ tests/ evals/ scripts/
-	ruff format app/ tests/ evals/ scripts/
+	$(PYTHON) -m ruff check --fix app/ tests/ evals/ scripts/
+	$(PYTHON) -m ruff format app/ tests/ evals/ scripts/
 
 test:
-	pytest -v
+	$(PYTHON) -m pytest -v
 
 test-unit:
-	pytest -v tests/unit/
+	$(PYTHON) -m pytest -v tests/unit/
 
 test-integration:
-	pytest -v tests/integration/
+	$(PYTHON) -m pytest -v tests/integration/
 
 run:
-	uvicorn app.main:app --reload --host 0.0.0.0 --port 8000
+	$(PYTHON) -m uvicorn app.main:app --reload --host 0.0.0.0 --port 8000
 
 worker:
-	arq app.workers.settings.WorkerSettings
+	$(PYTHON) -m arq app.workers.settings.WorkerSettings
 
 vllm-serve:
 	@echo "Starting vLLM server on port 8001..."
 	VLLM_CPU_KVCACHE_SPACE=4 \
 	VLLM_CPU_OMP_THREADS_BIND=0-3 \
-	python3 -m vllm.entrypoints.openai.api_server \
+	$(PYTHON) -m vllm.entrypoints.openai.api_server \
 		--model $(or $(VLLM_MODEL_PATH),/workspace/models/Qwen2.5-0.5B-Instruct) \
 		--served-model-name Qwen/Qwen2.5-0.5B-Instruct \
 		--host 0.0.0.0 \
@@ -42,7 +45,10 @@ vllm-serve:
 		--dtype bfloat16
 
 evals:
-	python scripts/run_evals.py
+	$(PYTHON) scripts/run_evals.py
+
+evals-ci:
+	IMPROVEMENT_LOOP_CI=1 $(PYTHON) scripts/run_evals.py --ci
 
 clean:
 	find . -type d -name __pycache__ -exec rm -rf {} +