feat: phase 1 — async ingestion, feedback loop, compose stack - #2
Open
rayperatta wants to merge 5 commits into
Open
rayperatta wants to merge 5 commits into
rayperatta wants to merge 5 commits into
Conversation
…nd compose
- Async ingestion: REDIS_URL activates arq queue; upload returns 202+job_id,
worker (worker.py) with retries (max_tries=3) and bounded concurrency;
GET /api/jobs/{job_id} for status. Sync fallback when REDIS_URL unset.
- Feedback loop: POST /api/feedback (+1/-1) persisted to data/feedback.jsonl
and mirrored as Langfuse trace score (Tracer.score_feedback);
/api/ask now returns trace_id; GET /api/feedback/summary aggregates.
- docker compose: API + worker + Redis one-command stack.
- Health endpoint reports ingestion_mode and feedback stats.
- Tests: 5 new (feedback store, async flag). Lint fixes for unused imports.
…t, not python -m pytest
- docs/rag-architecture.workflow.json: validated showcase spec (9/9 checks) - docs/rag-architecture.html: interactive viewer (themes, guided views, trace) - docs/rag-architecture.png: static render embedded in README
Light-themed 4-phase infographic (intake, hybrid retrieval, augmentation, generation) with async indexing and quality-loop side lanes, matching the reference style. The archify interactive viewer stays as docs/rag-architecture.html.
…ecision diamond Closer to the reference style: saturated phase-label columns, dotted connectors with joint dots, HYBRID_SEARCH decision diamond, icons in every box, dashed side lanes for async indexing and quality loop.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Phase 1 — Scale & reliability
Async ingestion (arq + Redis)
REDIS_URLactivates the queue:POST /api/upload→202 + job_id, worker processes in background (retriesmax_tries=3,max_jobs=4, backpressure via Redis)GET /api/jobs/{job_id}→ queued | in_progress | complete | failedREDIS_URL→ fully synchronous, same behaviour as before (feature-flag pattern, like Langfuse)Feedback loop
/api/askreturnstrace_idPOST /api/feedback(score +1/-1, optional comment) → local JSONL tuning dataset + Langfuse score mirroringGET /api/feedback/summary→ totals + thumbs-down rateOne-command stack
docker compose up→ API + arq worker + RedisVerification
ruff checkclean (0.7.0, same as CI)Pending in phase 1: publish baseline RAGAS metrics + p95 latency (requires API key for eval run).