Skip to content

Observability: Langfuse tracing + Postgres turn ledger + Grafana dashboards - #23

Merged
mr-j90 merged 3 commits into
masterfrom
feat/8-observability
Aug 9, 2026
Merged

mr-j90 merged 3 commits into
masterfrom
feat/8-observability

Conversation

@mr-j90

@mr-j90 mr-j90 commented Aug 9, 2026

Copy link
Copy Markdown
Owner

Parent

Closes #8SPEC §8

What this delivers

Every turn dual-writes: a Langfuse trace (self-hosted v3 via the compose observability profile — 6 pinned services, headless-init dev keys, trace id pinned to turn_id via an enclosing observation) and a flat Postgres row (model, tools, latency, tokens, cost, error). Feedback dual-writes by turn_id: row update + Langfuse score (flushed), replacing the JSONL sink. Grafana provisions a 7-panel dashboard as code over the turns table (volume, latency p50/p95, route split, feedback rate, cost/day, turn-error rate, tokens/day), anonymous read.

Verified live: traced turn → Postgres row with real tokens/cost; trace id == turn_id (8 observations); feedback → row + score on that same trace; dashboard served without login; row lands via BackgroundTask even on client disconnect.

Review notes (two-axis /code-review; both axes converged on three hard findings — all fixed)

  • UI feedback was silently landing nowhere: hooks refactor orphaned the turn_id patch; done handler now stores it (one line) and the API refuses to no-op silently (logs + skips the score when the row doesn't match, preventing store divergence).
  • Grafana datasource password never resolved (${VAR:-default} isn't Grafana syntax) — plain env ref now.
  • Client disconnects dropped the turn row — the write moved to a BackgroundTask, which Starlette runs even on disconnect; abandoned turns record error='client disconnected'.
  • Also: Opus pricing corrected 3×→ $5/$25 per MTok; deterministic price matching + loud unknown-model fallback; 'Tool-error rate' panel renamed 'Turn-error rate' (it measures turn errors — honest label until tool-level errors are worth plumbing).
  • Accepted/noted: hand-written pinned Langfuse stack instead of upstream's compose include (deliberate: one file, profile-gated, pinned); Langfuse span CM across generator yields can log a cosmetic detach warning on disconnect; per-write Postgres connections (fine at capstone volume).

mr-j90 and others added 3 commits August 8, 2026 22:51
…path

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…file

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Every turn dual-writes (SPEC §8): a Langfuse trace (self-hosted v3 via the
compose observability profile, headless-init dev keys, trace id pinned to
turn_id via an enclosing observation) and a flat Postgres row (model, tools,
latency, tokens, cost, error). Feedback dual-writes by turn_id: row update +
Langfuse score (flushed). Grafana provisions a 7-panel dashboard as code over
the turns table, anonymous read. Replaces the JSONL feedback sink.

Verified live: traced turn with exact count answer; trace/score/row all
keyed to one turn_id; dashboard served without login.

Closes #8

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@mr-j90
mr-j90 merged commit 4260861 into master Aug 9, 2026
2 checks passed
@mr-j90
mr-j90 deleted the feat/8-observability branch August 9, 2026 04:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Observability slice: Langfuse + Postgres dual-write + Grafana + feedback e2e

1 participant