From d721a402afbec67590987e84df8b96b949a34664 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E2=80=9CAlex?= Date: Wed, 27 May 2026 15:43:12 -0700 Subject: [PATCH 1/2] Incorporated workflow into base agentops-stacks --- README.md | 51 +- plugin/README.md | 40 +- plugin/commands/agentops-lifecycle.md | 22 + plugin/skills/agentops-lifecycle/SKILL.md | 932 +++++++++++++++++++++ plugin/skills/install_skills.sh | 27 +- workflows/single-account-single-agent.json | 337 ++++++++ 6 files changed, 1386 insertions(+), 23 deletions(-) create mode 100644 plugin/commands/agentops-lifecycle.md create mode 100644 plugin/skills/agentops-lifecycle/SKILL.md create mode 100644 workflows/single-account-single-agent.json diff --git a/README.md b/README.md index 09da536..48383ee 100644 --- a/README.md +++ b/README.md @@ -90,7 +90,56 @@ Set workspace hosts and Unity Catalog grants per `docs/setup.md` in the rendered If the bundle was scaffolded inside a Databricks **Git folder** in the workspace (the recommended path for Genie Code users), the workspace UI also surfaces a **Deployments panel** on the bundle that lets you pick a target and deploy with one click — no terminal required. This matches the layout produced by the workspace UI's native "Create → Bundle" flow. -> **Coming next:** production lifecycle patterns (evaluation gates, governance posture, monitoring, feedback loops) are in active design — they'll ship as Innovate-driven workflows on top of the scaffold. Subscribe to releases for updates. +--- + +## Production lifecycle + +The scaffold generates the envelope. The **AgentOps Lifecycle** skill guides +the project through the complete Single-Account Single-Agent lifecycle — from +first data ingestion to production monitoring — across three phases and 10 steps: + +| Phase | Steps | What happens | +|-------|-------|--------------| +| **Dev** | 1–5 | Data prep + Vector Search indexing, agent implementation with MLflow tracing, offline eval gate, SME calibration | +| **Staging** | 6–7 | CI gate (unit tests + bundle validate + eval gate on PR), staging deploy + integration tests | +| **Production** | 8–10 | CD deploy, batch inferencing baseline, real-time monitoring + SME feedback loop | + +MLflow is the operational spine at every level: experiment tracking in dev, +eval gate in CI, trace logging in prod, user feedback via `mlflow.log_feedback()`. +The `evaluation/gate.py` pattern (generated by the scaffold) runs at three +points — locally in dev, in CI on every PR, and against production data before +users are admitted — so quality regressions are caught before they reach users. + +### Using the lifecycle skill + +Install the plugin, then ask your coding assistant to continue from where the +scaffold left off: + +```bash +# Install (Claude Code / Cursor) +./plugin/skills/install_skills.sh + +# Then in the assistant: +/agentops-lifecycle +# or: "walk me through the agentops lifecycle" +``` + +In **Genie Code**, run `install_genie_code_skills.py` as a notebook to install, +then say "walk me through the agentops lifecycle." + +The skill provides step-by-step agent actions, copy-paste code examples +grounded in the generated project files, validation criteria, and common issue +resolutions for each of the 10 steps. + +### Machine-readable workflow definition + +`workflows/single-account-single-agent.json` is a structured definition of the +10-step lifecycle with all actions, validations, escalation hints, and phase +metadata. It can be consumed programmatically by workflow engines (e.g., +`/innovate`) or used as a reference when building custom automation on top of +the scaffold. + +--- ## Documentation diff --git a/plugin/README.md b/plugin/README.md index 04c606e..371e108 100644 --- a/plugin/README.md +++ b/plugin/README.md @@ -4,15 +4,27 @@ A portable plugin that scaffolds AgentOps Stacks projects from inside a coding a The plugin and the [DAB template](../template/) share the same scaffold contract (`.agentops-stacks/manifest.yml`). The CLI alone is sufficient for scaffolding; the plugin is the additive on-ramp for users working inside a coding assistant. -## What this plugin ships today +## What this plugin ships -- One skill: **`agentops-stacks`** — scaffolds a new project (DAB layout, dev/staging/prod targets, UC schema and volume, MLflow experiment, CI/CD wiring for one of GitHub Actions, GitHub Actions for GHES, GitLab, or Azure DevOps). -- One command: **`/init-agentops-stacks`** — discoverability wrapper around the skill for Claude Code / Cursor users. -- Installers under `skills/` that mirror [ai-dev-kit's pattern](https://github.com/databricks-solutions/ai-dev-kit#installation): - - `install_skills.sh` — CLI installer. Lands the skill into `.claude/skills/agentops-stacks/` locally, with optional `--install-to-genie` to also upload to the workspace. - - `install_genie_code_skills.py` — Databricks notebook for in-workspace installs without a local clone. +### Skills -Production patterns (evaluation gates, governance posture, monitoring, feedback loops) are not in this plugin yet. They land as separate skills as the project matures. +| Skill | Trigger | What it does | +|-------|---------|--------------| +| **`agentops-stacks`** | `scaffold a new agentops project` | Scaffolds a new project (DAB layout, dev/staging/prod targets, UC schema and volume, MLflow experiment, CI/CD wiring). One-time use at project start. | +| **`agentops-lifecycle`** | `walk me through the agentops lifecycle` | Guides an existing scaffold through the complete Single-Account Single-Agent lifecycle — data prep, agent dev, eval gate, SME calibration, CI/CD promotion, batch eval baseline, and production monitoring. 10 steps across dev → staging → prod. | + +### Commands + +| Command | Skill | +|---------|-------| +| `/init-agentops-stacks` | `agentops-stacks` | +| `/agentops-lifecycle` | `agentops-lifecycle` | + +### Installers + +Under `skills/`, mirroring [ai-dev-kit's pattern](https://github.com/databricks-solutions/ai-dev-kit#installation): +- `install_skills.sh` — CLI installer. Lands skills into `.claude/skills/` locally, with optional `--install-to-genie` to also upload to the workspace. +- `install_genie_code_skills.py` — Databricks notebook for in-workspace installs without a local clone. ## Prerequisites @@ -82,15 +94,21 @@ Prerequisites: `databricks` CLI installed, a coding assistant that loads `.claud plugin/ ├── README.md # this file ├── commands/ -│ └── init-agentops-stacks.md # CC/Cursor slash command +│ ├── init-agentops-stacks.md # scaffold command +│ └── agentops-lifecycle.md # lifecycle command └── skills/ ├── install_skills.sh # local + Genie upload installer ├── install_genie_code_skills.py # in-workspace notebook installer - └── agentops-stacks/ - └── SKILL.md # skill instructions for the assistant + ├── agentops-stacks/ + │ └── SKILL.md # scaffold skill + └── agentops-lifecycle/ + └── SKILL.md # lifecycle skill (10-step dev→prod guide) ``` -The skill is a single `SKILL.md`. There's no Python renderer and no vendored template tree — `databricks bundle init` reads the template directly from this repo at scaffold time. +Each skill is a single `SKILL.md`. There's no Python renderer and no vendored +template tree — `databricks bundle init` reads the template directly from this +repo at scaffold time, and the lifecycle skill references generated project +files by their scaffold-relative paths. ## Known UX notes diff --git a/plugin/commands/agentops-lifecycle.md b/plugin/commands/agentops-lifecycle.md new file mode 100644 index 0000000..23924da --- /dev/null +++ b/plugin/commands/agentops-lifecycle.md @@ -0,0 +1,22 @@ +--- +description: > + Guide an agentops-stacks project through its complete production lifecycle — + data prep, agent dev, eval gates, CI/CD promotion, and production monitoring. + Run after `databricks bundle init` (or `/init-agentops-stacks`). +--- + +Use the `agentops-lifecycle` skill. + +This command guides an existing agentops-stacks scaffold through the +Single-Account Single-Agent lifecycle: + +- Steps 1–5 (Dev): data preparation, agent implementation, offline evaluation, + SME calibration +- Steps 6–7 (Staging): CI gate, integration tests +- Steps 8–10 (Production): CD deploy, batch inferencing baseline, monitoring + +**Prerequisite:** `.agentops-stacks/manifest.yml` must exist. Run +`/init-agentops-stacks` first if you haven't scaffolded yet. + +Defer to the skill's SKILL.md for full step-by-step guidance, code examples, +validation criteria, and common issue resolutions. diff --git a/plugin/skills/agentops-lifecycle/SKILL.md b/plugin/skills/agentops-lifecycle/SKILL.md new file mode 100644 index 0000000..5c9f05d --- /dev/null +++ b/plugin/skills/agentops-lifecycle/SKILL.md @@ -0,0 +1,932 @@ +--- +name: agentops-lifecycle +description: > + Guide an agentops-stacks project through its full production lifecycle — data + preparation, agent development, evaluation gates, CI/CD promotion, and + production monitoring — following the Single-Account Single-Agent pattern from + the Big Book of AgentOps. Use after `databricks bundle init` has been run and + `.agentops-stacks/manifest.yml` exists. Triggers on "walk me through the + agentops lifecycle", "next step after scaffolding", "set up eval gate", + "deploy agent to staging", "wire production monitoring". +--- + +# AgentOps Lifecycle — Single-Account Single-Agent + +## Overview + +This skill guides a project scaffolded with agentops-stacks through its complete +production lifecycle: 10 steps across three phases (dev → staging → prod). +MLflow is the operational spine at every level. The `evaluation/gate.py` pattern +from the scaffold blocks every promotion — it runs locally in dev, in CI on every +PR, and against real production data before users are admitted. + +**Before using this skill:** run `databricks bundle init` (via the +`agentops-stacks` skill or directly) and confirm `.agentops-stacks/manifest.yml` +exists in the project root. + +### Architecture + +``` +Git provider +───────────────────────────────────────────────────────────────────────────── + feature branch ──commit──► PR to main ──CI gate──► main ──tag/release──► CD + │ │ │ + ▼ ▼ ▼ +┌─────────────────┐ ┌──────────────────────┐ ┌────────────────────────────┐ +│ DEV WORKSPACE │ │ STAGING WORKSPACE │ │ PRODUCTION WORKSPACE │ +│ │ │ │ │ │ +│ Data Prep │ │ Unit Tests (CI) │ │ App + Model Serving │ +│ └─ Ingest │ │ Bundle Validate │ │ Batch Inferencing Job │ +│ └─ Embed │ │ Eval Gate (CI) │ │ Automated Eval │ +│ └─ VS Index │ │ Integration Tests │ │ SME HITL sampling │ +│ │ │ Validation Tests │ │ Monitoring Dashboard │ +│ Agent Dev │ │ Staging MLflow │ │ Feedback Loop │ +│ └─ Tools │ └──────────────────────┘ └────────────────────────────┘ +│ └─ Agent Code │ +│ └─ MLflow │ Unity Catalog (one metastore, three catalogs) +│ Traces │ ────────────────────────────────────────────────────────── +│ │ dev catalog → dev compute only +│ Offline Eval │ staging catalog → staging compute only +│ SME HITL (dev) │ prod catalog → READ-ONLY from dev; prod agent exclusive +└─────────────────┘ +``` + +### Phase overview + +| Phase | Steps | Entry gate | Exit gate | +|-------|-------|------------|-----------| +| Dev | 1–5 | scaffold exists | SME sign-off + local eval gate passing | +| Staging | 6–7 | PR opened | all CI checks green + integration tests pass | +| Production | 8–10 | CD triggered | smoke test + batch eval baseline + monitoring live | + +--- + +## Step 1 — Scaffold AgentOps Project + +**Do this once at project start.** Bootstrap the production envelope that the +entire lifecycle runs inside. + +If the scaffold already exists (`.agentops-stacks/manifest.yml` present), skip +to Step 2. + +### Run + +```bash +# Non-interactive scaffold — fill in your values +cat > /tmp/agentops-stacks-inputs.json <<'EOF' +{ + "input_project_name": "my_agent", + "input_cloud": "aws", + "input_cicd_platform": "github_actions" +} +EOF + +databricks bundle init https://github.com/databricks-solutions/agentops-stacks \ + --config-file /tmp/agentops-stacks-inputs.json \ + --output-dir . + +cd my_agent +uv sync +databricks bundle validate -t dev +``` + +### Done when + +- `.agentops-stacks/manifest.yml` exists containing `contract_version`, + `project_name`, `cicd_platform`, `cloud`. +- `databricks bundle validate -t dev` exits 0. +- `uv.lock` is present (commit it). + +--- + +## Step 2 — Data Preparation & Vector Search Indexing + +Build the data foundation for the agent. Unoptimized retrieval degrades every +downstream step — hybrid search (BM25 + semantic) and metadata filters are +non-negotiable from the start. + +> **Note:** Vector Search indexes are not yet a DAB resource type. Create the +> index via notebook until DAB support lands. Document the creation notebook +> path in a comment in `databricks.yml` so it's findable. + +### Ingestion notebook pattern + +```python +# notebooks/01_ingest.py — run against dev workspace +# Databricks notebook source + +# COMMAND ---------- +# Ingest and parse source documents +from databricks.sdk import WorkspaceClient +import mlflow + +mlflow.set_experiment("/Shared/my_agent/data_prep") + +w = WorkspaceClient() + +# For unstructured documents (PDFs, DOCX, HTML): +# ai_parse_document is a Databricks AI Function available in SQL +spark.sql(""" + CREATE OR REPLACE TABLE my_agent_dev.my_agent.raw_docs AS + SELECT + path, + ai_parse_document(content) AS parsed + FROM read_files('/Volumes/my_agent_dev/my_agent/raw/', format => 'binaryFile') +""") + +# COMMAND ---------- +# Chunk and embed for Vector Search +spark.sql(""" + CREATE OR REPLACE TABLE my_agent_dev.my_agent.chunked_docs AS + SELECT + path, + chunk_index, + chunk_text, + ai_embed_text(chunk_text) AS embedding + FROM ( + SELECT + path, + posexplode( + ai_chunk_text(parsed.content, 512, 64) + ) AS (chunk_index, chunk_text) + FROM my_agent_dev.my_agent.raw_docs + ) +""") +``` + +### Vector Search index + +```python +# Create index via SDK — not yet a DAB resource type +from databricks.sdk import WorkspaceClient +from databricks.sdk.service.vectorsearch import VectorIndexType, DeltaSyncVectorIndexSpecRequest, EmbeddingSourceColumn + +w = WorkspaceClient() + +w.vector_search_indexes.create( + name="my_agent_dev.my_agent.docs_index", + endpoint_name="my_agent_vs_endpoint", # pre-existing VS endpoint + primary_key="path", + index_type=VectorIndexType.DELTA_SYNC, + delta_sync_index_spec=DeltaSyncVectorIndexSpecRequest( + source_table="my_agent_dev.my_agent.chunked_docs", + pipeline_type="TRIGGERED", + embedding_source_columns=[ + EmbeddingSourceColumn( + name="chunk_text", + embedding_model_endpoint_name="databricks-gte-large-en", + ) + ], + ), +) +``` + +### Done when + +- `databricks bundle validate -t dev` still passes after any new resources are + added to `databricks.yml`. +- VS index is queryable: `w.vector_search_indexes.query_index("my_agent_dev.my_agent.docs_index", columns=["path", "chunk_text"], query_text="test query")` returns results. +- Ingestion notebook runs end-to-end on sample data without errors. + +--- + +## Step 3 — Agent Development & Dev Deployment + +Implement the agent with `@mlflow.trace` on every decision point. MLflow tracing +is **mandatory from the first deploy** — traces are required for eval gate +feedback, SME review, and production monitoring. An agent with no traces cannot +be evaluated. + +### Agent class pattern + +```python +# src/my_agent.py +import mlflow +from mlflow.pyfunc import PythonModel +from databricks.sdk import WorkspaceClient +import pandas as pd + + +class MyAgent(PythonModel): + """Production agent with full MLflow tracing and guardrails.""" + + def __init__(self): + self._vs_client = None + + @property + def vs_client(self): + if self._vs_client is None: + self._vs_client = WorkspaceClient() + return self._vs_client + + @mlflow.trace(name="predict", span_type="CHAIN") + def predict(self, context, model_input: pd.DataFrame) -> list[dict]: + rows = model_input.to_dict(orient="records") + return [self._handle(row) for row in rows] + + @mlflow.trace(span_type="CHAIN") + def _handle(self, row: dict) -> dict: + query = self._validate_input(row.get("query", "")) + context_docs = self._retrieve(query) + response = self._generate(query, context_docs) + return {"response": response, "sources": [d["path"] for d in context_docs]} + + @mlflow.trace(span_type="GUARDRAIL") + def _validate_input(self, query: str) -> str: + # PII scrub, injection detection, length limits + if not query or len(query) > 2000: + raise ValueError("Query must be 1–2000 characters") + return query.strip() + + @mlflow.trace(span_type="RETRIEVER") + def _retrieve(self, query: str) -> list[dict]: + import os + catalog = os.environ["DATABRICKS_CATALOG"] + schema = os.environ.get("DATABRICKS_SCHEMA", "my_agent") + result = self.vs_client.vector_search_indexes.query_index( + index_name=f"{catalog}.{schema}.docs_index", + columns=["path", "chunk_text"], + query_text=query, + num_results=5, + ) + return [ + {"path": r.get("path", ""), "text": r.get("chunk_text", "")} + for r in (result.result.data_array or []) + ] + + @mlflow.trace(span_type="LLM") + def _generate(self, query: str, docs: list[dict]) -> str: + context = "\n\n".join(d["text"] for d in docs) + # Replace with your actual LLM call (e.g., MLflow AI Gateway, SDK) + from mlflow.deployments import get_deploy_client + client = get_deploy_client("databricks") + response = client.predict( + endpoint="databricks-meta-llama-3-1-70b-instruct", + inputs={ + "messages": [ + {"role": "system", "content": f"Answer using this context:\n{context}"}, + {"role": "user", "content": query}, + ] + }, + ) + return response["choices"][0]["message"]["content"] +``` + +### Register to Unity Catalog + +Run `notebooks/register_agent.py` (generated by the scaffold). Set the +`catalog` and `schema` widgets before running: + +```python +# In the notebook (already generated by scaffold): +# catalog = "my_agent_dev" +# schema = "my_agent" +# model_name = f"{catalog}.{schema}.my_agent" +``` + +After running, verify: + +```bash +# Check @champion alias is set +databricks models get-alias my_agent_dev.my_agent.my_agent champion +``` + +### Deploy dev bundle + +```bash +databricks bundle deploy -t dev +uv run pytest # exit 0 or exit 5 (no tests) both acceptable at this stage +``` + +### Done when + +- Model exists in Unity Catalog with `@champion` alias: `models:/my_agent_dev.my_agent.my_agent@champion`. +- At least one MLflow trace is present in the dev experiment for a sample prediction. +- `databricks bundle deploy -t dev` exits 0. +- If a Databricks App is declared in `databricks.yml`, the App URL is reachable in the dev workspace. + +--- + +## Step 4 — Offline Evaluation & Eval Gate Setup + +Build the evaluation framework **before** any code leaves dev. The `evaluation/` +directory is generated by the scaffold. Populate it and verify the gate passes +locally — it will run in CI on every PR. + +### Golden dataset + +`evaluation/golden_dataset.jsonl` — one JSON object per line: + +```jsonl +{"query": "What is the return policy?", "expected_response": "Items can be returned within 30 days with receipt.", "context": "Optional: known good context chunk"} +{"query": "How do I reset my password?", "expected_response": "Visit account settings and click 'Forgot password'.", "context": ""} +``` + +**Minimum:** 20 labeled examples. Target: 50–100. More examples = more stable +eval scores. SME or domain expert should label the `expected_response` values. + +### Thresholds + +`evaluation/thresholds.yml` (generated by scaffold — adjust thresholds after +first baseline run): + +```yaml +model: + uri: "models:/{catalog}.{schema}.my_agent@champion" + +dataset: + path: "evaluation/golden_dataset.jsonl" + +scorers: + - name: Safety + severity: blocking # hard failure: gate blocks CI + threshold: 1.0 # all responses must be Safe + - name: Correctness + severity: warning # soft failure: gate warns but does not block + threshold: 0.8 # >= 80% match expected_response +``` + +### Run the gate locally + +```bash +export DATABRICKS_CATALOG=my_agent_dev +export DATABRICKS_SCHEMA=my_agent +uv run python evaluation/gate.py +``` + +Expected output on pass: +``` +Loading model: models:/my_agent_dev.my_agent.my_agent@champion +Evaluating against 50 examples with 2 scorer(s) +PASS — Safety: 1.000 (threshold 1.000, blocking) +PASS — Correctness: 0.863 (threshold 0.800, warning) +Eval gate: all blocking thresholds met. +``` + +If Safety < 1.0, review flagged traces in MLflow, add output guardrails to the +agent's `_validate_input` or response post-processing, then re-evaluate. **Do +not lower the Safety threshold to pass.** + +### Custom scorer (optional) + +```python +# In evaluation/gate.py — extend SCORER_REGISTRY with domain-specific criteria +from mlflow.genai import make_judge + +domain_scorer = make_judge( + name="AnswerGrounded", + judge_prompt=( + "Is the answer grounded in the provided context? " + "Score 1 if fully grounded, 0 if hallucinated." + ), + score_type="int", +) + +SCORER_REGISTRY["AnswerGrounded"] = lambda: domain_scorer +``` + +### Done when + +- `evaluation/golden_dataset.jsonl` has ≥20 labeled examples. +- `uv run python evaluation/gate.py` exits 0 — all blocking thresholds met. +- MLflow experiment has at least one eval run with `safety/mean` and + `correctness/mean` metrics. + +--- + +## Step 5 — SME Human-in-the-Loop (Dev) + +Calibrate the LLM judge against real domain expert judgment **before** staging. +An uncalibrated judge that passes CI is worse than no judge — it becomes a +rubber stamp. + +### Export traces for SME review + +```python +import mlflow +import pandas as pd + +client = mlflow.tracking.MlflowClient() +experiment = client.get_experiment_by_name("/Shared/my_agent/dev") + +# Pull recent traces for review +runs = client.search_runs( + experiment_ids=[experiment.experiment_id], + max_results=50, + order_by=["start_time DESC"], +) + +traces = [] +for run in runs: + for trace in client.search_traces(experiment_ids=[experiment.experiment_id], + filter_string=f"run_id = '{run.info.run_id}'", + max_results=1): + spans = trace.data.spans + predict_span = next((s for s in spans if s.name == "predict"), None) + if predict_span: + traces.append({ + "trace_id": trace.info.request_id, + "query": predict_span.inputs.get("query", ""), + "response": predict_span.outputs.get("response", ""), + }) + +df = pd.DataFrame(traces) +df.to_csv("evaluation/traces_for_sme_review.csv", index=False) +print(f"Exported {len(df)} traces to evaluation/traces_for_sme_review.csv") +``` + +Share the CSV (or a Databricks App backed by the MLflow Trace UI) with the +domain SME. Add score columns for them to fill: + +| trace_id | query | response | sme_accuracy_1_5 | sme_tone_1_5 | sme_complete_1_5 | sme_safe_pass_fail | +|---|---|---|---|---|---|---| + +### Calibration run + +After SME scoring is returned, compare against judge scores: + +```python +# Run evaluation on the same traces using the LLM judge +result = mlflow.genai.evaluate( + data=pd.read_csv("evaluation/traces_for_sme_review.csv"), + predict_fn=lambda q: ..., # re-invoke agent on the same queries + scorers=[mlflow.genai.scorers.Correctness(), mlflow.genai.scorers.Safety()], +) + +# Compare judge scores to SME scores — compute agreement rate +import numpy as np +sme_df = pd.read_csv("evaluation/traces_for_sme_review.csv") +judge_scores = result.tables["eval_results"] + +# Agreement: abs(judge_correctness - sme_accuracy/5) < 0.2 +agreement = ( + (judge_scores["correctness/score"] - sme_df["sme_accuracy_1_5"] / 5).abs() < 0.2 +).mean() +print(f"Judge-SME agreement: {agreement:.1%}") +``` + +Target: ≥80% agreement. If <80%, refine the Correctness judge prompt using +disagreement examples as few-shots via `mlflow.genai.align()`. + +### Sign-off document + +Create `evaluation/sme_calibration.md`: + +```markdown +# SME Calibration Sign-Off + +- **Reviewer:** Jane Smith (jane.smith@company.com) +- **Date:** 2025-06-01 +- **Traces reviewed:** 50 +- **Agreement rate:** 84% +- **Judge threshold adjustments:** Correctness raised from 0.80 → 0.82 post-calibration +- **Sign-off:** ✓ Judge calibrated. Approved for promotion to staging. +``` + +### Done when + +- `evaluation/sme_calibration.md` exists with reviewer name, date, agreement %, and explicit sign-off. +- Agreement rate ≥80% (documented). + +--- + +## Step 6 — CI Gate: PR to Main + +Open the PR to main. The CI workflow runs `unit_tests`, `validate_bundle`, and +`eval_gate` in a clean environment. **Do not skip or suppress CI checks.** + +### Open the PR + +```bash +git checkout -b feature/my_agent_initial +git add -A +git commit -m "[my_agent] Initial implementation + eval gate" +git push -u origin feature/my_agent_initial + +gh pr create \ + --title "[my_agent] Initial implementation + eval gate" \ + --body "Adds agent code, golden dataset (50 examples), eval gate (Safety 1.0, Correctness 0.82), and SME calibration sign-off." +``` + +### What CI runs + +The CI workflow (`.github/workflows/my_agent-bundle-ci.yml`, generated by +scaffold) runs three jobs: + +```yaml +# Generated by scaffold — do not modify the gate-triggering logic +jobs: + unit_tests: + # uv run pytest + validate_bundle: + # databricks bundle validate -t staging + eval_gate: + # if: hashFiles('evaluation/thresholds.yml') != '' + # uv run python evaluation/gate.py + # env: + # DATABRICKS_CATALOG: ${{ vars.STAGING_CATALOG }} + # DATABRICKS_SCHEMA: ${{ vars.STAGING_SCHEMA }} +``` + +CI secrets required (set in GitHub repo settings before opening the PR): +- `DATABRICKS_STAGING_HOST` — staging workspace URL +- `DATABRICKS_STAGING_TOKEN` or OIDC wiring for the service principal +- `STAGING_CATALOG` variable — e.g., `my_agent_staging` + +### Address failures + +- **`validate_bundle` fails:** check staging workspace host in `databricks.yml` + targets.staging.workspace.host; verify secrets are configured in CI. +- **`eval_gate` fails:** fix the agent — do not disable the gate or lower + thresholds to pass. A CI gate failure is a signal that the agent regressed. + +### Done when + +- All CI jobs green: `unit_tests`, `validate_bundle`, `eval_gate`. +- PR merged to main (squash merge). + +--- + +## Step 7 — Staging Deployment & Integration Tests + +Deploy to staging and run the full integration + validation test suites. +Staging mirrors production configuration — this is where cross-component +integration is verified against real endpoints. + +### Deploy to staging + +CD is triggered automatically on merge to main by +`.github/workflows/my_agent-bundle-cd-staging.yml`. To trigger manually: + +```bash +databricks bundle deploy -t staging +``` + +### Integration tests + +`tests/` contains unit tests generated by the scaffold. Add integration tests +that call real staging endpoints: + +```python +# tests/test_agent_integration.py +import os +import mlflow +import pandas as pd +import pytest + +@pytest.fixture(scope="session") +def staging_model(): + catalog = os.environ["DATABRICKS_CATALOG"] # my_agent_staging + schema = os.environ.get("DATABRICKS_SCHEMA", "my_agent") + return mlflow.pyfunc.load_model(f"models:/{catalog}.{schema}.my_agent@champion") + +def test_agent_returns_response(staging_model): + result = staging_model.predict(pd.DataFrame([{"query": "What is the return policy?"}])) + assert isinstance(result, list) + assert len(result) == 1 + assert "response" in result[0] + assert len(result[0]["response"]) > 0 + +def test_agent_handles_empty_query(staging_model): + """Agent should raise or return a structured error, not crash.""" + with pytest.raises(Exception): + staging_model.predict(pd.DataFrame([{"query": ""}])) + +def test_mlflow_traces_logged(staging_model): + """Each prediction must produce an MLflow trace.""" + import mlflow + client = mlflow.tracking.MlflowClient() + experiment = client.get_experiment_by_name( + f"/Shared/my_agent/staging" + ) + before_count = len(client.search_traces( + experiment_ids=[experiment.experiment_id], max_results=1000 + )) + staging_model.predict(pd.DataFrame([{"query": "trace check"}])) + after_count = len(client.search_traces( + experiment_ids=[experiment.experiment_id], max_results=1000 + )) + assert after_count > before_count, "Prediction did not log an MLflow trace" +``` + +Run locally against staging: + +```bash +export DATABRICKS_CATALOG=my_agent_staging +uv run pytest tests/test_agent_integration.py -v +``` + +### Done when + +- Databricks App and Model Serving endpoint are live in staging workspace. +- Staging MLflow experiment has at least one trace from the integration test run. +- All integration tests pass. +- All validation tests pass (edge cases, schema validation, timeout behaviors). + +--- + +## Step 8 — Production Deployment via CD + +Merge to the release branch (or tag a release) to trigger CD to production. + +### Trigger CD + +```bash +# Create a release tag on main +git checkout main +git pull +git tag v1.0.0 +git push origin v1.0.0 +# GitHub Actions picks up the v* tag and runs the CD prod workflow +``` + +Or: merge `main` → `release` branch if your CD is branch-triggered. + +### Verify deployment + +```bash +# Monitor CD via GitHub Actions or run manually +databricks bundle deploy -t prod + +# Smoke test — confirm agent responds +databricks serving-endpoints query my_agent_prod \ + --request '{"dataframe_records": [{"query": "smoke test query"}]}' +``` + +The endpoint is healthy when it returns HTTP 200 on `/health`. The smoke test +passes when it returns a non-empty `response` field. + +### Register production model + +```bash +# Run notebooks/register_agent.py with prod catalog/schema widgets +# Then verify @champion alias in prod UC +databricks models get-alias my_agent_prod.my_agent.my_agent champion +``` + +### Done when + +- CD workflow exits 0 and prod bundle is deployed. +- Model Serving endpoint returns HTTP 200 on `/health`. +- Smoke test: agent returns non-error, non-empty response. +- Prod UC schema has model with `@champion` alias. + +--- + +## Step 9 — Batch Inferencing & Production Eval Baseline + +Run batch inferencing on a representative production dataset to establish a +quality baseline **before opening to users**. Production data distributions +differ from golden datasets — the batch eval is the first real-world quality +check. + +### Batch inferencing job + +Add a Databricks Job to `resources/` for batch inference: + +```yaml +# resources/batch_inference_job.yml +resources: + jobs: + batch_inference: + name: "${bundle.name}_batch_inference" + tasks: + - task_key: run_batch + notebook_task: + notebook_path: notebooks/batch_inference.py + base_parameters: + input_table: "${var.catalog}.${var.schema}.batch_eval_inputs" + output_table: "${var.catalog}.${var.schema}.batch_eval_outputs" + model_uri: "models:/${var.catalog}.${var.schema}.${bundle.name}@champion" +``` + +```python +# notebooks/batch_inference.py +# Databricks notebook source + +# COMMAND ---------- +import mlflow +import pandas as pd +from pyspark.sql import functions as F + +dbutils.widgets.text("input_table", "") +dbutils.widgets.text("output_table", "") +dbutils.widgets.text("model_uri", "") + +input_table = dbutils.widgets.get("input_table") +output_table = dbutils.widgets.get("output_table") +model_uri = dbutils.widgets.get("model_uri") + +# COMMAND ---------- +# Load model once and broadcast across partitions +model = mlflow.pyfunc.load_model(model_uri) + +input_df = spark.table(input_table).toPandas() +results = model.predict(input_df[["query"]]) + +output_df = pd.DataFrame(results) +output_df["query"] = input_df["query"].values + +spark.createDataFrame(output_df).write.mode("overwrite").saveAsTable(output_table) +print(f"Batch inference complete: {len(output_df)} rows written to {output_table}") +``` + +### Run eval gate on batch output + +```bash +# Point the gate at batch results instead of golden dataset +export DATABRICKS_CATALOG=my_agent_prod +export DATABRICKS_SCHEMA=my_agent +# Override dataset path via environment or edit thresholds.yml temporarily +uv run python evaluation/gate.py +``` + +### Log baseline metrics + +Create `evaluation/production_baseline.md`: + +```markdown +# Production Eval Baseline + +- **Date:** 2025-06-15 +- **Dataset:** my_agent_prod.my_agent.batch_eval_inputs (500 rows) +- **Model:** models:/my_agent_prod.my_agent.my_agent@champion (v1) + +| Metric | Value | +|---|---| +| P95 Latency | 1.2s | +| Safety/mean | 1.00 | +| Correctness/mean | 0.84 | +| Cost/request | $0.003 | + +Staging baseline for comparison: Correctness 0.82. Production +0.02 — within bounds. +``` + +### Done when + +- Batch inferencing job completes without errors. +- Eval gate passes on production batch results — all blocking thresholds met. +- `evaluation/production_baseline.md` has P95 latency, Safety/mean, Correctness/mean, cost/request. + +--- + +## Step 10 — Production Monitoring & Continuous Feedback Loop + +Wire the complete production observability stack. This step closes the +continuous improvement loop: production traces feed back into the eval dataset, +driving future iterations. + +### Verify MLflow autolog on the endpoint + +The Model Serving endpoint should have `MLFLOW_TRACKING_URI` wired to the prod +MLflow server. Verify by checking the endpoint environment in `databricks.yml`: + +```yaml +# In resources/model_serving.yml (or wherever the serving endpoint is declared) +resources: + model_serving_endpoints: + my_agent_endpoint: + name: "${bundle.name}_endpoint" + config: + served_models: + - model_name: "${var.catalog}.${var.schema}.${bundle.name}" + model_version: "1" + workload_size: Small + scale_to_zero_enabled: true + auto_capture_config: + catalog_name: "${var.catalog}" + schema_name: "${var.schema}" + table_name_prefix: "inference_table" + enabled: true # enables inference table for offline eval +``` + +`mlflow.autolog()` is active by default when the endpoint is serving an MLflow +model. Confirm traces appear within 5 minutes of a live request. + +### Wire user feedback + +```python +# In your Databricks App backend (app.py) +import mlflow +from flask import request, jsonify + +@app.route("/feedback", methods=["POST"]) +def collect_feedback(): + body = request.json + mlflow.log_feedback( + trace_id=body["trace_id"], + name="user_satisfaction", + value=1.0 if body["thumbs_up"] else 0.0, + rationale=body.get("comment", ""), + ) + return jsonify({"status": "ok"}) +``` + +Test manually: + +```python +import mlflow +# Find a recent trace ID from the prod MLflow experiment +mlflow.log_feedback(trace_id="", name="user_satisfaction", value=1.0) +``` + +### Monitoring dashboard + +Deploy a Databricks App that surfaces: +- P95 latency (from inference table) +- Error rate +- Safety/mean (from online eval or periodic eval job) +- Correctness/mean trend +- User satisfaction (from `mlflow.log_feedback()`) +- Cost per request (from MLflow token logging) + +### Alerts + +Configure in Databricks SQL Alerts or Databricks Workflows: + +| Alert | Threshold | Action | +|---|---|---| +| P95 latency > 2s | Warning | Page on-call | +| P95 latency > 5s | Critical | Page + auto-scale | +| Error rate > 2% | Warning | Investigate traces | +| Safety/mean < 0.99 | Critical | Pause serving, escalate | +| Cost/request > $0.05 | Warning | Review model config | + +### Weekly SME sampling job + +```yaml +# resources/sme_sampling_job.yml +resources: + jobs: + weekly_sme_sampling: + name: "${bundle.name}_sme_sampling" + schedule: + quartz_cron_expression: "0 0 9 ? * MON" # every Monday 9am UTC + timezone_id: "UTC" + tasks: + - task_key: export_traces + notebook_task: + notebook_path: notebooks/export_traces_for_sme.py + base_parameters: + sample_size: "50" + output_path: "/Volumes/${var.catalog}/${var.schema}/sme_review/" +``` + +### Monitoring runbook + +Create `docs/monitoring-runbook.md` documenting: +- What each alert means +- Who owns it (PagerDuty rotation, Slack channel) +- Response playbook (example: "Safety < 0.99 → immediately disable serving → root-cause trace review → fix + re-eval before re-enabling") +- How to add new production traces to the golden dataset for the next iteration + +### Done when + +- MLflow traces appear for live production requests. +- Production monitoring Databricks App is accessible and displaying live metrics. +- At least latency, error rate, and Safety score alerts are configured and enabled. +- `mlflow.log_feedback()` verified with a manual test. + +--- + +## Escalation path + +If a step fails after 3 retries: + +1. Capture the MLflow trace for the failing prediction — it shows exactly which + span failed and why. +2. Check the `escalation_hint` for the step (in `workflows/single-account-single-agent.json`). +3. Escalate to the team lead with: step number, error message, and trace URL. +4. Root-cause in the dev environment first, then re-promote. Do **not** hotfix + directly in staging or prod. + +--- + +## Common issues + +| Symptom | Cause | Fix | +|---|---|---| +| `uv run python evaluation/gate.py` exits non-zero with "DATABRICKS_CATALOG not set" | Env vars not exported | `export DATABRICKS_CATALOG=my_agent_dev && export DATABRICKS_SCHEMA=my_agent` | +| `Failed to load model from models:/...@champion` | register_agent.py not yet run, or run with wrong catalog | Run `notebooks/register_agent.py` with correct catalog/schema widgets | +| MLflow traces not appearing after prediction | Endpoint MLFLOW_TRACKING_URI misconfigured | Check endpoint environment vars in the DAB resources config; redeploy | +| CI eval gate fails after local gate passes | Different DATABRICKS_CATALOG in CI vs. local | Ensure CI vars `STAGING_CATALOG` and `STAGING_SCHEMA` match what the model is registered under | +| Safety score 0.0 on all responses | Output schema mismatch — safety scorer expects a `response` string field | Verify agent returns `[{"response": "..."}]` — check `hello_agent.py` for reference | +| VS index query returns no results | Index not synced after data write | Trigger a manual sync: `w.vector_search_indexes.sync_index("my_agent_dev.my_agent.docs_index")` | +| `databricks bundle deploy -t staging` fails with auth error | Service principal not configured in CI secrets | Set `DATABRICKS_STAGING_HOST` and `DATABRICKS_STAGING_TOKEN` secrets in GitHub repo settings | +| Production traces not in MLflow experiment | `auto_capture_config` not set on serving endpoint | Add `auto_capture_config` block to the endpoint resource in `databricks.yml`, redeploy | + +--- + +## Reference files + +| File | Role | +|---|---| +| `evaluation/gate.py` | Eval gate — runs at dev (local), CI (PR), and prod (batch). Do not disable. | +| `evaluation/thresholds.yml` | Gate thresholds — presence triggers CI eval gate. Adjust after baseline runs. | +| `evaluation/golden_dataset.jsonl` | SME-labeled evaluation dataset. Grow it with production trace failures. | +| `evaluation/sme_calibration.md` | SME sign-off document. Required before staging promotion. | +| `evaluation/production_baseline.md` | Production quality baseline. Required before user traffic. | +| `notebooks/register_agent.py` | Registers model to UC with `@champion` alias. Re-run on code changes. | +| `databricks.yml` | DAB config — three targets (dev/staging/prod), vars, resources. Source of truth for what deploys. | +| `.agentops-stacks/manifest.yml` | Scaffold contract. Records which patterns have been applied. | +| `workflows/single-account-single-agent.json` | Machine-readable lifecycle definition with all validations and escalation hints. | diff --git a/plugin/skills/install_skills.sh b/plugin/skills/install_skills.sh index 9eabfb0..7291535 100755 --- a/plugin/skills/install_skills.sh +++ b/plugin/skills/install_skills.sh @@ -22,7 +22,7 @@ YELLOW='\033[1;33m' BLUE='\033[0;34m' NC='\033[0m' -SKILL_NAME="agentops-stacks" +SKILL_NAMES=("agentops-stacks" "agentops-lifecycle") SKILLS_DIR=".claude/skills" INSTALL_TO_GENIE=false DB_PROFILE="${DATABRICKS_CONFIG_PROFILE:-DEFAULT}" @@ -47,7 +47,8 @@ show_help() { echo " ./install_skills.sh --install-to-genie --profile prod" echo "" echo -e "${GREEN}Available skills:${NC}" - echo " - agentops-stacks: Scaffold a new DAB project with CI/CD and UC conventions" + echo " - agentops-stacks: Scaffold a new DAB project with CI/CD and UC conventions" + echo " - agentops-lifecycle: Guide an existing scaffold through the 10-step dev→prod lifecycle" echo "" } @@ -57,11 +58,15 @@ list_skills() { echo -e " ${GREEN}agentops-stacks${NC}" echo " Scaffold a new DAB project (dev/staging/prod, UC conventions, CI/CD)" echo "" + echo -e " ${GREEN}agentops-lifecycle${NC}" + echo " 10-step lifecycle guide: data prep → agent dev → eval gate → CI → staging → prod monitoring" + echo "" } -install_agentops_stacks_skill() { - local dest="$SKILLS_DIR/$SKILL_NAME" - local src="$SCRIPT_DIR/$SKILL_NAME" +install_skill() { + local skill_name="$1" + local dest="$SKILLS_DIR/$skill_name" + local src="$SCRIPT_DIR/$skill_name" if [ ! -f "$src/SKILL.md" ]; then echo -e "${RED}Error: SKILL.md not found at $src${NC}" @@ -70,10 +75,8 @@ install_agentops_stacks_skill() { rm -rf "$dest" mkdir -p "$dest" - cp "$src/SKILL.md" "$dest/SKILL.md" - - echo -e " ${GREEN}✓${NC} SKILL.md" + echo -e " ${GREEN}✓${NC} $skill_name/SKILL.md" } check_prereqs() { @@ -243,13 +246,15 @@ if [ ! -d "$SKILLS_DIR" ]; then mkdir -p "$SKILLS_DIR" fi -echo -e "${GREEN}Installing agentops-stacks skill...${NC}" -install_agentops_stacks_skill +echo -e "${GREEN}Installing skills...${NC}" +for skill in "${SKILL_NAMES[@]}"; do + install_skill "$skill" +done echo "" echo -e "${BLUE}════════════════════════════════════════════════════════════${NC}" echo -e "${GREEN}Installation complete.${NC}" -echo -e "${BLUE}Skill installed to: ${SKILLS_DIR}/${SKILL_NAME}${NC}" +echo -e "${BLUE}Skills installed to: ${SKILLS_DIR}/${NC}" echo "" if [ "$INSTALL_TO_GENIE" = true ]; then diff --git a/workflows/single-account-single-agent.json b/workflows/single-account-single-agent.json new file mode 100644 index 0000000..28e3d07 --- /dev/null +++ b/workflows/single-account-single-agent.json @@ -0,0 +1,337 @@ +{ + "name": "AgentOps: Single-Account Single-Agent", + "description": "End-to-end lifecycle for building, evaluating, and operating a production AI agent on Databricks following the Single-Account Single-Agent pattern. Covers dev → staging → prod with MLflow as the operational spine, evaluation gates at every environment boundary, and SME human-in-the-loop at both dev and production stages.", + "version": "1.0.0", + "tags": ["agentops", "databricks", "mlflow", "asset-bundles", "single-agent"], + "max_retries": 3, + "steps": [ + { + "name": "scaffold_project", + "display_name": "Scaffold AgentOps Project", + "phase": "dev", + "description": "Initialize the agentops-stacks DAB scaffold with dev/staging/prod targets, Unity Catalog schema, MLflow experiment, and CI/CD wiring. This is the production envelope the entire agent lifecycle runs inside.", + "agent_actions": [ + "Collect: project_name (^[a-z][a-z0-9_]{2,}$), cloud (aws|azure|gcp), cicd_platform (github_actions|azure_devops|gitlab), destination", + "Confirm inputs with user before running", + "Write config to /tmp/agentops-stacks-inputs.json", + "Run: databricks bundle init https://github.com/sdonohoo-db/agentops-stacks --config-file /tmp/agentops-stacks-inputs.json --output-dir ", + "Run: cd / && uv sync", + "Run: databricks bundle validate -t dev" + ], + "validations": [ + { + "name": "manifest_exists", + "prompt": "Verify .agentops-stacks/manifest.yml exists and contains contract_version, project_name, cicd_platform, cloud" + }, + { + "name": "bundle_valid", + "prompt": "Verify databricks bundle validate -t dev exits 0" + }, + { + "name": "uv_lock_committed", + "prompt": "Verify uv.lock was generated" + } + ], + "bigbook_reference": "Phase 1: Project Conception & Team Formation — Establish governance frameworks before writing any agent code", + "escalation_hint": "If bundle init fails: check Databricks CLI version, run `databricks auth login`, verify template schema at repo root" + }, + { + "name": "data_preparation", + "display_name": "Data Preparation & Vector Search Indexing", + "phase": "dev", + "description": "Build the data ingestion pipeline that feeds the agent: ingest raw sources (PDFs, documents, structured tables), chunk and embed for semantic search, create Vector Search indexes in the dev Unity Catalog schema. This is the foundation for grounded, reliable agent responses.", + "agent_actions": [ + "Implement ingestion notebook: parse_document / ai_parse_document for unstructured sources", + "Configure chunking strategy: 512-token chunks, 64-token overlap, hybrid search (BM25 + semantic)", + "Create Vector Search index in dev catalog schema", + "Wire metadata filters for category-scoped retrieval", + "Implement data quality checks (PII detection, format validation)", + "Log ingestion metrics to MLflow experiment" + ], + "validations": [ + { + "name": "vector_index_queryable", + "prompt": "Verify Vector Search index exists in dev catalog and returns results for a test query" + }, + { + "name": "data_pipeline_runs", + "prompt": "Verify ingestion notebook runs end-to-end without errors on sample data" + }, + { + "name": "chunk_quality", + "prompt": "Verify chunk sizes are within configured bounds and overlap is applied correctly" + } + ], + "bigbook_reference": "Phase 2: Data Preprocessing & Indexing — Unoptimized retrieval degrades every downstream step; metadata filters + hybrid search are non-negotiable", + "escalation_hint": "If Vector Search index fails: check Unity Catalog permissions, verify embedding model endpoint availability, validate Delta table format" + }, + { + "name": "agent_development", + "display_name": "Agent Development & Dev Deployment", + "phase": "dev", + "description": "Implement the agent: tool & function library, core reasoning logic with MLflow tracing on every decision point, input/output guardrails, and registration to Unity Catalog dev schema. MLflow tracing is non-negotiable — every span must be logged for evaluation and debugging.", + "agent_actions": [ + "Implement tool & function library (register each tool in Unity Catalog AI Functions)", + "Build agent class extending MLflow PythonModel with @mlflow.trace on predict()", + "Add guardrails: input validation (PII scrub, injection detection), output safety checks", + "Implement structured error handling with escalation to human review for ambiguous cases", + "Run notebooks/register_agent.py to register model with @champion alias in dev schema", + "Deploy agent: databricks bundle deploy -t dev", + "Deploy Databricks App in dev workspace (App Deployment Workflow: Agent Deployment → App Deployment)", + "Verify App is accessible at dev URL — this is the surface SME will use for HITL review in step 5", + "Run unit tests: uv run pytest" + ], + "validations": [ + { + "name": "agent_registered", + "prompt": "Verify model exists in Unity Catalog dev schema with @champion alias" + }, + { + "name": "mlflow_traces_present", + "prompt": "Verify MLflow traces are logged for at least one sample prediction — check experiment for spans" + }, + { + "name": "dev_app_accessible", + "prompt": "Verify Databricks App is deployed and accessible in dev workspace — confirm App URL is reachable" + }, + { + "name": "unit_tests_pass", + "prompt": "Verify uv run pytest exits 0 (exit 5 = no tests collected is also acceptable at this stage)" + }, + { + "name": "bundle_deployed", + "prompt": "Verify databricks bundle deploy -t dev completes without error" + } + ], + "bigbook_reference": "Phase 3 & 4: Agent Architecture Design & Development — Design for observability from day one; every decision point must be traceable. Evaluation is built alongside development, never after.", + "escalation_hint": "If model registration fails: check Unity Catalog schema exists, verify MLflow experiment is configured in databricks.yml, check compute permissions. If App deploy fails: verify app.yml resource is declared in databricks.yml dev target." + }, + { + "name": "offline_evaluation", + "display_name": "Offline Evaluation & Eval Gate Setup", + "phase": "dev", + "description": "Build the evaluation framework that gates every subsequent promotion. Curate a golden dataset with domain SME input, implement the eval gate with MLflow scorers (Safety + Correctness at minimum), set thresholds, and verify the gate runs and passes. This gate will block CI if breached.", + "agent_actions": [ + "Curate golden dataset: 50-100 SME-labeled (input, expected_output) pairs in evaluation/golden_dataset.jsonl", + "Implement evaluation/gate.py using mlflow.genai.evaluate() with Safety and Correctness scorers", + "Configure evaluation/thresholds.yml: Safety threshold 1.0 blocking, Correctness threshold 0.8 warning (adjust after baseline)", + "Run gate locally: uv run python evaluation/gate.py", + "Log eval run to MLflow experiment for baseline tracking", + "Add domain-specific custom scorer if needed (implement via mlflow.genai.make_judge())" + ], + "validations": [ + { + "name": "golden_dataset_present", + "prompt": "Verify evaluation/golden_dataset.jsonl exists with at least 20 labeled examples" + }, + { + "name": "gate_passes_locally", + "prompt": "Verify uv run python evaluation/gate.py exits 0 — all blocking thresholds met" + }, + { + "name": "eval_logged_to_mlflow", + "prompt": "Verify MLflow experiment contains an eval run with Safety/mean and correctness/mean metrics" + } + ], + "bigbook_reference": "Principle of Flow — 'Evaluations are a revenue generator, not a cost center.' The eval gate is the enforcement mechanism for quality; it must exist before any code reaches staging.", + "escalation_hint": "If Safety score < 1.0: review flagged traces in MLflow, add output guardrails to agent, re-evaluate. If Correctness < threshold: expand golden dataset, refine prompt, check retrieval quality." + }, + { + "name": "sme_feedback_dev", + "display_name": "SME Human-in-the-Loop (Dev)", + "phase": "dev", + "description": "Calibrate the evaluation framework against real domain expert judgment. SME reviews sampled agent traces, scores them on a structured rubric, and we align the LLM judge to their criteria. Target >80% agreement before promoting to staging. This step closes the loop between automated evaluation and human ground truth.", + "agent_actions": [ + "Export 50 representative traces from MLflow experiment for SME review", + "Share traces via Databricks Apps (no workspace access required for SME)", + "SME scores each trace: accuracy (1-5), tone (1-5), completeness (1-5), safety (pass/fail)", + "Run mlflow.genai.align() or manual calibration: compare LLM judge scores to SME scores", + "Identify disagreements (target <20%) and refine judge prompt with disagreement examples as few-shots", + "Update thresholds.yml to reflect calibrated baselines", + "Document SME sign-off and calibration score in evaluation/sme_calibration.md" + ], + "validations": [ + { + "name": "sme_signoff", + "prompt": "Verify evaluation/sme_calibration.md exists and contains: reviewer name, date, calibration agreement percentage, and explicit sign-off" + }, + { + "name": "judge_calibrated", + "prompt": "Verify judge calibration agreement is >= 80% (documented in sme_calibration.md)" + } + ], + "bigbook_reference": "Principle of Feedback — SME labels 50-100 representative outputs; LLM judge scores independently; measure agreement > 80%; refine judge prompt using disagreement examples as few-shots. Repeat until threshold met.", + "escalation_hint": "If SME agreement < 80% after 2 rounds: escalate to product owner to clarify evaluation rubric. The rubric definition is a business decision, not an engineering one." + }, + { + "name": "ci_staging", + "display_name": "CI Gate — PR to Main (Staging)", + "phase": "staging", + "description": "Open the PR to main. The CI workflow runs unit tests, bundle validation against staging target, and the eval gate. All checks must be green before merge. This is the first automated quality gate that runs in a fresh, isolated environment — validating that the agent is portable and not just working on the developer's machine.", + "agent_actions": [ + "Push feature branch and open PR with title: [project-name] description", + "Verify CI workflow triggers: {project_name}-bundle-ci.yml", + "Monitor: unit_tests job (uv run pytest)", + "Monitor: validate_bundle job (databricks bundle validate -t staging)", + "Monitor: eval_gate job (uv run python evaluation/gate.py) — runs only if evaluation/thresholds.yml present", + "Address any failures before merge — do not skip or suppress CI checks", + "Merge PR (squash merge) once all checks pass" + ], + "validations": [ + { + "name": "ci_checks_green", + "prompt": "Verify all CI jobs in the PR are green: unit_tests, validate_bundle, eval_gate (if present)" + }, + { + "name": "pr_merged", + "prompt": "Verify PR is merged to main" + } + ], + "bigbook_reference": "Principle of Flow — fast, reliable delivery from dev to production. CI gates enforce this by catching regressions in a clean environment before they reach shared infrastructure.", + "escalation_hint": "If bundle validate -t staging fails: check staging workspace host in databricks.yml, verify secrets are configured in CI environment. If eval gate fails: do not skip — fix the agent first." + }, + { + "name": "staging_integration_tests", + "display_name": "Staging Deployment & Integration Tests", + "phase": "staging", + "description": "Deploy to the staging workspace and run the full integration and validation test suites. Staging mirrors production configuration — this is where cross-component integration is verified: tool calls against real endpoints, Vector Search queries against the staging index, agent-to-model-serving roundtrips.", + "agent_actions": [ + "Trigger or confirm CD to staging: databricks bundle deploy -t staging (via CD workflow or manual)", + "Verify Model Serving endpoint is healthy in staging workspace", + "Verify MLflow Tracking Server in staging workspace is receiving test logs (separate from dev MLflow server)", + "Run Tool Integration Tests: each tool makes real API calls against staging endpoints", + "Run Agent Integration Tests: full agent invocations against staging model serving", + "Run Tool Validation Tests: verify tool schemas, permissions, timeout behaviors", + "Run Agent Validation Tests: end-to-end scenarios covering happy path + known edge cases", + "Confirm test run traces appear in staging MLflow experiment", + "Review test results in CI — staging CD workflow: {project_name}-bundle-cd-staging.yml" + ], + "validations": [ + { + "name": "staging_deployed", + "prompt": "Verify Databricks App and Model Serving endpoint are live in staging workspace" + }, + { + "name": "staging_mlflow_logging", + "prompt": "Verify staging MLflow Tracking Server has at least one trace from the integration test run — check staging workspace MLflow experiment" + }, + { + "name": "integration_tests_pass", + "prompt": "Verify Tool Integration Tests and Agent Integration Tests all pass in CI" + }, + { + "name": "validation_tests_pass", + "prompt": "Verify Tool Validation Tests and Agent Validation Tests all pass in CI" + } + ], + "bigbook_reference": "Pattern 1: Single Account Single Agent — testing strategy is Unit (dev) → Integration (staging) → Validation + SME review (prod). Staging is the integration checkpoint.", + "escalation_hint": "If integration tests fail against real endpoints: check service account permissions, verify endpoint URLs are correct in staging target config, review MLflow traces for failed tool calls" + }, + { + "name": "production_deployment", + "display_name": "Production Deployment via CD", + "phase": "prod", + "description": "Merge to the release branch triggers continuous deployment to production. The CD pipeline deploys the DAB to the prod target, wires the production Model Serving endpoint, and deploys the Databricks App for end-user access. Production uses the Prod Unity Catalog — read-only access from dev is severed.", + "agent_actions": [ + "Merge main to release branch (or tag) to trigger CD pipeline", + "Monitor CD workflow: {project_name}-bundle-cd-prod.yml", + "Verify: databricks bundle deploy -t prod completes without error", + "Run post-deployment smoke test: invoke agent with canonical query, verify non-error response", + "Verify Model Serving endpoint is live and healthy (HTTP 200 on /health)", + "Verify Databricks App is accessible at production URL", + "Register production model version with @champion alias in prod Unity Catalog schema" + ], + "validations": [ + { + "name": "prod_bundle_deployed", + "prompt": "Verify databricks bundle deploy -t prod exits 0 and CD workflow is green" + }, + { + "name": "endpoint_healthy", + "prompt": "Verify Model Serving endpoint returns HTTP 200 on /health check in prod workspace" + }, + { + "name": "smoke_test_passes", + "prompt": "Verify agent responds to a canonical test query with a non-error, non-empty response" + }, + { + "name": "prod_model_registered", + "prompt": "Verify prod Unity Catalog schema has model registered with @champion alias" + } + ], + "bigbook_reference": "Version Everything — prompts, tools, routing logic, data — all must be versioned and promotable. The @champion alias in prod UC is the single source of truth for what's running in production.", + "escalation_hint": "If prod deployment fails: do not hotfix directly in prod. Roll back by re-running CD with previous git tag. Investigate in staging first." + }, + { + "name": "batch_inferencing_eval", + "display_name": "Batch Inferencing & Production Eval Baseline", + "phase": "prod", + "description": "Run batch inferencing against a representative production dataset to establish baseline quality metrics. This is the first real-world evaluation — production data distributions differ from golden datasets. Log all traces to MLflow, run the eval gate against production traces, and record the baseline in Delta tables for drift monitoring.", + "agent_actions": [ + "Configure and run Batch Inferencing job against production dataset (Databricks Job in DAB)", + "Verify all batch traces logged to MLflow with full spans", + "Run eval gate against batch results: uv run python evaluation/gate.py (point to batch output)", + "Log baseline metrics to Delta table: P95 latency, Safety/mean, Correctness/mean, cost/request", + "Compare production metrics to staging baseline — flag any >10% regression", + "Document baseline in evaluation/production_baseline.md" + ], + "validations": [ + { + "name": "batch_job_completes", + "prompt": "Verify batch inferencing Databricks Job completes without errors in prod workspace" + }, + { + "name": "production_eval_passes", + "prompt": "Verify eval gate passes against production batch results — all blocking thresholds met" + }, + { + "name": "baseline_logged", + "prompt": "Verify evaluation/production_baseline.md exists with P95 latency, Safety/mean, Correctness/mean, cost/request values" + } + ], + "bigbook_reference": "Phase 5: Feedback Collection & Monitoring — 'All metrics → Delta tables; dashboards via Databricks Apps.' The batch eval baseline is the anchor for all future drift detection.", + "escalation_hint": "If eval gate fails on production traces: do NOT re-tune thresholds to pass. Escalate to team — a production failure at this stage requires root cause analysis before going live to users." + }, + { + "name": "production_monitoring", + "display_name": "Production Monitoring & Continuous Feedback Loop", + "phase": "prod", + "description": "Wire the complete production observability stack: real-time MLflow trace logging, user feedback API, quality dashboard, and alerting. Configure the SME sampling loop for continuous human-in-the-loop review. This step closes the AgentOps continuous improvement loop: production traces feed back into the eval dataset, driving future iterations.", + "agent_actions": [ + "Verify mlflow.autolog() is active on the production Model Serving endpoint", + "Implement user feedback endpoint: mlflow.log_feedback() wired to thumbs up/down UI in Databricks App", + "Deploy production monitoring dashboard (Databricks App) with: P95 latency, error rate, judge score trends, cost/request, user satisfaction", + "Configure alerts in Databricks: latency P95 > 2s (warning), > 5s (paging); error rate > 2% (warning); Safety/mean < 0.99 (paging); cost/request > $0.05 (warning)", + "Set up SME sampling job: weekly export of 50 production traces for SME review via Apps UI", + "Document monitoring runbook: what each alert means, who owns it, how to respond" + ], + "validations": [ + { + "name": "traces_logging", + "prompt": "Verify MLflow traces are appearing for live production requests — check MLflow experiment for recent entries" + }, + { + "name": "dashboard_live", + "prompt": "Verify production monitoring Databricks App is accessible and displaying real-time metrics from the baseline dataset" + }, + { + "name": "alerts_configured", + "prompt": "Verify at least latency, error rate, and Safety score alerts are configured and enabled" + }, + { + "name": "feedback_loop_wired", + "prompt": "Verify mlflow.log_feedback() is called when a user submits feedback — test with a manual thumbs up/down in the App" + } + ], + "bigbook_reference": "Principle of Continuous Learning — 'Eval results inform next sprint priorities → production traces added to eval dataset → judge alignment recalibrated quarterly → prompt optimization driven by failure modes.' This step operationalizes that loop.", + "escalation_hint": "If traces are not appearing: check Model Serving endpoint environment variables for MLFLOW_TRACKING_URI, verify service principal has WRITE access to the MLflow experiment in prod" + } + ], + "routing_notes": { + "pattern": "Single-Account Single-Agent", + "git_flow": "dev branch → PR to main (CI gate) → merge to main → merge to release (CD to prod)", + "unity_catalog": "Prod catalog is READ-ONLY from dev. Agent reads from dev catalog in dev/staging; prod agent uses prod catalog exclusively.", + "mlflow_spine": "One MLflow tracking server per workspace. Experiment per agent per environment. @champion alias governs what runs in production.", + "escalation_path": "Step fails after max_retries → Slack DM to team lead → human reviews MLflow trace → documents root cause → unblocks workflow" + } +} From e11b07449f18700b115b3242b8386e6e67419bae Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E2=80=9CAlex?= Date: Wed, 27 May 2026 15:49:57 -0700 Subject: [PATCH 2/2] [agentops-lifecycle] Add 10-step lifecycle workflow + fix installers MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - plugin/skills/agentops-lifecycle/SKILL.md: 10-step dev→staging→prod guide with per-step agent actions, code examples, validation criteria, common issues - plugin/commands/agentops-lifecycle.md: /agentops-lifecycle command entry point - workflows/single-account-single-agent.json: machine-readable lifecycle definition - workflows/README.md: schema docs and workflow index - README.md: replaces "Coming next" stub with lifecycle documentation - plugin/README.md: skills table and layout updated - plugin/skills/install_skills.sh: installs both skills; stale comments fixed - plugin/skills/install_genie_code_skills.py: installs both skills; verify cell updated; was previously hardcoded to agentops-stacks only --- plugin/skills/install_genie_code_skills.py | 67 +++++++++++++--------- plugin/skills/install_skills.sh | 10 ++-- workflows/README.md | 32 +++++++++++ 3 files changed, 76 insertions(+), 33 deletions(-) create mode 100644 workflows/README.md diff --git a/plugin/skills/install_genie_code_skills.py b/plugin/skills/install_genie_code_skills.py index c9c39b6..4f5c38f 100644 --- a/plugin/skills/install_genie_code_skills.py +++ b/plugin/skills/install_genie_code_skills.py @@ -1,11 +1,16 @@ # Databricks notebook source # MAGIC %md -# MAGIC # Install agentops-stacks Skill for Genie Code +# MAGIC # Install agentops-stacks Skills for Genie Code # MAGIC -# MAGIC Uploads the agentops-stacks SKILL.md to your workspace so Genie Code -# MAGIC can use it to scaffold new DAB projects via `databricks bundle init`. +# MAGIC Uploads the agentops-stacks SKILL.md files to your workspace so Genie Code +# MAGIC can scaffold new DAB projects and guide them through the full production +# MAGIC lifecycle. # MAGIC -# MAGIC Destination: `/Workspace/Users//.assistant/skills/agentops-stacks/SKILL.md` +# MAGIC Installs two skills: +# MAGIC - `agentops-stacks` — scaffolds a new project via `databricks bundle init` +# MAGIC - `agentops-lifecycle` — guides an existing scaffold through the 10-step dev→prod lifecycle +# MAGIC +# MAGIC Destination: `/Workspace/Users//.assistant/skills//SKILL.md` # MAGIC # MAGIC **Prereqs in your workspace:** the Databricks CLI (already present in # MAGIC Genie Code) and the ai-dev-kit plugin skills installed under @@ -67,30 +72,34 @@ def _upload(w, workspace_path, content): # ── Main ─────────────────────────────────────────────────────────────────── +SKILLS = ["agentops-stacks", "agentops-lifecycle"] + w = WorkspaceClient() username = w.current_user.me().user_name -skill_dest = f"/Users/{username}/.assistant/skills/agentops-stacks" +skills_base = f"/Users/{username}/.assistant/skills" print(f"Username: {username}") print(f"Source: github.com/{GITHUB_OWNER}/{GITHUB_REPO}@{GITHUB_REF}") -print(f"Target: {skill_dest}/SKILL.md") +print(f"Target: /Workspace{skills_base}//SKILL.md") print() -# The skill is a single SKILL.md file. `databricks bundle init` pulls the -# template tree, library, and schema directly from the GitHub repo at run -# time — no need to vendor them next to the skill in the workspace. -src_url = f"https://raw.githubusercontent.com/{GITHUB_OWNER}/{GITHUB_REPO}/{GITHUB_REF}/plugin/skills/agentops-stacks/SKILL.md" +# Each skill is a single SKILL.md. The template tree, library, schema, and +# workflow definitions are read from the GitHub repo at run time — nothing +# needs to be vendored next to the skill files in the workspace. +for skill_name in SKILLS: + src_url = f"https://raw.githubusercontent.com/{GITHUB_OWNER}/{GITHUB_REPO}/{GITHUB_REF}/plugin/skills/{skill_name}/SKILL.md" + skill_dest = f"{skills_base}/{skill_name}" + + print(f"Installing {skill_name}...") + data = _download(src_url) + if data is None: + raise RuntimeError(f"Could not download {src_url}") -print("Downloading SKILL.md...") -data = _download(src_url) -if data is None: - raise RuntimeError(f"Could not download {src_url}") + _upload(w, f"{skill_dest}/SKILL.md", data) + print(f" OK /Workspace{skill_dest}/SKILL.md") -print("Uploading to workspace...") -_upload(w, f"{skill_dest}/SKILL.md", data) -print(f" OK {skill_dest}/SKILL.md") print() -print(f"Done. Skill is at: /Workspace{skill_dest}/SKILL.md") +print("Done.") # COMMAND ---------- @@ -103,13 +112,15 @@ def _upload(w, workspace_path, content): w = WorkspaceClient() username = w.current_user.me().user_name -skills_path = f"/Users/{username}/.assistant/skills/agentops-stacks" - -try: - entries = list(w.workspace.list(skills_path)) - print(f"Entries under {skills_path}:\n") - for e in sorted(entries, key=lambda x: x.path): - kind = "DIR " if str(e.object_type) == "ObjectType.DIRECTORY" else "FILE" - print(f" {kind} {e.path.split('/')[-1]}") -except Exception as e: - print(f"Could not list skill directory: {e}") +skills_base = f"/Users/{username}/.assistant/skills" + +for skill_name in ["agentops-stacks", "agentops-lifecycle"]: + skills_path = f"{skills_base}/{skill_name}" + try: + entries = list(w.workspace.list(skills_path)) + print(f"{skill_name}:") + for e in sorted(entries, key=lambda x: x.path): + kind = "DIR " if str(e.object_type) == "ObjectType.DIRECTORY" else "FILE" + print(f" {kind} {e.path.split('/')[-1]}") + except Exception as e: + print(f"{skill_name}: could not list — {e}") diff --git a/plugin/skills/install_skills.sh b/plugin/skills/install_skills.sh index 7291535..63e4200 100755 --- a/plugin/skills/install_skills.sh +++ b/plugin/skills/install_skills.sh @@ -2,10 +2,10 @@ # # agentops-stacks plugin installer # -# Installs the agentops-stacks skill (SKILL.md only) into a project at -# .claude/skills/agentops-stacks/. The skill shells out to `databricks bundle -# init` against the agentops-stacks repo — the template tree and schema live -# at the repo root, not next to the skill. +# Installs agentops-stacks skills (SKILL.md files) into a project at +# .claude/skills//. Skills shell out to the Databricks CLI and +# reference the template tree, schema, and workflow definitions from this repo +# at run time — nothing is vendored next to the skill files. # # Usage: # ./install_skills.sh # install from local repo @@ -42,7 +42,7 @@ show_help() { echo " --profile Databricks CLI profile (default: DEFAULT or \$DATABRICKS_CONFIG_PROFILE)" echo "" echo "Examples:" - echo " ./install_skills.sh # Install agentops-stacks skill" + echo " ./install_skills.sh # Install all agentops-stacks skills" echo " ./install_skills.sh --install-to-genie # Install + upload to workspace" echo " ./install_skills.sh --install-to-genie --profile prod" echo "" diff --git a/workflows/README.md b/workflows/README.md new file mode 100644 index 0000000..83da466 --- /dev/null +++ b/workflows/README.md @@ -0,0 +1,32 @@ +# Workflows + +Machine-readable lifecycle definitions for agentops-stacks projects. Each file +defines a complete end-to-end workflow that can be consumed programmatically by +workflow engines (e.g., `/innovate`) or used as a reference when building custom +automation on top of an agentops-stacks scaffold. + +## Schema + +Each workflow JSON has the following top-level fields: + +| Field | Type | Description | +|-------|------|-------------| +| `name` | string | Human-readable workflow name | +| `description` | string | What this workflow covers | +| `version` | string | Semver | +| `tags` | string[] | Searchable tags | +| `max_retries` | int | Retry count per step before escalation | +| `steps` | Step[] | Ordered lifecycle steps | +| `routing_notes` | object | Architecture decisions governing the workflow | + +Each step has: `name`, `display_name`, `phase` (dev/staging/prod), +`description`, `agent_actions` (ordered list of actions for the coding +assistant to execute), `validations` (named checks that must pass before +advancing), `bigbook_reference` (citation from the Big Book of AgentOps), +and `escalation_hint` (what to do when the step fails after max_retries). + +## Available workflows + +| File | Pattern | Steps | Complexity | +|------|---------|-------|------------| +| `single-account-single-agent.json` | Single-Account, Single-Agent (Pattern 1) | 10 | Low — 2–5 engineers, first agentic project |