DAIS 2026: operational demo consolidation and platform hardening

.gitignore

@@ -1,5 +1,4 @@
 .state/
-data/generator/cache/
 stages/agent.py
 
 *conf*.json
@@ -9,6 +8,7 @@ __pycache__
 .env
 .databricks
 .claude
+.cursor
 .bundle
 /.vscode/
 node_modules/

AGENTS.md

@@ -51,8 +51,11 @@ Running `databricks bundle destroy` only removes the Job definition - **all runt
 
 **Cleanup workflow**:
 ```bash
-databricks bundle run cleanup    # Delete runtime resources via uc_state
-databricks bundle destroy        # Delete bundle resources via DABs
+# Cleanup is a bundle SCRIPT, so the catalog override uses --var (not --params).
+# --params is for job tasks only; passing it to cleanup is silently ignored.
+# `--var catalog=X` on `bundle run cleanup` does NOT reach the script — use BUNDLE_VAR_catalog=.
+BUNDLE_VAR_catalog=<name> databricks bundle run cleanup -t <target>    # Delete runtime resources via uc_state
+databricks bundle destroy      -t <target>                         # Delete bundle resources via DABs
 ```
 
 **Location**: `/utils/uc_state/` - See `README.md` for API usage
@@ -320,8 +323,10 @@ rm -rf .databricks .bundle
 
 1. **Clean existing deployment**
    ```bash
-   databricks bundle run cleanup --params "CATALOG=testcatalog"
-   databricks bundle destroy -t <target>
+   # Cleanup is a script — use --var (not --params).  See the Quick Reference
+   # at the bottom of this doc for full details.
+   BUNDLE_VAR_catalog=testcatalog databricks bundle run cleanup -t <target>
+   databricks bundle destroy     -t <target>
    ```
 
 2. **Clear local cache** (if experiencing cache issues)
@@ -629,6 +634,40 @@ Use full redeploy instead if:
   - Ensure it's plumbed through all layers
   - Check `databricks.yml` parameters, stage parameter parsing, and implementation usage
 
+#### 5a. `--var catalog` vs `--params CATALOG` drift
+- **Why fragile**: Two separate dials use a catalog name and they can disagree.
+  - `bundle deploy --var catalog=X` → resolves `${var.catalog}` at deploy
+    time.  Bakes `X` into every DABs-managed resource name (e.g. the `all`
+    target's `caspers_ops_warehouse` becomes `X-ops-warehouse`), every
+    AI/BI dashboard name, every dashboard `dataset_catalog`, and every job
+    parameter `default: ${var.catalog}...` (incl. `CATALOG`,
+    `REFUND_AGENT_ENDPOINT_NAME`, `COMPLAINT_AGENT_ENDPOINT_NAME`,
+    `OPS_WAREHOUSE_NAME`, `SUPERVISOR_ENDPOINT_NAME`).
+  - `bundle run caspers --params "CATALOG=Y"` → overrides ONLY the
+    run-time `CATALOG` widget value inside stage notebooks.  Cannot rename
+    anything DABs already created.
+- **Symptom when they disagree**: stages that reconstruct a DABs-managed
+  resource name from the run-time `CATALOG` widget fail to find it.
+  Example: deploying with the default and then running
+  `--params CATALOG=mycatalog` against the `all` target makes
+  `stages/operational_app.ipynb` fail with
+  `RuntimeError: Warehouse 'mycatalog-ops-warehouse' not found` because
+  DABs created `caspersdev-ops-warehouse`.
+- **Best practice**:
+  - When in doubt, pass the same catalog to both:
+    `bundle deploy -t <target> --var catalog=X` then
+    `bundle run caspers --params "CATALOG=X"`.
+  - When adding a stage that needs the name of a DABs-managed resource,
+    do NOT reconstruct it from the `CATALOG` widget.  Add a dedicated job
+    parameter with a `${var.catalog}-...` default in `databricks.yml`
+    (this is how `OPS_WAREHOUSE_NAME`, `REFUND_AGENT_ENDPOINT_NAME` and
+    `COMPLAINT_AGENT_ENDPOINT_NAME` are wired), then read that parameter
+    via `dbutils.widgets.get(...)` in the stage.  The deploy-time value
+    rides through the job parameter into the run-time widget, so the two
+    dials physically cannot disagree.
+  - The long comment in `stages/operational_app.ipynb` cell 8 (the
+    `_agent_ep_name` helper) is the canonical example of this pattern.
+
 ### 6. Resource Dependencies
 - **Why fragile**: Stages create resources that others depend on (endpoints, tables, etc.)
 - **When touching**: Creation/deletion order, stage dependencies
@@ -754,8 +793,13 @@ databricks bundle run caspers [--params "CATALOG=mycatalog"]
 
 ### Cleanup
 ```bash
-databricks bundle run cleanup [--params "CATALOG=mycatalog"]
-databricks bundle destroy -t <target>
+# IMPORTANT: cleanup is a bundle SCRIPT, not a job task.
+#   Catalog override:  --var catalog=<name>   (NOT --params "CATALOG=...")
+#   Target selection:  -t <target>            (passes through normally)
+# --params is silently ignored by scripts; passing it will clean the
+# default catalog (caspersdev) which is rarely what you want.
+BUNDLE_VAR_catalog=<name> databricks bundle run cleanup -t <target>
+databricks bundle destroy     -t <target>
 rm -rf .databricks .bundle  # If cache issues
 ```
 

README.md

@@ -21,34 +21,76 @@ See the [`skill` branch](../../tree/skill) for details.
 
 ## Deploy
 
+Replace `all` and `mycatalog` with your target and catalog:
+
 ```bash
-databricks bundle deploy -t <target>
-databricks bundle run caspers
+databricks bundle deploy -t all --var catalog=mycatalog
+databricks bundle run caspers -t all --params "CATALOG=mycatalog"
 ```
 
+Use the same target for deploy and run. Omit `-t` to use `default` (refund demo only).
+
 Available targets:
 
 | Target | What it deploys |
 |--------|----------------|
-| `default` | Data generation, Lakeflow pipeline, refund agent, Lakebase + app |
-| `support` | Data generation, Lakeflow pipeline, support triage agent, Lakebase + app |
-| `complaints` | Data generation, Lakeflow pipeline, complaint agent, Lakebase |
-| `free` | Data generation, Lakeflow pipeline (Free Edition compatible) |
-| `menus` | Document intelligence, DLT pipeline, Genie, Knowledge Assistants, Multi-Agent Supervisor |
+| `default` | Canonical data replay, Lakeflow pipeline, refund agent + stream, Lakebase Autoscale reverse-ETL, Refund Manager app |
+| `support` | Canonical data replay, Lakeflow pipeline, support triage agent + streams, Lakebase + app |
+| `complaints` | Canonical data replay, Lakeflow pipeline, complaint agent + streams, Lakebase |
+| `free` | Canonical data replay, Lakeflow pipeline (Free Edition compatible) |
+| `all` | Full platform demo: refund + complaints paths above, document intelligence pipeline, 3 Genies + 6 Knowledge Assistants + Multi-Agent Supervisor, Operational Dashboard app (Lakebase-backed), 5 AI/BI dashboards. See [`demos/dais2026-runbooks/SETUP.ipynb`](demos/dais2026-runbooks/SETUP.ipynb) for workspace prep. |
+
+Refund and complaint evaluation tasks (`Refund_Evaluation`, `Complaint_Evaluation`) are **skipped by default** (`SKIP_EVAL=true`). Pass `--params "SKIP_EVAL=false"` to opt in. The supervisor/KA `Evaluation` task on `all` (`stages/operational_evaluation`) always runs — it is not gated by `SKIP_EVAL`.
+
+Optionally specify a catalog (default: `caspersdev`).  There are **two** dials
+that take a catalog name and they must agree:
 
-Optionally specify a catalog (default: `caspersdev`):
+| Dial | When | What it controls |
+|---|---|---|
+| `bundle deploy --var catalog=<name>` | deploy time | the catalog baked into every DABs-managed resource — the `all` target's `caspers_ops_warehouse` SQL warehouse, AI/BI dashboard names, dashboard `dataset_catalog`, and the *default* value of every job parameter that uses `${var.catalog}` (including `CATALOG`, `REFUND_AGENT_ENDPOINT_NAME`, `OPS_WAREHOUSE_NAME`, etc.) |
+| `bundle run caspers --params "CATALOG=<name>"` | run time | only the value of the `CATALOG` widget inside stage notebooks.  Cannot rename anything DABs already created. |
+
+If they disagree (e.g. `bundle deploy -t all` with the default + `bundle run
+--params CATALOG=mycatalog`), the `all` target will fail at the
+`Operational_App` stage because the warehouse DABs created (`caspersdev-ops-warehouse`)
+is not what the stage looks up (`mycatalog-ops-warehouse`).  The fix is to
+pass the same catalog to both:
 
 ```bash
-databricks bundle run caspers --params "CATALOG=mycatalog"
+databricks bundle deploy -t all --var catalog=mycatalog
+databricks bundle run caspers -t all --params "CATALOG=mycatalog"
 ```
 
+For targets other than `all` (no DABs-owned warehouse/dashboards),
+`--params CATALOG=mycatalog` alone usually works, but passing both keeps the
+deploy-time and run-time catalogs in sync and is the safer habit.
+
 ## Clean Up
 
+Cleanup is **destructive**: it runs `destroy.ipynb`, which `DROP CATALOG … CASCADE` on the catalog you pass. It does **not** rebuild data or schemas afterward.
+
+**Catalog default:** if you omit `BUNDLE_VAR_catalog=...`, cleanup uses `caspersdev` (the bundle default) — or whatever catalog was last passed to `bundle deploy --var catalog=...` on this machine.
+
+Cleanup is a bundle **script** — use `BUNDLE_VAR_catalog=...` on the command line (not `--params CATALOG=...`, which is ignored for scripts). **`--var catalog=...` on `bundle run cleanup` does not work** — the CLI does not pass it into the script.
+
 ```bash
-databricks bundle run cleanup
-databricks bundle destroy
+# 1. Delete runtime UC resources (catalog + everything in it)
+BUNDLE_VAR_catalog=mycatalog databricks bundle run cleanup -t all
+# With a non-default CLI profile, append --profile <name> at the end.
+
+# 2. Delete bundle-managed resources (job, warehouses, dashboards)
+databricks bundle destroy -t all
 ```
 
+To **rebuild** after cleanup, deploy and run the job again (deploy alone does not recreate the catalog):
+
+```bash
+databricks bundle deploy -t all --var catalog=mycatalog
+databricks bundle run caspers -t all --params "CATALOG=mycatalog"
+```
+
+The first task (`Canonical_Data`) runs `CREATE CATALOG IF NOT EXISTS` and repopulates schemas/tables.
+
 ## Blog
 
 Check out the [Casper's Kitchens Blog](https://databricks-solutions.github.io/caspers-kitchens/).

apps/caspers-ops-dashboard/app.yaml

@@ -0,0 +1,56 @@
+command:
+  - uvicorn
+  - app.main:app
+env:
+  - name: LAKEBASE_ENDPOINT_PATH
+    value: ''
+  - name: LAKEBASE_DATABASE_NAME
+    value: 'caspers-ops'
+  - name: DATABRICKS_CATALOG
+    value: ''
+  - name: SUPERVISOR_ENDPOINT
+    value: ''
+  - name: SUPERVISOR_TILE_ID
+    value: ''
+  - name: SUPERVISOR_MLFLOW_EXPERIMENT_ID
+    value: ''
+  - name: GENIE_ID_REVENUE
+    value: ''
+  - name: GENIE_ID_OPS
+    value: ''
+  - name: GENIE_ID_MENU
+    value: ''
+  - name: KA_ID_INSPECTION
+    value: ''
+  - name: KA_ID_MENU
+    value: ''
+  - name: KA_ID_LEGAL
+    value: ''
+  - name: KA_ID_REGULATORY
+    value: ''
+  - name: KA_ID_AUDITS
+    value: ''
+  - name: KA_ID_CONSULTANCY
+    value: ''
+  - name: DATABRICKS_WAREHOUSE_ID
+    value: ''
+  # Operations dashboard is published in two variants — light + dark —
+  # because Databricks embedded dashboards always render in light mode
+  # regardless of the `?theme=` URL param.  The frontend swaps the iframe
+  # src between these two IDs when the user toggles the app theme.  The
+  # stage writes both runtime values; OPS_DASHBOARD_ID is the light variant.
+  - name: OPS_DASHBOARD_ID
+    value: ''
+  - name: OPS_DASHBOARD_ID_DARK
+    value: ''
+  # Multi-page dashboard — leave empty so the iframe URL doesn't append a
+  # #page= fragment and the user sees the dashboard's default landing page.
+  - name: OPS_DASHBOARD_PAGE
+    value: ''
+  # Custom agent endpoints — populated by operational_app stage from CATALOG
+  - name: REFUND_AGENT_ENDPOINT
+    value: ''
+  - name: COMPLAINT_AGENT_ENDPOINT
+    value: ''
+  - name: REFUND_MANAGER_APP_URL
+    value: ''