Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions .ddx/beads.jsonl

Large diffs are not rendered by default.

5 changes: 5 additions & 0 deletions apps/data-profiling/tests/test_config.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,11 @@
default that silently filled in is visible rather than assumed.
"""

# @covers US-047-AC1
# @covers US-047-AC3
# @covers US-047-AC4
# @covers US-047-AC5

from __future__ import annotations

import pytest
Expand Down
5 changes: 5 additions & 0 deletions apps/data-profiling/tests/test_diagnostics.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,11 @@
on a warehouse that is not running (the two-second startup budget).
"""

# @covers US-049-AC1
# @covers US-049-AC2
# @covers US-049-AC4
# @covers US-049-AC5

from __future__ import annotations

import pytest
Expand Down
6 changes: 6 additions & 0 deletions apps/data-profiling/tests/test_fr23_stack.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,12 @@
workspace. Live deploy-and-drive remains operational evidence.
"""

# @covers US-047-AC2
# @covers US-047-AC6
# @covers US-048-AC5
# @covers US-049-AC3
# @covers US-049-AC6

from __future__ import annotations

from pathlib import Path
Expand Down
6 changes: 6 additions & 0 deletions apps/data-profiling/tests/test_provision.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,12 @@
without a workspace.
"""

# @covers US-048-AC1
# @covers US-048-AC2
# @covers US-048-AC3
# @covers US-048-AC4
# @covers US-048-AC6

from __future__ import annotations

import pytest
Expand Down
2 changes: 1 addition & 1 deletion docs/helix/01-frame/concerns.md
Original file line number Diff line number Diff line change
Expand Up @@ -70,4 +70,4 @@ This project uses the following area labels for concern scoping:
| unity-catalog (three-part runtime naming) vs. testing (local engines without catalogs) | The relation seam (`core/relations.py`) renders engine-appropriate references; conformance fixtures pin the per-engine expected form — never hardcode either shape in emitters |
| sample-data (varied, generated data) vs. testing (byte-for-byte golden assertions) | Golden/parity tests use pinned deterministic generation configs; variability is exercised in generator unit tests, not in cross-engine golden comparisons |
| unity-catalog (no hardcoded catalogs) vs. databricks-app (must address one concrete location at runtime) | ADR-019: the app receives a **declared** `(catalog, schema, volume)` as a deployment input and resolves it through one precedence chain — the location is concrete at runtime but never literal in tracked source |
| databricks-app (a deployed operator UI) vs. verification (evidence must be observed, and e2e = compile→artifacts→backbone for this project) | The project-level e2e override covers the library runtime, not the app. The app currently has **no whole-stack exercise**; until one exists, app changes are evidenced by deploy-and-drive checks recorded per change, and the gap is tracked in the `e2e-framework` slot |
| databricks-app (a deployed operator UI) vs. verification (evidence must be observed, and e2e = compile→artifacts→backbone for this project) | The project-level e2e override covers the library runtime, not the app. The app has a **unit whole-stack path** (`test_fr23_stack.py`); live deploy-and-drive remains operational residual, not a missing composition gate |
8 changes: 4 additions & 4 deletions docs/helix/01-frame/feature-registry.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ ddx:
| FEAT-031 | [Multi-Source Ingestion](features/FEAT-031-multi-source-ingestion.md) | Discriminated `source:` contract (delimited/parquet/jdbc/json) with kind-dependent raw typing, ingestion reader seam, JDBC compiled read specs, and database discovery; Northwind end-to-end is the acceptance goal. DUMP/PARQ/JDBC/JSON cores + story floor US-040/042/043/050 shipped. | Specified | P1 | Platform / Data Engineering | PRD: Source Acquisition (FR-21.1–FR-21.7); ADR-015 | 2026-07-22 |
| FEAT-032 | [Embeddings & Document Corpus](features/FEAT-032-embeddings-and-document-corpus.md) | Dimensioned EMBEDDING type compiling to ARRAY<FLOAT>, GX dimensionality validation, and the governed document-corpus contract; SEC 10-K demo is the acceptance goal. Type core + CORP example + notebook contract shipped; workspace job not CI-gated. | Built | P1 | Platform / Data Engineering | PRD: UMF Model and I/O (FR-1.11); ADR-016 | 2026-07-22 |
| FEAT-033 | [Guidebook](features/FEAT-033-guidebook.md) | Render a directory of UMFs into a navigable, self-contained HTML guidebook — one page per table — with column metadata, FK + derivation lineage, group/flat indexes, and search; CLI + `generate_guidebook` API. | Built | P1 | Platform / Developer Experience | PRD: Guidebook (FR-22.1–FR-22.4); ADR-018 | 2026-07-22 |
| FEAT-034 | [App Deployment & Configuration](features/FEAT-034-app-deployment-configuration.md) | Deploy the guidebook + profiling app into any Databricks environment: metadata location as declared input, idempotent provisioning of schema/volume/governance tables, and fail-fast startup validation. Desired state; implementation gaps tracked in alignment beads. | Specified | P1 | Platform / Developer Experience | PRD: App Deployment & Configuration (FR-23.1–FR-23.6) | 2026-07-22 |
| FEAT-034 | [App Deployment & Configuration](features/FEAT-034-app-deployment-configuration.md) | Deploy the guidebook + profiling app into any Databricks environment: metadata location as declared input, idempotent provisioning of schema/volume/governance tables, and fail-fast startup validation. Unit path shipped; live workspace deploy residual. | Built | P1 | Platform / Developer Experience | PRD: App Deployment & Configuration (FR-23.1–FR-23.6) | 2026-07-22 |

## Status Definitions

Expand All @@ -65,9 +65,9 @@ enum). **Delivery stage** is tracked only in this registry
(Draft / Specified / Designed / In Test / In Build / Built / Deployed / …).
Spec-lifecycle status does not imply delivery **Built**: a feature's spec may be
final while delivery is still Specified or In Build when residual work remains
(FEAT-032 is Approved / In Build; FEAT-031 and FEAT-034 are Specified on both
axes). Legacy FEAT-001–029 are Built; FEAT-030 and
FEAT-033 are Built; FEAT-032 type core is shipped with CORP/DEMO residual.
(FEAT-032 Approved with demo residual; FEAT-031 Specified with US-044/045 demo
residual; FEAT-034 Built with live-deploy residual). Legacy FEAT-001–029,
FEAT-030, and FEAT-033 are Built; FEAT-032 type core + CORP example shipped.

## Dependencies

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ ddx:
# Feature Specification: FEAT-031 — Multi-Source Ingestion (Source-Shape Contract)

**Feature ID**: FEAT-031
**Status**: Specified (DUMP/PARQ/JDBC/JSON cores + story floor US-040/042/043/050 shipped; demo residual on US-044/045)
**Status**: Specified (DUMP/PARQ/JDBC/JSON cores + story floor US-040/042/043/050 shipped; US-044 Kaggle notebooks Built; US-045 workspace job residual)
**Priority**: P1
**Owner**: Platform / Data Engineering
**Covered PRD Subsystem(s)**: Source Acquisition
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -10,21 +10,32 @@ ddx:

# Feature Specification: FEAT-034 — App Deployment & Configuration

**Status**: Specified
**Status**: Built (unit whole-stack path green; live workspace deploy residual)
**Priority**: P1
**Feature ID**: FEAT-034
**Owner**: Platform / Developer Experience
**Covered PRD Subsystem(s)**: App Deployment & Configuration
**Covered PRD Requirements**: FR-23.1, FR-23.2, FR-23.3, FR-23.4, FR-23.5, FR-23.6
**Cross-Subsystem Rationale**: None — single subsystem.

> **Phase status (honest 2026-07-22).** Specs describe the desired end state.
>
> | Slice | Status | Evidence / residual |
> |-------|--------|---------------------|
> | CFG-01..04 config precedence | **Shipped** | `profiler/config.py` + `tests/test_config.py` |
> | PROV-01..04 idempotent provision | **Shipped** | `profiler/provision.py` + `tests/test_provision.py` |
> | PKG-01..03 declared app.yaml inputs | **Shipped** | `apps/data-profiling/app.yaml` env inputs |
> | DIAG-01..04 startup + optionals | **Shipped** | `profiler/diagnostics.py` + `tests/test_diagnostics.py` |
> | Unit whole-stack FR-23 path | **Shipped** | `tests/test_fr23_stack.py` |
> | Live deploy-and-drive on a workspace | **Residual** | Operational evidence; not CI-gated |

## Overview

Make the guidebook + profiling application deployable into any Databricks
environment as a configured step of the tablespec process. Today the
application is bound to one workspace by environment literals embedded in its
source; this feature makes the metadata location a declared input and adds an
explicit provisioning step, satisfying FR-23.1–FR-23.6.
environment as a configured step of the tablespec process. The metadata location
is a declared input; an explicit provisioning step creates schema/volume/
governance tables; startup validation fails fast with actionable messages —
satisfying FR-23.1–FR-23.6.

## Ideal Future State

Expand Down
4 changes: 2 additions & 2 deletions docs/helix/01-frame/prd.md
Original file line number Diff line number Diff line change
Expand Up @@ -209,7 +209,7 @@ edits; do not renumber on edit.

### Subsystem: Source Acquisition

**FR-21** requirement family. *Governed by FEAT-031; decision recorded in ADR-015. The raw landing contract generalizes from flat files to declared source shapes. Phase honesty (2026-07-22, alignment drain): delimited/parquet/jdbc/json cores + DUMP options + PARQ typed-raw cast + JDBC discovery (US-039) + story floor US-040/042/043/050 are shipped; remaining residual is primarily demo notebook evidence (US-044/045).*
**FR-21** requirement family. *Governed by FEAT-031; decision recorded in ADR-015. The raw landing contract generalizes from flat files to declared source shapes. Phase honesty (2026-07-22): delimited/parquet/jdbc/json cores + DUMP/PARQ + JDBC discovery (US-039) + story floor US-040/042/043/050 shipped; US-044 Kaggle notebook pair Built under `notebooks/kaggle-demo/`; US-045 workspace job residual remains operational.*

- **FR-21.1** — **Declared source shape.** UMF carries a discriminated `source:` block (`kind: delimited | parquet | jdbc | json`); today's `file_format` is the body of the `delimited` variant and remains a back-compat alias resolved via a non-persisting accessor. Raw landing typing follows the kind: all-STRING for text-landed sources, native-typed for typed sources.
- **FR-21.2** — **Dump-dialect text landing.** The `delimited` variant covers database dump files: multi-character line terminators, `\N`-style null escapes, footer handling, and `skip_rows` honored end-to-end — every declared option actually consumed by the compiled readers. *(Shipped — model + dump reader + unit tests.)*
Expand Down Expand Up @@ -324,7 +324,7 @@ edits; do not renumber on edit.

### Subsystem: App Deployment & Configuration

**FR-23** requirement family. *Governs deployment and configuration of the guidebook + profiling Databricks App (`apps/data-profiling/`) so one source tree deploys into any workspace. The metadata substrate is already location-parameterized (`delta_repo.ensure_tables(catalog, schema)`, `VolumeRef`); this family is the desired deployability contract. Feature decomposition is FEAT-034; configuration precedence is ADR-019. Implementation gaps (portable config, provisioning, startup diagnostics, app e2e) are tracked in the alignment bead epic, not by shrinking these requirements.*
**FR-23** requirement family. *Governs deployment and configuration of the guidebook + profiling Databricks App (`apps/data-profiling/`) so one source tree deploys into any workspace. Feature decomposition is FEAT-034; configuration precedence is ADR-019. Phase honesty (2026-07-22): config precedence, idempotent provision, declared app.yaml inputs, startup diagnostics, and the unit whole-stack path (`test_fr23_stack.py`) are shipped; live workspace deploy-and-drive remains operational residual.*

- **FR-23.1** — **Resolved runtime configuration.** The app resolves every environment-specific setting through one configuration object with a fixed precedence: deployment-supplied environment variables → `connections.yaml` → built-in defaults. No catalog, schema, volume, warehouse id, or workspace URL appears as a literal in application code.
- **FR-23.2** — **Declared governance location.** The metadata home is a declared `(catalog, schema)` pair plus an output volume. Every governance table and every volume path derives from that declaration, so relocating the metadata is a configuration change, not a code change.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ ddx:
**Feature Requirements**: SRC-01, SRC-04, SRC-05, DUMP-05
**PRD Requirements**: FR-21.1
**Priority**: P1
**Status**: Draft (planned 2026-06-12; notebooks not yet authored)
**Status**: Built (notebooks under `notebooks/kaggle-demo/`; workspace job residual)

## Story

Expand Down Expand Up @@ -65,29 +65,37 @@ dataset-specific code, so any similarly-shaped CSV drops in.

## Acceptance Criteria

- [ ] **US-044-AC1 (land + profile + spec)** — Given the Kaggle CSV
- [x] **US-044-AC1 (land + profile + spec)** — Given the Kaggle CSV
staged in a volume and a `source: {kind: delimited}` declaration, when
notebook 02 lands the file through the ingestion reader seam and runs
`NativeSparkProfiler` + `SparkToUmfMapper`, then the file lands
all-STRING raw with reader options derived from the declaration (no
hardcoded reader), and one UMF spec is emitted that passes
`tablespec validate` with zero errors and zero manual edits.
- [ ] **US-044-AC2 (schema workbook)** — Given the derived UMF, when
**Evidence**: `notebooks/kaggle-demo/02-kaggle-tablespec-demo.py`
(land → profile → map → validate cells; AC1 scorecard row).
- [x] **US-044-AC2 (schema workbook)** — Given the derived UMF, when
`tablespec export-excel` runs, then a workbook is produced whose rows
match the UMF columns/types and a re-import round-trips without loss
(FEAT-009 contract).
- [ ] **US-044-AC3 (artifacts + staged validation)** — Given the derived
**Evidence**: notebook 02 Excel export + round-trip assert (AC2).
- [x] **US-044-AC3 (artifacts + staged validation)** — Given the derived
UMF, when artifact generation and staged validation run, then compiled
artifacts (raw DDL, ingest SQL, expectation suites) are produced
deterministically from the spec, and staged validation against the
landed table yields a report with real per-expectation results (no
silent `success=False` stubs).
- [ ] **US-044-AC4 (demo lane + swappability)** — Given the notebook
**Evidence**: notebook 02 compile + staged validation cells (AC3).
- [x] **US-044-AC4 (demo lane + swappability)** — Given the notebook
pair committed under `notebooks/kaggle-demo/`, when it runs as a
Databricks workspace job with the default dataset widgets, then the
job exits PASS; and when the widgets point at a different
similarly-shaped CSV, notebook 02 runs unmodified (dataset-specific
values appear only in widgets and notebook 01).
**Evidence**: notebook pair + README “Swapping datasets”; scorecard
asserts no dataset-specific code in notebook 02.
**Limitation**: workspace job PASS/FAIL is not CI-gated (Databricks
residual, same posture as US-039/US-045).

## Edge Cases

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ ddx:
**Feature**: FEAT-034 — App Deployment & Configuration
**PRD Requirements**: FR-23.1, FR-23.2, FR-23.4
**Priority**: P1
**Status**: Specified
**Status**: Built

## Story

Expand Down Expand Up @@ -36,12 +36,18 @@ US-048; startup diagnostics are US-049.

## Acceptance Criteria

- [ ] **US-047-AC1** — Given deployment inputs naming a catalog, schema, volume, and compute, when the app is deployed, then it reads and writes metadata only at the declared location.
- [ ] **US-047-AC2** — Given two deployments targeting different environments, when both are deployed from the same commit, then the tracked application source is byte-identical between them and only declared inputs differ.
- [ ] **US-047-AC3** — Given a setting supplied both as a deployment input and in the connection registry, when the app resolves configuration, then the deployment input wins.
- [ ] **US-047-AC4** — Given a setting absent from both deployment inputs and the connection registry, when the app resolves configuration, then the built-in default is used and the resolved value is reported.
- [ ] **US-047-AC5** — Given a search of tracked application source for environment-identifying literals (catalog, schema, volume, compute identifier, workspace URL), when the search runs, then it returns no matches.
- [ ] **US-047-AC6** — Given a running deployment, when the operator views the app, then the resolved metadata location is displayed without opening source or configuration files.
- [x] **US-047-AC1** — Given deployment inputs naming a catalog, schema, volume, and compute, when the app is deployed, then it reads and writes metadata only at the declared location.
**Evidence**: `apps/data-profiling/tests/test_config.py` (`@covers US-047-AC1`); `profiler/config.py` `resolve_config`.
- [x] **US-047-AC2** — Given two deployments targeting different environments, when both are deployed from the same commit, then the tracked application source is byte-identical between them and only declared inputs differ.
**Evidence**: `app.yaml` declared env inputs; `test_fr23_stack.py` (`@covers US-047-AC2`); deployment-checklist `app_deploy`.
- [x] **US-047-AC3** — Given a setting supplied both as a deployment input and in the connection registry, when the app resolves configuration, then the deployment input wins.
**Evidence**: `test_config.py` precedence tests (`@covers US-047-AC3`).
- [x] **US-047-AC4** — Given a setting absent from both deployment inputs and the connection registry, when the app resolves configuration, then the built-in default is used and the resolved value is reported.
**Evidence**: `test_config.py` default-tier tests (`@covers US-047-AC4`).
- [x] **US-047-AC5** — Given a search of tracked application source for environment-identifying literals (catalog, schema, volume, compute identifier, workspace URL), when the search runs, then it returns no matches.
**Evidence**: `TestNoEnvironmentLiterals` in `test_config.py` (`@covers US-047-AC5`); generic defaults.
- [x] **US-047-AC6** — Given a running deployment, when the operator views the app, then the resolved metadata location is displayed without opening source or configuration files.
**Evidence**: `AppConfig.describe` / streamlit metadata caption; stack path `@covers US-047-AC6`.

## Edge Cases

Expand Down
Loading
Loading