Skip to content

feat(databricks): add explicit discovery workflow - #188

Merged
ElliotSun merged 11 commits into
mainfrom
11-separate-databricks-workflows
Aug 30, 2026
Merged

feat(databricks): add explicit discovery workflow#188
ElliotSun merged 11 commits into
mainfrom
11-separate-databricks-workflows

Conversation

@ElliotSun

@ElliotSun ElliotSun commented Aug 30, 2026

Copy link
Copy Markdown
Collaborator

Summary

Add the missing Databricks read-side discovery workflow for M1 while preserving the existing platform observation boundary.

This PR is scoped to Discovery + Observation only. Contract import is an already implemented SemaPact capability and is not implemented, refactored, or exercised here.

Read-side boundaries

Initialized WorkspaceClient
        │
        ├── Discovery
        │     tables.list(...)
        │        ↓
        │     table FQNs
        │
        └── Observation
              tables.get(...)
                 ↓
          ObservedPlatformState

The existing contract-import path remains separate and unchanged. Discovery and observation must not depend on or invoke it.

Changes

  • add semapact.platforms.databricks.discover_databricks_tables(...) backed by WorkspaceClient.tables.list(...);
  • return deterministic, deduplicated fully-qualified table names only;
  • require an initialized client for discovery and keep auth resolution outside the workflow;
  • keep discovery free of observation, ODCS import, lifecycle, and governance dependencies;
  • preserve observe_databricks_table(...) behavior unchanged;
  • rely on dependency-isolation tests to keep discovery/observation independent from importer and governance layers;
  • extend the fresh semapact[databricks] smoke to verify the official SDK read-side surfaces only.

Scope semantics

Discovery
→ Which Databricks assets are available in this scope?
→ Output: table FQNs

Observation
→ What state does Databricks currently report for a selected asset?
→ Output: ObservedPlatformState

Discovery does not imply observation. Observation does not imply contract import. Neither operation mutates governed ODCS state.

Out of scope

  • contract import implementation/refactoring/testing — already implemented;
  • new discovery/observation CLI commands;
  • authentication mechanisms;
  • reconciliation/drift classification;
  • observation fingerprints;
  • rich evidence/lineage;
  • release/deployment behavior.

@ElliotSun ElliotSun changed the title feat(databricks): separate discovery from observation and import feat(databricks): add explicit discovery workflow Aug 30, 2026
@ElliotSun
ElliotSun merged commit c278c97 into main Aug 30, 2026
9 checks passed
@ElliotSun
ElliotSun deleted the 11-separate-databricks-workflows branch August 30, 2026 10:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant