feat(observation): add platform-neutral Databricks observed state

.agents/skills/semapact-system/SKILL.md

@@ -18,31 +18,44 @@ SemaPact is an enterprise data contract control plane and governance platform. I
 ## 2. Layered Architecture Boundaries (CRITICAL)
 
 ### A. Ingestion / Import Layer
-- **Role:** Converts external data structures (Delta Tables, Spark DDL, Unity Catalog) into Open Data Contract Standard (ODCS) models.
+- **Role:** Converts external data structures into Open Data Contract Standard (ODCS) models when a contract import is explicitly requested.
 - **Rules:**
   - Must remain strictly stateless and idempotent.
   - **NEVER** place merge, governance, or GitOps logic inside importers.
+  - Contract import is distinct from platform observation; observing a platform must not implicitly create or mutate an ODCS contract.
 
-### B. Core Contract Model
-- **Role:** Single canonical representation of the schema and metadata.
+### B. Governed Contract Model
+- **Role:** Single canonical representation of governed desired contract state.
 - **Rules:**
-  - The ODCS YAML/Pydantic model is the single source of truth across the architecture. No alternative models are allowed.
+  - The ODCS YAML/Pydantic model is the single source of truth for governed contract state.
+  - External platform state must not become governed truth merely because it was observed.
 
-### C. Lifecycle Governance Layer
+### C. Platform Observation Model
+- **Role:** Represents point-in-time external platform state for assurance and reconciliation workflows.
+- **Rules:**
+  - `ObservedPlatformState` is a read-side model, not an alternative canonical contract format.
+  - Core observation models must remain platform-neutral; provider hierarchy belongs in adapter-local mapping into a generic ordered `namespace`.
+  - Platform-local identity must remain distinct from ODCS contract identity.
+  - Provider adapters may reuse official platform SDK access, but must not route observation through ODCS import/projection.
+  - Observation must not invoke lifecycle merge, governance evaluation, release mutation, or platform writeback.
+  - Rich metadata, constraints, relationships, and lineage are evidence enrichments, not prerequisites for the minimal observed-state model.
+  - Converting observed state into an ODCS contract is an explicit import workflow, never an implicit observation side effect.
+
+### D. Lifecycle Governance Layer
 - **Role:** Handles breaking change checks, deprecation rules, merge policies, and version bump calculations.
 - **Rules:**
   - This is the **ONLY** place where contract lifecycle logic is allowed.
-  - It must remain fully decoupled from the UI and ingestion layers.
+  - It must remain fully decoupled from the UI, ingestion, and platform observation layers.
 
-### D. Export Layer
+### E. Export Layer
 - **Role:** Converts contracts to downstream assets (Great Expectations suites, Spark DDL, Graph cypher).
 - **Rules:**
   - Exporters must be read-only and **NEVER** modify the original contracts.
 
-### E. Orchestration Layer
+### F. Orchestration Layer
 - **Role:** Coordinates multi-step workflows (e.g. import → merge → export → PR).
 - **Rules:**
   - Coordinates execution paths but must NOT contain custom business logic.
 
-### F. DevOps Layer
+### G. DevOps Layer
 - **Role:** Automates PR creation, version bumps, release manifest building, and metadata auditing.

.github/workflows/ci.yml

@@ -43,6 +43,37 @@ jobs:
       - name: Verify package build
         run: uv build
 
+  databricks-extra:
+    runs-on: ubuntu-latest
+    timeout-minutes: 30
+    env:
+      UV_NO_PROGRESS: "1"
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v7
+
+      - name: Set up Python
+        uses: actions/setup-python@v7
+        with:
+          python-version: "3.12"
+
+      - name: Set up uv
+        uses: astral-sh/setup-uv@v7
+
+      - name: Create isolated environment
+        run: uv venv
+
+      - name: Install SemaPact Databricks extra from project metadata
+        run: uv pip install --python .venv/bin/python -e ".[databricks]" pytest
+
+      - name: Verify Databricks SDK and existing Unity import boundaries
+        run: >-
+          .venv/bin/python -c
+          "from databricks.sdk import WorkspaceClient; from databricks.sdk.service.catalog import TableInfo; from semapact.importers.unity_importer import import_unity_contract; print(WorkspaceClient.__name__, TableInfo.__name__, import_unity_contract.__name__)"
+
+      - name: Run Databricks observation tests with official SDK installed
+        run: .venv/bin/python -m pytest tests/test_observation_databricks.py
+
   coverage:
     runs-on: ubuntu-latest
     timeout-minutes: 30

ARCHITECTURE.md

@@ -97,7 +97,36 @@ Important boundary:
 - `GovernanceService` creates `ChangeContext`
 - lifecycle/governance lower layers consume that context and must not regenerate it
 
-### 5. Exporters
+### 5. Platform Observation
+
+Location:
+
+- `semapact/observation/`
+
+Responsibilities:
+
+- represent point-in-time external platform state independently from governed contracts
+- keep the core observed-state domain platform-neutral
+- map provider-local identity into `platform + ordered namespace + asset`
+- observe physical asset/property state without creating or mutating ODCS contracts
+
+Key modules:
+
+- `semapact/observation/models.py`
+- `semapact/observation/databricks.py`
+
+Important boundary:
+
+- `ObservedPlatformState` is read-side state, not governed truth
+- observed asset identity is platform-local and distinct from ODCS contract identity
+- provider-specific hierarchy such as Databricks `catalog/schema` belongs in the adapter, not the core model
+- Databricks observation consumes the official SDK `WorkspaceClient.tables.get(...) -> TableInfo` boundary instead of reimplementing the Unity Catalog REST transport
+- observation projects `TableInfo` directly into observed state; it must not route through datacontract-cli's ODCS projection
+- rich metadata, constraints, relationships, and lineage are follow-up evidence enrichments rather than prerequisites for the minimal observation model
+- observation must not invoke lifecycle merge, governance evaluation, release mutation, or platform writeback
+- explicit contract import remains a separate workflow
+
+### 6. Exporters
 
 Location:
 
@@ -115,7 +144,7 @@ Key modules:
 - `semapact/quality/ge_exporter.py`
 - `semapact/exporters/sql_exporter.py`
 
-### 6. Orchestration
+### 7. Orchestration
 
 Location:
 
@@ -130,7 +159,7 @@ Key module:
 
 - `semapact/orchestrator/pipeline.py`
 
-### 7. Interfaces
+### 8. Interfaces
 
 Location:
 
@@ -148,14 +177,28 @@ Current interface:
 - CLI in `semapact/interfaces/cli.py`
 - command adapters in `semapact/interfaces/commands/`
 
-## Current Contract Model
+## Governed Contract Model
+
+SemaPact assumes:
+
+- Open Data Contract Standard (ODCS) is the single canonical representation of governed desired contract state
+- `open_data_contract_standard.model.OpenDataContractStandard` is the canonical governed contract domain model
+
+The system may temporarily work with Python `dict` objects at contract boundaries, but contract normalization should converge back to ODCS objects or ODCS-shaped mappings.
+
+Platform observations are intentionally different. `ObservedPlatformState` is a non-canonical read-side model describing what an external platform reports at a point in time. It must not replace or mutate the governed ODCS contract.
+
+Conceptually:
 
-SemaPact currently assumes:
+```text
+Approved ODCS Contract
+= desired governed state
 
-- Open Data Contract Standard (ODCS) is the single canonical contract representation
-- `open_data_contract_standard.model.OpenDataContractStandard` is the canonical runtime model
+ObservedPlatformState
+= observed external platform state
+```
 
-The system may temporarily work with Python `dict` objects at boundaries, but normalization should converge back to ODCS objects or ODCS-shaped mappings.
+Converting external metadata into a new ODCS contract is an explicit import workflow. Observing platform state does not implicitly perform that conversion.
 
 ## Root Contract Governance
 
@@ -367,10 +410,13 @@ Precedence:
 
 ## Current Design Principles
 
-- main contract is canonical and immutable from presentation paths
+- main governed contract is canonical and immutable from presentation paths
+- ODCS is the canonical model for governed desired contract state
+- platform observation is separate read-side state and cannot become governed truth implicitly
+- core observation models are platform-neutral; provider-specific hierarchy belongs in adapters
+- reuse official platform access/SDK layers where practical, while keeping ODCS import projection separate from observation
 - service layer is the application boundary between interfaces and system logic
 - lifecycle logic belongs in the lifecycle layer
-- ODCS is the canonical contract model
 - datacontract-cli is reused where possible instead of reimplemented
 
 ## Authoritative Governance Invariants
@@ -396,6 +442,10 @@ Precedence:
 
 ## Known Next Steps
 
+- separate platform discovery, observation, and contract import application workflows, starting with Databricks Unity Catalog
+- add stable observed-state fingerprints and reconciliation semantics
+- add governance-relevant metadata/constraint/relationship evidence independently from the minimal observation model
+- add lineage as optional runtime evidence rather than a core observation dependency
 - formalize draft promotion flow
 - continue reducing interface-specific logic that still lives near command/editor helpers
-- keep converging helper logic toward ODCS model-driven behavior
+- keep converging governed contract helper logic toward ODCS model-driven behavior

pyproject.toml

@@ -74,7 +74,7 @@ tui = [
   "textual>=8.2.7",
 ]
 databricks = [
-  "databricks-sql-connector>=3.0.0",
+  "datacontract-cli[databricks]>=0.12.0",
   "pyspark>=3.3.0",
 ]
 azure = [
@@ -101,7 +101,7 @@ all = [
   "openai>=1.0.0",
   "litellm>=1.83.14",
   "textual>=8.2.7",
-  "databricks-sql-connector>=3.0.0",
+  "datacontract-cli[databricks]>=0.12.0",
   "pyspark>=3.3.0",
   "azure-identity>=1.16.0",
   "azure-storage-file-datalake>=12.16.0",

semapact/__init__.py

@@ -1,57 +1,96 @@
-"""SemaPact enterprise library."""
-
-from semapact.core.loader import ContractLoader, load_contract
-from semapact.core.validator import ContractValidator
-from semapact.devops.pr_creator import AzureDevOpsConfig, PullRequestCreator
-from semapact.devops.release_workflow import (
-    BatchReleaseManifestBuild,
-    BatchReleaseTask,
-    ReleasePullRequestPlan,
-    RepositoryContractChange,
-    batch_manifest_build_to_dict,
-    build_batch_release_manifest,
-    build_release_pr_plan,
-    create_release_pull_request,
-    create_release_pull_requests_from_manifest,
-    load_batch_release_tasks,
-    repository_change_to_dict,
-)
-from semapact.exporters.sql_exporter import (
-    SparkSqlContractExporter,
-    export_contract_to_spark_sql,
-)
-from semapact.importers.delta_importer import DeltaTableImporter
-from semapact.importers.sql_importer import SQLFolderImporter
-from semapact.lifecycle.merge_engine import ContractMergeEngine
-from semapact.lifecycle.policy import evaluate_merge_policy
-from semapact.orchestrator.pipeline import ContractPipeline
-from semapact.quality.ge_exporter import GreatExpectationsExporter
-from semapact.quality.validation import run_contract_tests
-
-__all__ = [
-    "ContractLoader",
-    "ContractValidator",
-    "DeltaTableImporter",
-    "SQLFolderImporter",
-    "ContractMergeEngine",
-    "evaluate_merge_policy",
-    "GreatExpectationsExporter",
-    "SparkSqlContractExporter",
-    "export_contract_to_spark_sql",
-    "ContractPipeline",
-    "PullRequestCreator",
-    "AzureDevOpsConfig",
-    "BatchReleaseManifestBuild",
-    "BatchReleaseTask",
-    "ReleasePullRequestPlan",
-    "RepositoryContractChange",
-    "batch_manifest_build_to_dict",
-    "build_batch_release_manifest",
-    "build_release_pr_plan",
-    "create_release_pull_request",
-    "create_release_pull_requests_from_manifest",
-    "load_batch_release_tasks",
-    "repository_change_to_dict",
-    "load_contract",
-    "run_contract_tests",
-]
+"""SemaPact enterprise library.
+
+The package root intentionally lazy-loads public exports so installing one
+optional platform capability does not import unrelated platform dependencies.
+"""
+
+from __future__ import annotations
+
+from importlib import import_module
+from typing import Any
+
+_EXPORTS: dict[str, tuple[str, str]] = {
+    "ContractLoader": ("semapact.core.loader", "ContractLoader"),
+    "load_contract": ("semapact.core.loader", "load_contract"),
+    "ContractValidator": ("semapact.core.validator", "ContractValidator"),
+    "AzureDevOpsConfig": ("semapact.devops.pr_creator", "AzureDevOpsConfig"),
+    "PullRequestCreator": ("semapact.devops.pr_creator", "PullRequestCreator"),
+    "BatchReleaseManifestBuild": (
+        "semapact.devops.release_workflow",
+        "BatchReleaseManifestBuild",
+    ),
+    "BatchReleaseTask": ("semapact.devops.release_workflow", "BatchReleaseTask"),
+    "ReleasePullRequestPlan": (
+        "semapact.devops.release_workflow",
+        "ReleasePullRequestPlan",
+    ),
+    "RepositoryContractChange": (
+        "semapact.devops.release_workflow",
+        "RepositoryContractChange",
+    ),
+    "batch_manifest_build_to_dict": (
+        "semapact.devops.release_workflow",
+        "batch_manifest_build_to_dict",
+    ),
+    "build_batch_release_manifest": (
+        "semapact.devops.release_workflow",
+        "build_batch_release_manifest",
+    ),
+    "build_release_pr_plan": (
+        "semapact.devops.release_workflow",
+        "build_release_pr_plan",
+    ),
+    "create_release_pull_request": (
+        "semapact.devops.release_workflow",
+        "create_release_pull_request",
+    ),
+    "create_release_pull_requests_from_manifest": (
+        "semapact.devops.release_workflow",
+        "create_release_pull_requests_from_manifest",
+    ),
+    "load_batch_release_tasks": (
+        "semapact.devops.release_workflow",
+        "load_batch_release_tasks",
+    ),
+    "repository_change_to_dict": (
+        "semapact.devops.release_workflow",
+        "repository_change_to_dict",
+    ),
+    "SparkSqlContractExporter": (
+        "semapact.exporters.sql_exporter",
+        "SparkSqlContractExporter",
+    ),
+    "export_contract_to_spark_sql": (
+        "semapact.exporters.sql_exporter",
+        "export_contract_to_spark_sql",
+    ),
+    "DeltaTableImporter": ("semapact.importers.delta_importer", "DeltaTableImporter"),
+    "SQLFolderImporter": ("semapact.importers.sql_importer", "SQLFolderImporter"),
+    "ContractMergeEngine": ("semapact.lifecycle.merge_engine", "ContractMergeEngine"),
+    "evaluate_merge_policy": ("semapact.lifecycle.policy", "evaluate_merge_policy"),
+    "ContractPipeline": ("semapact.orchestrator.pipeline", "ContractPipeline"),
+    "GreatExpectationsExporter": (
+        "semapact.quality.ge_exporter",
+        "GreatExpectationsExporter",
+    ),
+    "run_contract_tests": ("semapact.quality.validation", "run_contract_tests"),
+}
+
+__all__ = list(_EXPORTS)
+
+
+def __getattr__(name: str) -> Any:
+    """Resolve public exports only when callers actually request them."""
+    target = _EXPORTS.get(name)
+    if target is None:
+        raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
+
+    module_name, attribute = target
+    value = getattr(import_module(module_name), attribute)
+    globals()[name] = value
+    return value
+
+
+def __dir__() -> list[str]:
+    """Include lazy public exports in interactive discovery."""
+    return sorted({*globals(), *__all__})