fix: 0.7.13 batched fixes (HTTP backend integrity, query platform filter, trace depth, SQL lineage extraction)

CHANGELOG.md

@@ -1,5 +1,38 @@
 # Changelog
 
+## v0.7.13 (Unreleased)
+
+Compatibility notes: this is a patch release — no public surface is removed. The `excel` extra is deprecated (unused since the scanner removal) and the strict full-protocol backend validation considered for this release was softened to a core-method duck check; both removals/tightenings are deferred to 0.8.0.
+
+- fix: `Ledger(HttpLedgerBackend(url))` no longer corrupts the remote ledger. `append_snapshot` sends the real model name instead of `""` (which auto-registered a phantom empty-name model and attached every recorded event to it) and raises `ModelNotFoundError` for unresolvable hashes; registration logs exactly one `registered` event instead of double-firing (re-registration is now idempotent); `get_snapshot()` serves appended snapshots from a client-side cache and reconstructs older ones from `GET /changelog` instead of always returning `None` (hashes rebuilt from the changelog match server-minted identities; hashes of snapshots created client-side by `Ledger.record()` are process-local and resolve via the cache only). Defense in depth: `RecordInput.model_name` requires `min_length=1` and rejects whitespace-only names, so old clients that POST an empty model name get a 422 instead of silently corrupting the inventory. A new integration suite drives the SDK against the real `create_app()` handlers (not mocks).
+- fix: `HttpLedgerBackend.list_snapshots` fetches the model's full history. It called `GET /changelog` without a `since` bound, inheriting the changelog tool's 7-day default — so for any model whose events were older than a week, `get_snapshot` reconstruction returned `None`, `latest_snapshot` returned `None` (making `Ledger.tag` raise on healthy old models), re-registration lost its idempotency check, and the `platform` query filter returned zero results. It now passes an explicit all-time `since`. `list_models` also caches name mappings for the hashes it hands out, so they stay resolvable within the process (`batch_platforms`, and with it the platform filter, works from a fresh client).
+- fix: registration over `HttpLedgerBackend` carries the caller's payload, tier, and actor to the server. `Ledger.register()` dispatches to an optional backend `register_model` hook (implemented by the HTTP backend as a single `POST /record`); previously the payload was silently dropped, and every remote registration landed as `tier="unclassified"`, `actor="user"`. `RecordInput` gains an optional `tier` field (additive).
+- fix: the `query` tool's `platform` filter is actually applied. It was advertised in `QueryInput` and the REST/MCP schemas but never read — any platform value returned the full inventory. Platform is resolved from snapshot data in one batched `batch_platforms` dispatch and filtered/paginated in the tool; a schema-vs-implementation parity test now guards every `QueryInput` filter field.
+- fix: the introspector registry dispatches XGBoost sklearn-API models to the xgboost introspector. `SklearnIntrospector.can_handle` now rejects estimators from wrapper frameworks with dedicated introspectors (xgboost, lightgbm) instead of claiming them via `BaseEstimator`, making dispatch independent of entry-point ordering.
+- fix: SQL lineage extraction accepts unqualified (bare) table names — `INSERT INTO feature_store SELECT * FROM raw_events` previously parsed to zero reads and zero writes, silently producing no lineage. CTE names (`WITH ... AS`, including nested and `RECURSIVE` forms) and `DELETE FROM` targets are excluded from read tables, so dbt-style SQL does not produce phantom lineage inputs; a keyword blocklist keeps subqueries, table functions, and row sources out of the results (a bare table genuinely named a filtered keyword must be schema-qualified to extract); `INSERT OVERWRITE` is recognized, including the Hive/Spark `INSERT OVERWRITE TABLE t` form.
+- fix: the `trace` tool reports real BFS depth instead of fabricating it from list position. `depth` was computed from each node's index in the flat transitive list, so any model with N transitive upstreams rendered as an N-deep linear chain (a rich 8-level fan-out graph showed as "43 nodes at depths 1–43"); downstream had the same bug. Traversal is now a breadth-first walk that records each node's actual level (shortest edge distance, one entry per node at its minimum depth) and terminates at `input.depth` — bounded queries now do bounded work instead of filtering after a full traversal. `total_nodes` counts distinct models (a node reachable both upstream and downstream with `direction="both"` was counted twice).
+- fix: `strip_template_vars` strips spaced template variables like `{{ ds }}`; its docstring example now shows the actual output.
+- fix: `sql_connector` raises a clear `TypeError` naming the dict-row requirement when the connection yields tuple rows, instead of an opaque indexing error; name-addressable rows without the full dict API (e.g. `sqlite3.Row`, which has `keys()` but no `.get()`) are accepted. The connectors guide documents the requirement with a `row_factory = sqlite3.Row` example.
+- fix: `create_app()` / `create_server()` validate the backend at construction and raise a clear `TypeError` (with hints for the common Ledger-instead-of-backend and path-string mistakes) instead of failing deep inside the first request with `AttributeError`. Validation is a core-method duck check, not full-protocol `isinstance`, so partial custom backends that worked on 0.7.12 keep working.
+- fix(cli): `export --output` help text now says "Output file path" (the command writes a single file, not a directory) and the success message prints the real artifact path. The default output filename is unchanged (`audit_pack`).
+- feat: `Ledger.register()` accepts an optional `payload` dict merged into the registration event's payload (caller keys win) — used by the record tool so the single `registered` event still carries the caller's payload, over local and HTTP backends alike. `Ledger` also gains a public read-only `backend` property.
+- chore: deprecate the `excel` extra — nothing has imported openpyxl since the scanner module was removed post-v0.7.7. The extra is kept so `model-ledger[excel]` pins keep resolving within 0.7.x; removal planned for 0.8.0.
+- test: pandas-dependent tests skip instead of hard-failing when pandas is absent alongside the ML introspection libs.
+- test: introspector dispatch tests execute in every environment via stub sklearn/xgboost/lightgbm modules (the real-library variants still run where the ML extras are installed) — previously all dispatch tests skipped in CI.
+
+## v0.7.12
+
+- fix: escape backslashes in single-quoted SQL literals — JSON payloads containing escaped double quotes (`\"`) or backslashes reached `PARSE_JSON` mangled on the SQL fallback write path, which re-introduced the poisoned-buffer failure (INSERT fails, buffer never clears, every subsequent flush-before-read 500s) on common data. Also from the post-merge review of #33/#34: privilege denials short-circuit quietly again instead of warning per flush, unexpected pandas-path failures log with `exc_info`, and the failure-path staging-table DROP is gone. (#35)
+
+## v0.7.11
+
+- fix: the pandas bulk write path falls back to the DDL-free SQL path on ANY failure, not just privilege errors. Observed in production: `write_pandas` failing inside the file-transfer agent (not a privilege error) propagated, left the snapshot buffer poisoned, and 500'd every subsequent read on the process. Any pandas-path failure now logs a warning, best-effort drops the staging table, clears cleanly, and falls back. (#34)
+
+## v0.7.10
+
+- fix: the SQL fallback snapshot flush carries PAYLOAD and TAGS — the DDL-free path taken by least-privilege deployments previously inserted only the 7 scalar columns, silently persisting every snapshot with NULL payload and tags. (#33)
+- ci: publish to the official MCP Registry via GitHub OIDC on each release. (#32)
+
 ## v0.7.9
 
 - chore: publish to the official MCP Registry as `io.github.block/model-ledger` — adds `server.json` and the PyPI ownership marker in the README (#31)

docs/guides/connectors.md

@@ -38,6 +38,19 @@ ledger.add(etl_jobs.discover())
 ledger.connect()            # links ETL outputs to model inputs automatically
 ```
 
+!!! note "The connection must return name-addressable rows"
+    `sql_connector` addresses row values by column name (`row["owner"]`), so the
+    connection's `execute()` must return mappings, not raw DB-API tuples — tuple
+    rows raise a `TypeError` at discovery. For `sqlite3` the stdlib row type is
+    enough; for most other drivers, use the `DictCursor` equivalent:
+
+    ```python
+    import sqlite3
+
+    conn = sqlite3.connect("registry.db")
+    conn.row_factory = sqlite3.Row
+    ```
+
 ## REST APIs
 
 Works with MLflow, SageMaker, Vertex AI, or any JSON API:

docs/installation.md

@@ -27,7 +27,7 @@ uv add model-ledger
 | `model-ledger[introspect-sklearn]` | scikit-learn introspector | extract algorithm/features from fitted models |
 | `model-ledger[introspect-xgboost]` | XGBoost introspector | " |
 | `model-ledger[introspect-lightgbm]` | LightGBM introspector | " |
-| `model-ledger[excel]` | openpyxl | spreadsheet import/export |
+| `model-ledger[excel]` | openpyxl | deprecated (unused since the scanner removal); kept for pin compatibility, removal planned for 0.8.0 |
 | `model-ledger[all]` | Snowflake + pandas + httpx | the common production set |
 
 Combine them: `pip install "model-ledger[mcp,rest-api,snowflake]"`.

pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "model-ledger"
-version = "0.7.12"
+version = "0.7.13"
 description = "Developer-first model inventory and governance framework for SR 11-7, EU AI Act, and NIST AI RMF compliance"
 readme = "README.md"
 requires-python = ">=3.10"
@@ -43,6 +43,9 @@ dependencies = [
 
 [project.optional-dependencies]
 cli = ["typer>=0.9", "rich>=13.0"]
+# Deprecated: nothing imports openpyxl since the scanner module was removed
+# (post-0.7.7). Kept so `model-ledger[excel]` pins keep resolving within
+# 0.7.x; removal planned for 0.8.0.
 excel = ["openpyxl>=3.1"]
 introspect-sklearn = ["scikit-learn>=1.0"]
 introspect-xgboost = ["xgboost>=1.7"]

src/model_ledger/adapters/sql.py

@@ -9,24 +9,95 @@
 
 import re
 
+# Bare (unqualified) identifiers that can follow FROM/JOIN/INTO in valid SQL
+# without naming a table — subqueries, table functions, literal row sources,
+# and statement keywords. Qualified names (with dots) are never filtered.
+_SQL_NON_TABLE_KEYWORDS = frozenset(
+    {
+        "select",
+        "lateral",
+        "unnest",
+        "table",
+        "values",
+        "dual",
+        "generator",
+        "into",
+        "overwrite",
+        "if",
+        "not",
+        "exists",
+        "or",
+        "replace",
+    }
+)
+
+_TABLE_IDENTIFIER = r"([a-zA-Z_][a-zA-Z0-9_]*(?:\.[a-zA-Z_][a-zA-Z0-9_]*){0,2})"
+
+# A CTE binding: `WITH [RECURSIVE] name [(cols)] AS (` or a comma-continued
+# `, name [(cols)] AS (`. The comma form is only meaningful inside a WITH
+# clause, so _extract_cte_names guards on the statement containing WITH.
+_CTE_BINDING_PATTERN = re.compile(
+    r"(?:\bWITH\s+(?:RECURSIVE\s+)?|,\s*)"
+    r"([a-zA-Z_][a-zA-Z0-9_]*)\s*(?:\([^()]*\)\s*)?AS\s*\(",
+    re.IGNORECASE,
+)
+
+# `DELETE FROM target` — the target is neither a read nor (for lineage
+# purposes here) a write; strip the FROM so the read extractor skips it.
+_DELETE_FROM_PATTERN = re.compile(r"\bDELETE\s+FROM\b", re.IGNORECASE)
+
+
+def _extract_cte_names(sql: str) -> set[str]:
+    """Lowercased WITH-clause binding names (including nested/recursive CTEs).
+
+    CTE names are query-local aliases: a reference to one resolves to the
+    CTE for the whole statement (even when it shadows a real table), so
+    they must never surface as read tables.
+    """
+    if not re.search(r"\bWITH\b", sql, re.IGNORECASE):
+        return set()
+    return {m.group(1).lower() for m in _CTE_BINDING_PATTERN.finditer(sql)}
+
+
+def _is_table_name(identifier: str) -> bool:
+    """True unless a bare identifier is really a SQL keyword."""
+    return "." in identifier or identifier.lower() not in _SQL_NON_TABLE_KEYWORDS
+
 
 def extract_tables_from_sql(sql: str | None) -> list[str]:
     """Extract table names from SQL FROM and JOIN clauses.
 
+    Handles fully-qualified (``db.schema.table``), schema-qualified
+    (``schema.table``), and bare (``table``) identifiers.
+    CTE names (``WITH name AS (...)``) and ``DELETE FROM`` targets are
+    excluded — neither is a read table.
     Returns deduplicated list preserving first-seen order.
 
+    Note:
+        A bare table genuinely named one of the filtered SQL keywords
+        (``values``, ``table``, ``dual``, ...) is not extracted; qualify it
+        (``schema.values``) to force extraction.
+
     Example:
         >>> extract_tables_from_sql("SELECT * FROM schema.table1 JOIN schema.table2 ON 1=1")
         ['schema.table1', 'schema.table2']
+        >>> extract_tables_from_sql("SELECT * FROM raw_events")
+        ['raw_events']
     """
     if not sql:
         return []
-    pattern = r"(?:FROM|JOIN)\s+([a-zA-Z_][a-zA-Z0-9_]*(?:\.[a-zA-Z_][a-zA-Z0-9_]*){1,2})"
+    cte_names = _extract_cte_names(sql)
+    sql = _DELETE_FROM_PATTERN.sub("DELETE", sql)
+    pattern = rf"(?:FROM|JOIN)\s+{_TABLE_IDENTIFIER}"
     matches = re.findall(pattern, sql, re.IGNORECASE)
     seen: set[str] = set()
     result: list[str] = []
     for table in matches:
+        if not _is_table_name(table):
+            continue
         lower = table.lower()
+        if lower in cte_names:
+            continue
         if lower not in seen:
             seen.add(lower)
             result.append(table)
@@ -36,24 +107,33 @@ def extract_tables_from_sql(sql: str | None) -> list[str]:
 def extract_write_tables(sql: str | None) -> list[str]:
     """Extract tables that a SQL statement writes to.
 
-    Handles INSERT INTO, CREATE [OR REPLACE] TABLE, MERGE INTO.
+    Handles INSERT INTO, INSERT OVERWRITE [TABLE|INTO], CREATE [OR REPLACE]
+    TABLE, MERGE INTO, with qualified or bare table identifiers.
+
+    Note:
+        A bare table genuinely named one of the filtered SQL keywords
+        (``values``, ``table``, ``dual``, ...) is not extracted; qualify it
+        (``schema.values``) to force extraction.
 
     Example:
         >>> extract_write_tables("INSERT INTO schema.output SELECT * FROM source")
         ['schema.output']
+        >>> extract_write_tables("INSERT INTO feature_store SELECT * FROM raw_events")
+        ['feature_store']
     """
     if not sql:
         return []
-    table_pattern = r"([a-zA-Z_][a-zA-Z0-9_]*(?:\.[a-zA-Z_][a-zA-Z0-9_]*){1,2})"
     results: list[str] = []
     seen: set[str] = set()
 
     for pattern in [
-        rf"INSERT\s+(?:INTO\s+)?{table_pattern}",
-        rf"CREATE\s+(?:OR\s+REPLACE\s+)?TABLE\s+(?:IF\s+NOT\s+EXISTS\s+)?{table_pattern}",
-        rf"MERGE\s+INTO\s+{table_pattern}",
+        rf"INSERT\s+(?:OVERWRITE\s+)?(?:TABLE\s+|INTO\s+)?{_TABLE_IDENTIFIER}",
+        rf"CREATE\s+(?:OR\s+REPLACE\s+)?TABLE\s+(?:IF\s+NOT\s+EXISTS\s+)?{_TABLE_IDENTIFIER}",
+        rf"MERGE\s+INTO\s+{_TABLE_IDENTIFIER}",
     ]:
         for match in re.finditer(pattern, sql, re.IGNORECASE):
+            if not _is_table_name(match.group(1)):
+                continue
             t = match.group(1).lower()
             if t not in seen:
                 seen.add(t)
@@ -152,13 +232,17 @@ def extract_comment_tags(sql: str | None, prefix: str = "@") -> dict[str, str]:
 def strip_template_vars(sql: str | None) -> str:
     """Strip {{var}} template wrappers from SQL.
 
-    Converts {{schema}}.{{table}} → schema.table.
-    Common in ETL platforms that use template variables in SQL.
+    Converts ``{{schema}}.{{table}}`` → ``schema.table``. Whitespace inside
+    the braces is tolerated, so spaced variables like ``{{ ds }}`` (common
+    in scheduler-templated SQL) are stripped too. Common in ETL platforms
+    that use template variables in SQL.
 
     Example:
         >>> strip_template_vars("SELECT * FROM {{schema}}.{{table_name}}")
-        'SELECT * FROM app_compliance.cash.table'
+        'SELECT * FROM schema.table_name'
+        >>> strip_template_vars("SELECT * FROM {{ analytics }}.{{ events }}")
+        'SELECT * FROM analytics.events'
     """
     if not sql:
         return ""
-    return re.sub(r"\{\{(\w+)\}\}", r"\1", sql)
+    return re.sub(r"\{\{\s*(\w+)\s*\}\}", r"\1", sql)