feat(models): port production UMF model extensions from pulseflow

[dmautz1]

Summary

First PR in a series porting production-hardened capabilities from the pulseflow fork of tablespec back upstream (models first — they unblock the SQL generation, ingestion, and validation ports that follow). All additions are adapted to this repo's structure and conventions rather than copied verbatim, and every ported field is wired through its upstream consumers so nothing lands as a model-only no-op.

New model capabilities

Upsert merge semantics

• MergeCondition + UMFColumn.merge_strategy / merge_source / merge_condition — column-level merge behavior during upsert (keep_minimum→LEAST, keep_maximum→GREATEST, keep_existing→COALESCE), optionally gated on a source-row condition (e.g. only for certain load modes)
• UMF.effective_primary_key + DEFAULT_PRIMARY_KEY (meta_checksum) — default merge key for row-level dedup when no explicit PK is configured
• IngestionConfig.update_mode (upsert | update_only) — how update-file rows merge into the ingested table

SQL plan generation controls (consumed by a follow-up SQLPlanGenerator port)

• UMFMetadata.base_table_filter, base_join_column, final_filter, union_base_tables, union_type, union_exclude_base, union_coalesce_base, final_dedup
• ForeignKey.join_filter — extra predicate appended to the JOIN ON clause
• OutgoingRelationship.alternative_joins — OR-logic alternative join paths
• DerivationCandidate.union_value — per-UNION-branch literal for synthetic columns

Internal helper columns — wired end to end

• UMFColumn.internal: participates in derivation but excluded from the output table. Enforced in generate_sql_ddl / generate_pyspark_schema / generate_json_schema, and in GX baseline column-level expectations, column-count/ordered-list structural checks, and cross-column date pairs

Cross-pipeline references

• TableReference model (models/pipeline.py) with parse(), is_external(), to_qualified_name()
• parse_table_reference() on ForeignKey, OutgoingRelationship, IncomingRelationship, DerivationCandidate

Report workbook authoring

• UMFColumn.report_sheet: worksheet tab assignment for multi-sheet workbook reports (bounded to Excel's 31-char tab-name limit), with full Excel export/import round-trip (header-located read, so older workbooks import unchanged)
• Excel export now emits Survivorship/Derivations sheets only for tables that actually have derivation metadata — source-layer specs get a lean workbook (matches the existing conditional Validation/Relationships/File Format sheets; importer already tolerated absent sheets)

Boolean nullable — wired end to end

• UMFColumn.nullable accepts plain booleans alongside Nullable contexts. Consumers fixed to honor it: GX baseline emits the global not-null for nullable: false (previously silently skipped as falsy), Excel export writes the bool into every context cell (previously defaulted all contexts to False), and the compatibility checker maps bool to a synthetic "*" context (previously crashed on model_fields_set) so true → false is still a breaking tightening

Quality-of-life

• ValidationRules warns when ingested-stage expectation types are placed in raw validation_rules (shift-left guidance)
• Domain type registry cached at module level instead of re-constructed on every column validation

Deliberately NOT ported (upstream direction wins)

• PySpark-style data_type names/validation — this repo's SQL-style type system + EMBEDDING stands
• bool→LOB dict nullable normalization — conflicts with the generalized arbitrary-context Nullable
• lowercase table_name constraint and required canonical_name
• Advisory warnings for missing/nullable primary keys — would break the filterwarnings = error test policy repo-wide; can be revisited as opt-in lint checks
• UMF.mtime cache timestamp — deferred to the loader port that actually consumes it

Fields whose consumers arrive in later port batches (UMFMetadata SQL-gen controls, alternative_joins, join_filter, union_value → SQL generator; merge_*, update_mode, effective_primary_key → ingestion/upsert) are intentionally declarative-only in this PR.

Testing

• New test classes covering every ported field and every consumer wiring: model round-trips, generator exclusion, GX baseline internal/bool-nullable behavior, compatibility bool-nullable, Excel round-trips (report_sheet, bool nullable, lean source-spec workbooks)
• Two existing validator tests gained filterwarnings markers for the new (correct) misclassification warning
• Merged with latest main; full local run post-merge: 2,923 unit + 361 integration/conformance pass; ruff clean; pyright 0 errors on touched modules

🤖 Generated with Claude Code