feat: Great Expectations Exporter v2.0 - Enhanced Semantic Naming & Deduplication

[julienguilhempartner-spec]

Objective

Improve Great Expectations expectation generation by making them more semantically rich, intelligible, and free of redundancies through humanized names, intelligent deduplication logic, and comprehensive metadata enrichment for governance and traceability.

🚀 Changes Implemented

1. Removal of expect_table_columns_to_match_ordered_list

This expectation is removed from all exports as it's rarely useful in practice
Simplifies the generated expectation suite and reduces noise in validation reports

2. Enrichment of expect_column_values_to_be_of_type Metadata

Before: empty meta {}
After: enriched meta with expectation_id, name, description, dimension, severity, checkType
Example: order_id_must_be_of_type_varchar

3. Intelligent Deduplication of Rules

When primaryKey: true:
✅ Generates primary_key_not_null and primary_key_unique
❌ Skips standalone required and unique (to avoid duplicates)

When primaryKey: false:
✅ Generates not_null if required: true
✅ Generates unique if unique: true

4. Humanized Expectation Names Based on Column Names

Meta contains readable descriptions using the actual column name:

• "{Column Name} must be of type {type}"
• "{Column Name} must be filled"
• "{Column Name} must be unique"
• "{Column Name} must belong to allowed values"

5. Standardized expectation_id in snake_case

Before: short technical IDs (column_type, not_null, pattern_match)
After: semantic names in snake_case based on column names
Examples:

• order_id_must_be_of_type_varchar
• email_must_match_pattern_email
• quantity_must_be_between_0_and_9999

6. Constraint-Based Expectations Enrichment

All automatically-generated constraint expectations now include semantic governance metadata:

Added to constraint meta blocks:

{
  "severity": "critical",
  "checkType": "technical"
}

This enables:
- **Priority Categorization:** severity flag distinguishes critical vs. minor data quality issues
- **Check Classification:** checkType indicates whether the check is technical (type validation) or functional (business logic)
- **Governance Integration:** Allows downstream tools (data governance platforms, data catalogs) to filter and prioritize expectations by severity and type

**Affected Constraints:**
- Unique constraints → `severity: "critical"`
- Required constraints → `severity: "critical"`
- Primary key constraints → `severity: "critical"`
- LogicalTypeOptions (pattern, enum, minLength, etc.) → `severity: "critical"`

### 7. Contract-Level Metadata Enrichment
Suite-level meta blocks now include contract identity information for full traceability:

**Added to suite meta:**
```json
{
  "contract_id": "<data-contract-id>",
  "contract_version": "<contract-version>"
}

This enables:

• End-to-End Traceability: Each expectation suite can be traced back to its source data contract
• Version Tracking: Ensures audit trail between contract versions and expectation suite versions
• Impact Analysis: Facilitates impact assessment when contracts are modified or deprecated
• Lineage Documentation: Enables automated documentation of expectations' origin

📊 LogicalTypeOptions Coverage

All major ODCS options are now supported:

Option	Great Expectations Expectation
minLength / maxLength	expect_column_value_lengths_to_be_between
minimum / maximum	expect_column_values_to_be_between
exclusiveMinimum / exclusiveMaximum	expect_column_values_to_be_between + exclusive flag
pattern	expect_column_values_to_match_regex
format (email, url, uuid, etc.)	expect_column_values_to_match_regex (pre-built regex)
enum	expect_column_values_to_be_in_set

✅ Testing & Quality Assurance

• 32 tests all passing ✅
• Ruff linting: Zero errors ✅
• Complete coverage:
  • Type casting for all engines (Spark, Pandas, SQL, Trino)
  • Deduplication logic (primaryKey + required/unique)
  • Format validation (email, url, uuid, etc.)
  • Numeric ranges (inclusive & exclusive bounds)
  • String patterns & length constraints
  • Date ranges
  • Enum validation
  • Custom quality checks
  • Severity and checkType metadata on all constraints
  • Contract ID and version in suite meta

📝 Technical Details

• Expectation names and IDs are now fully traceable and debuggable
• Consistent naming based on actual database column names ensures reproducibility
• Backward compatible: existing expectations continue to work
• Enhanced metadata structure provides governance-ready data quality definitions
• Contract traceability: suite-level meta enables linking expectations back to source contracts
• Severity-based filtering: enables priority-driven data quality monitoring

Implementation Pattern

All constraint-derived expectations receive the enhanced meta structure:

"meta": {
  "expectation_id": "{contract_id}.{column}.{semantic_rule_name}",
  "rule_location": "quality_column",
  "name": "{Column Name} must {semantic_rule}",
  "description": "{semantic_rule} description",
  "dimension": "{quality_dimension}",
  "severity": "critical",
  "checkType": "technical",
}

Suite-level meta structure:

"meta": {
  "contract_id": "{odcs.id}",
  "contract_version": "{odcs.version}"
}

📂 Files Modified

• great_expectations_exporter.py
• test_export_great_expectations.py
• datacontract_all_constraints.yaml
• datacontract_quality_meta.yaml

🎯 Benefits

✅ Richer metadata for better governance and traceability
✅ Semantic clarity through humanized names and descriptions
✅ Improved prioritization with severity and checkType fields
✅ Full contract lineage with contract ID and version tracking
✅ Enhanced integration with data governance platforms and tools