Skip to content

fix(nyc-taxi): stop tag/term attachments overwriting each other; create owner corpGroup; honest dry-run - #220

Open
simpleciki wants to merge 2 commits into
datahub-project:mainfrom
simpleciki:simpleciki-patch-1
Open

fix(nyc-taxi): stop tag/term attachments overwriting each other; create owner corpGroup; honest dry-run#220
simpleciki wants to merge 2 commits into
datahub-project:mainfrom
simpleciki:simpleciki-patch-1

Conversation

@simpleciki

Copy link
Copy Markdown

Fixes #218. Also fixes the Windows Unicode crash reported in #211 for this script.

Four fixes in datasets/nyc-taxi/add_metadata.py:

  1. Per-table accumulationattach_tags/attach_glossary now build the full association list per table and emit one aspect per table, so attachments no longer overwrite each other (previously each emit replaced the whole globalTags/glossaryTerms aspect with a single association — last write wins, and every table ended up with only pipeline_stage). Verified on a quickstart instance (server v1.5.0.6): end state matches the README's Metadata section exactly — raw_trips/staging_trips carry 4 tags + 2 terms, mart carries daily_refresh + pipeline_stage and Freshness SLA + Empty Load + Pipeline Stage.
  2. Create the owner group — new create_groups() emits corpGroupInfo for data_platform_team before ownership attachment; the owner chip in the UI now resolves instead of "Not Found".
  3. Honest dry-run--dry-run now prints the planned tag/term/ownership operations and ends with "DRY RUN — no changes made" instead of the unconditional "✅ Metadata complete" banner (matching what add_lineage.py --dry-run already does).
  4. Windows-safe output — the ✓/→ characters crash default cp1252 Windows consoles (Sample dataset scripts crash on Windows (cp1252): UnicodeEncodeError printing check marks #211, reported by another hackathon participant); a two-line sys.stdout.reconfigure at startup fixes it for this script (tested on a native cp1252 console, exit 0). The same two-liner applies to the other dataset scripts if you'd like a follow-up.

No changes to lineage, ingestion recipes, or the .db files.

…te owner corpGroup; honest dry-run

Per-table accumulation for globalTags/glossaryTerms aspects (one emit per table with the full list — no more last-write-wins); create the data_platform_team corpGroup entity before attaching ownership; --dry-run now prints planned changes and "DRY RUN — no changes made"; UTF-8 stdout reconfigure so the script's Unicode output no longer crashes default Windows consoles (datahub-project#211).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

nyc-taxi: add_metadata.py tag/term attachments overwrite each other; ownership points to a group that is never created

1 participant