Skip to content

Reconcile own indices with pre-existing OHDSI-standard indices#36

Draft
nicoloesch wants to merge 11 commits into
mainfrom
index-reconciliation
Draft

Reconcile own indices with pre-existing OHDSI-standard indices#36
nicoloesch wants to merge 11 commits into
mainfrom
index-reconciliation

Conversation

@nicoloesch

Copy link
Copy Markdown
Collaborator

Note before merging

This PR targets the branch staging-overhaul. It depends on ReservedSchema.STAGING / orm_loader.backends.STAGING_SCHEMA, which only exists on staging-overhaul. To restore the history of this branch and exclude the staging changes again, we need to merge it after #29 was merged into main using

git rebase --onto origin/main origin/staging-overhaul index-reconciliation
git push --force-with-lease

Summary

  • Adds live, column-set-based reconciliation (_is_plain_index, _find_equivalent_index, _find_shape_conflict in cli_indexes.py) so indexes enable recognizes an index that already exists under a foreign name (e.g. from the official OHDSI CDM DDL script) and never creates a duplicate.
  • Adds capture/restore bookkeeping (a dropped_indexes table in a new reserved schema, omop_alchemy_maintenance on PostgreSQL) so indexes disable can safely drop such a foreign-named index for bulk-load speed, and a later indexes enable restores it under its original name. Includes schema-scoping to prevent collisions across two schemas.
  • Adds a ReservedSchema StrEnum (_cli_utils.py) centralizing STAGING and MAINTENANCE, guarded via reject_reserved_schema() wired into the omop_command decorator so every maintenance CLI command is protected, not just vocab load-vocab-source.
  • Updates cli_schema_reconcile.py so a foreign-named equivalent index reports a single "renamed" issue instead of a misleading "missing" + "unexpected" pair, and doesn't count as drift. This includes the CLUSTER comparison and in the doctor/summary rendering layers.
  • Fixes cli_vocab.py's bulk vocabulary loader silently discarding indexes disable's warning results; they now surface on VocabularyLoadReport.index_warnings and render as a dedicated panel.
  • Fixes a crash: a second indexes disable run (without an intervening enable) when a different foreign index appears on the same columns now degrades gracefully instead of raising an uncaught IntegrityError.
  • Fixes load_vocab_source()'s reserved-schema guard only covering STAGING, not MAINTENANCE, when called directly (bypassing the CLI).
  • Extracts a shared _resolve_physical_cluster_name() helper used by both manage_indexes()'s cluster step and the standalone indexes cluster command, fixing a hardcoded unique=False in the latter and removing a duplicated resolution policy that could have drifted out of sync.
  • Replaces every ad hoc status: str field across all ten cli_*.py maintenance modules with one shared typed Status StrEnum, each member carrying a Severity (OK/INFO/WARNING/ERROR, itself carrying a render style).
  • Relocates FullTextResult/FullTextAction from omop_alchemy/backends/base.py to omop_alchemy/maintenance/cli_fulltext.py.
  • Collapses manage_indexes()'s independent Optional outcome locals into a single _IndexOutcome discriminated dataclass, so two of them being set simultaneously by a future edit becomes structurally impossible

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Duplicate indices when a database is already indexed by the official OHDSI CDM DDL script

1 participant