CMR-11155: Version Control Index Set [NOT READY FOR REVIEW] by jmaeng72 · Pull Request #2454 · nasa/Common-Metadata-Repository

jmaeng72 · 2026-06-24T18:50:43Z

Overview

What is the objective?

To allow for disaster recovery and keeping track of historical changes of index-set.

What are the changes?

Created new index-set table in database as new source of truth
Created new /sync-with-database api to be able to recover index-sets from database if we lose the index in elastic
Created new /index-sets/:id?revision_id=X to get different revision versions of the index-set for historical reference
Index set is treated like a concept now, old revisions are deleted up to 10 versions and there is a new revision created for every create, update, delete. Delete index-set is tombstoned in the db

What areas of the application does this impact?

Indexer

Required Checklist

New and existing unit and int tests pass locally and remotely
clj-kondo has been run locally and all errors in changed files are corrected
I have commented my code, particularly in hard-to-understand areas
I have made changes to the documentation (if necessary)
My changes generate no new warnings

Additional Checklist

I have removed unnecessary/dead code and imports in files I have changed
I have cleaned up integration tests by doing one or more of the following:
- migrated any are2 tests to are3 in files I have changed
- de-duped, consolidated, removed dead int tests
- transformed applicable int tests into unit tests
- reduced number of system state resets by updating fixtures. Ex) (use-fixtures :each (ingest/reset-fixture {})) to be :once instead of :each

…-- still not fully working

…sitory into CMR-11155

…rovider for tool and index-set

…sitory into CMR-11155

codecov-commenter · 2026-06-24T18:52:01Z

Codecov Report

❌ Patch coverage is 22.34432% with 212 lines in your changes missing coverage. Please review.
✅ Project coverage is 29.33%. Comparing base (0bcadbd) to head (ac4ffa5).

Files with missing lines	Patch %	Lines
...app/src/cmr/indexer/services/index_set_service.clj	4.95%	115 Missing ⚠️
indexer-app/src/cmr/indexer/data/elasticsearch.clj	0.00%	24 Missing ⚠️
indexer-app/src/cmr/indexer/api/routes.clj	0.00%	17 Missing ⚠️
...-test/src/cmr/system_int_test/utils/index_util.clj	52.17%	11 Missing ⚠️
...adata_db/migrations/093_setup_index_sets_table.clj	47.36%	10 Missing ⚠️
.../migrations/092_update_cmr_subscriptions_table.clj	0.00%	8 Missing ⚠️
...cmr/metadata_db/data/oracle/concepts/index_set.clj	30.00%	7 Missing ⚠️
metadata-db-app/src/config/mdb_migrate_helper.clj	14.28%	6 Missing ⚠️
...-test/src/cmr/system_int_test/utils/url_helper.clj	71.42%	4 Missing ⚠️
...exer-app/src/cmr/indexer/common/index_set_util.clj	0.00%	2 Missing ⚠️
... and 6 more

❗ There is a different number of reports uploaded between BASE (0bcadbd) and HEAD (ac4ffa5). Click for more details.

HEAD has 19 uploads less than BASE

Flag BASE (0bcadbd) HEAD (ac4ffa5)

23 4

Additional details and impacted files

@@             Coverage Diff             @@
##           master    #2454       +/-   ##
===========================================
- Coverage   58.20%   29.33%   -28.88%     
===========================================
  Files        1069     1011       -58     
  Lines       74278    71019     -3259     
  Branches     2166     1198      -968     
===========================================
- Hits        43233    20830    -22403     
- Misses      29028    49053    +20025     
+ Partials     2017     1136      -881

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

jmaeng72 · 2026-06-24T18:52:47Z

            collection-response (ingest/bulk-update-task-status "PROV1" (:task-id response))
            collection-statuses (:collection-statuses collection-response)]
        (is (= "COMPLETE" (:task-status collection-response)))
-        (is (= "Collection with concept-id [C1200000009-PROV1] is deleted. Can not be updated."


because we added a new index-set table which is treated like a concept, when these tests are run they were hardcoded to get the concept id as expected with the number of time the concept_seq_id would increment. Because index-set is also using the concept_seq_id to increment its id, the concept_seq_id was increased by one in all these tests.

jmaeng72 · 2026-06-24T18:56:35Z

   ;;     instead.
-   [cmr.metadata-db.data.oracle.concepts.generic-documents]))
+   [cmr.metadata-db.data.oracle.concepts.generic-documents]
+   [cmr.metadata-db.data.oracle.concepts.index-set]))


added to allow index set to be seen by metadata db during sys tests later

jmaeng72 · 2026-06-24T18:59:46Z

  (println "cmr.metadata-db.migrations.092-update-cmr-sub-notifications-table up...")
-  (h/sql "ALTER TABLE METADATA_DB.CMR_SUB_NOTIFICATIONS DROP COLUMN AWS_ARN")
-  (h/sql "ALTER TABLE METADATA_DB.CMR_SUBSCRIPTIONS ADD AWS_ARN VARCHAR(2048) NULL"))
+  (when (h/column-exists? "CMR_SUB_NOTIFICATIONS" "AWS_ARN")


had to add this because my local had a wrong version of AWS_ARN and had to be replaced. This is helpful to any other dev with local db issues. This will not have adverse affects since the logic still remains the same

jmaeng72 · 2026-06-24T19:00:04Z

+(def ^:private index-sets-column-sql
+  "id NUMBER,
+  concept_id VARCHAR(255) NOT NULL,
+  native_id VARCHAR(1030) NOT NULL,


native_id is the id of the index-set

jmaeng72 · 2026-06-24T19:02:36Z

       (let [reshard-start-resp (bootstrap/start-reshard-index "1_small_collections_100_shards" {:synchronous true :num-shards 50 :elastic-name gran-elastic-name})
-             task-id (:task-id reshard-start-resp)]
+             task-id (:task-id reshard-start-resp)
+             status-check-attempts (range 3)]


because some runs of this may be slow, it could take some time for reshard to finish, so I put 3 attempts here to prevent unnecessary failures on non-immediate completes

jmaeng72 · 2026-06-24T19:02:59Z

-                 :reshard-status "COMPLETE"}
-                (bootstrap/get-reshard-status "1_small_collections_100_shards" {:elastic-name gran-elastic-name :task-id task-id})))))
+         (run! (fn [i]
+                 (Thread/sleep 2000) ;; wait for 2 secs


put a wait here in case the reshard takes longer than expected

jmaeng72 added 17 commits May 19, 2026 16:47

update for consistency with combined vs separate es index-set saving …

a3313dc

…-- still not fully working

add comments for future edits

a3fd3c1

all tests pass by updating logic

7408bfb

valid-data-crud tests pass

0ae67e5

Merge branch 'master' of https://github.com/nasa/Common-Metadata-Repo…

4d4f93e

…sitory into CMR-11155

Merge branch 'master' of https://github.com/nasa/Common-Metadata-Repo…

83bbe03

…sitory into CMR-11155

temp commit

b001331

remove print statements, update delete index func

c21738d

Merge branch 'master' of https://github.com/nasa/Common-Metadata-Repo…

56b0ff1

…sitory into CMR-11155

Merge branch 'master' of https://github.com/nasa/Common-Metadata-Repo…

78264e7

…sitory into CMR-11155

Merge branch 'master' of https://github.com/nasa/Common-Metadata-Repo…

1cbe5ee

…sitory into CMR-11155

add delete index-set table per test, add required import, update by p…

015fdca

…rovider for tool and index-set

remove provider id from index set

c943e2c

fix int tests to inc their hardcoded concept ids

d20b84b

fix bulk update test

4813c92

Merge branch 'master' of https://github.com/nasa/Common-Metadata-Repo…

07a6e2d

…sitory into CMR-11155

clean up PR

4060d5b

jmaeng72 commented Jun 24, 2026

View reviewed changes

Touchups for PR

ac4ffa5

jmaeng72 marked this pull request as draft June 24, 2026 19:11

jmaeng72 self-assigned this Jun 24, 2026

jmaeng72 changed the title ~~CMR-11155: Version Control Index Set~~ CMR-11155: Version Control Index Set [NOT READY FOR REVIEW] Jun 24, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

CMR-11155: Version Control Index Set [NOT READY FOR REVIEW]#2454

CMR-11155: Version Control Index Set [NOT READY FOR REVIEW]#2454
jmaeng72 wants to merge 18 commits into
masterfrom
CMR-11155

jmaeng72 commented Jun 24, 2026 •

edited

Loading

Uh oh!

codecov-commenter commented Jun 24, 2026 •

edited

Loading

Uh oh!

jmaeng72 Jun 24, 2026

Uh oh!

jmaeng72 Jun 24, 2026

Uh oh!

jmaeng72 Jun 24, 2026

Uh oh!

jmaeng72 Jun 24, 2026

Uh oh!

jmaeng72 Jun 24, 2026

Uh oh!

jmaeng72 Jun 24, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

jmaeng72 commented Jun 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

What is the objective?

What are the changes?

What areas of the application does this impact?

Required Checklist

Additional Checklist

Uh oh!

codecov-commenter commented Jun 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

jmaeng72 Jun 24, 2026

Choose a reason for hiding this comment

Uh oh!

jmaeng72 Jun 24, 2026

Choose a reason for hiding this comment

Uh oh!

jmaeng72 Jun 24, 2026

Choose a reason for hiding this comment

Uh oh!

jmaeng72 Jun 24, 2026

Choose a reason for hiding this comment

Uh oh!

jmaeng72 Jun 24, 2026

Choose a reason for hiding this comment

Uh oh!

jmaeng72 Jun 24, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

jmaeng72 commented Jun 24, 2026 •

edited

Loading

codecov-commenter commented Jun 24, 2026 •

edited

Loading