CMR-11155: Version Control Index Set - Part 1

[jmaeng72]

Overview

What is the objective?

To allow for disaster recovery and keeping track of historical changes of index-set.

What are the changes?

This is PART 1 of 2 PRs to reduce PR size. This PR mainly introduces helper funcs that will be needed for the major changes in PART 2.

• Added helper funcs for new index-set db table
• Added helper funcs for index-set sys tests
• Added funcs to allow for proper reset-fixture updates for sys tests

What areas of the application does this impact?

Indexer

Required Checklist

• New and existing unit and int tests pass locally and remotely
• clj-kondo has been run locally and all errors in changed files are corrected
• I have commented my code, particularly in hard-to-understand areas
• I have made changes to the documentation (if necessary)
• My changes generate no new warnings

Additional Checklist

• I have removed unnecessary/dead code and imports in files I have changed
• I have cleaned up integration tests by doing one or more of the following:
  • migrated any are2 tests to are3 in files I have changed
  • de-duped, consolidated, removed dead int tests
  • transformed applicable int tests into unit tests
  • reduced number of system state resets by updating fixtures. Ex) (use-fixtures :each (ingest/reset-fixture {})) to be :once instead of :each

[jmaeng72]

had to add this because my local had a wrong version of AWS_ARN and had to be replaced. This is helpful to any other dev with local db issues. This will not have adverse affects since the logic still remains the same

[eereiter]

We should be running migrate db when someone changes database structures, that way all of our databases will be up to date. I forget to do this sometimes. Or did you migrate and it didn't work?

[codecov-commenter]

Codecov Report

❌ Patch coverage is 23.00885% with 87 lines in your changes missing coverage. Please review.
✅ Project coverage is 58.13%. Comparing base (0bcadbd) to head (e108558).

Files with missing lines	Patch %	Lines
...app/src/cmr/indexer/services/index_set_service.clj	9.30%	39 Missing ⚠️
...-test/src/cmr/system_int_test/utils/index_util.clj	23.07%	10 Missing ⚠️
...-test/src/cmr/system_int_test/utils/url_helper.clj	35.71%	9 Missing ⚠️
.../migrations/092_update_cmr_subscriptions_table.clj	0.00%	8 Missing ⚠️
...cmr/metadata_db/data/oracle/concepts/index_set.clj	30.00%	7 Missing ⚠️
metadata-db-app/src/config/mdb_migrate_helper.clj	14.28%	6 Missing ⚠️
...src/cmr/system_int_test/utils/metadata_db_util.clj	16.66%	5 Missing ⚠️
...a-db-app/src/cmr/metadata_db/services/messages.clj	33.33%	2 Missing ⚠️
...c/cmr/metadata_db/services/concept_validations.clj	88.88%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #2455      +/-   ##
==========================================
- Coverage   58.20%   58.13%   -0.07%     
==========================================
  Files        1069     1070       +1     
  Lines       74278    74385     +107     
  Branches     2166     2164       -2     
==========================================
+ Hits        43233    43247      +14     
- Misses      29028    29116      +88     
- Partials     2017     2022       +5     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[jmaeng72]

to keep consistent test execution. One of the unit tests kept failing the first time and succeeding the second time, which made me realize the caches were not in clean states per test

[jmaeng72]

all the following changes are additions and should not affect CMR behavior at all when merged into master

[eereiter]

I first glance I am wondering why you chose the path to "create" a new concept, instead of just creating a database table with get/set functions for it for the index set - which I thought we already had - maybe I am remembering the redis cache.

[jmaeng72]

In the beginning, it was easier to treat index-set like a concept and borrow the metadata-db funcs to keep revisions, delete old revs, updates/deletes, etc. I wanted to keep all database interactions under the same pattern. Index-set table did not exist prior to this change.