Automatically Set Needs to Be Reindexed Based on Time Since Prod or ReIndexed

[CarsonDavis]

Description

After a collection is indexed for the first time, we want to then wait two months and then mark it as needs to be reindexed. After it is successfully reindexed, we then wait again for two months, and mark it again as need to be indexed.

This time should be easily configurable if we change our minds down the road.

[Kirandawadi]

Draft

Add configuration setting

• Add reindexing interval setting to make it easily configurable:
  COLLECTION_REINDEX_INTERVAL_DAYS = 60

Model Changes

• Add new field last_indexed_date (DateTimeField) to Collection model

Signal Handler Updates

• Modify existing post_save signal handler to update last_indexed_date when:
  • workflow_status changes to INDEXING_FINISHED_ON_DEV
  • reindexing_status changes to REINDEXING_FINISHED_ON_DEV

New Celery Task

• Create check_collections_reindexing_needed task that:
  • Calculates threshold date (2 months ago)
  • Finds collections last indexed before threshold
  • Performs bulk update to set REINDEXING_NEEDED_ON_DEV status

Celery Beat Configuration

• Add daily scheduled task to check collections
• Task runs automatically without manual intervention

This implementation ensures collections are automatically marked for reindexing two months after their last indexing.

[CarsonDavis]

Is it possible to use the existing ReindexingHistory.created_at fields to identify when the last indexing/reindexing occurred? Long-term, I was hoping that we would be able to keep many different kinds of statistics based on the various dates on the status change fields.

However, when deciding whether this is possible, we need to validate two things:

• the logic for setting these values in the code base is consistently applied and will always result in there being a created_at value for a status
• and for historic statuses, we need to ensure a date of some kind exists, or otherwise be able to set a default value to support their reindexing timeline if one does not

It is very likely that all statuses in the last year have a date, but further back than that might not. Therefor you can write a query to find this information and then set the reindexing parameter accordingly: if no created date, reindex=true. And then the modern process will result in a created at date the next time around.

[Kirandawadi]

Automatic Collection Reindexing

This feature identifies collections that need reindexing based on two scenarios:

---

1. Previously Reindexed Collections

• Targets collections with REINDEXING_INDEXED_ON_PROD status
• Checks their latest ReindexingHistory entry
• Marks for reindexing if:
  • No reindexing history exists, or
  • Last reindex was more than 2 months ago

---

2. First-Time Indexed Collections

• Targets collections with:
  • PROD_PERFECT workflow status and
  • REINDEXING_NOT_NEEDED reindexing status
• Checks their WorkflowHistory for when they reached PROD_PERFECT
• Marks for reindexing if:
  • No workflow history exists, or
  • Reached PROD_PERFECT more than 2 months ago

---

Collections meeting these criteria will be marked as REINDEXING_NEEDED_ON_DEV.