Increase Loading Speed of COSMOS Landing Page

[code-geek]

Description

Right now since we load all the data all at once, the landing page has become very slow (over 2 seconds to load every single time).

The goal of this issue is to make the landing page faster.

Implementation Considerations

• Django ORM select_related etc can be implemented to fetch data more judiciously
• Lazy load data on the homepage with infinite scroll that calls API (pagination on the server side instead of pagination on the client side)
• Caching (redis/memcached)
• Database indexes
• How does this affect Universal Search

Deliverable

• A faster landing page

Dependencies

No response

[Kirandawadi]

This task is currently on hold due to its complexity. I will focus on other tasks before returning to this one.

[CarsonDavis]

I am hopefully working on a benchmarking suite for some of the backend processes, specifically with pattern application times. Hopefully we will be able to tie these two tasks together once that work is finished.

This task will be taken up in 25.2.

[saifrk]

Collection list page load time optimization report:

Collection List Page Load Time Optimization Report.pdf

[CarsonDavis]

So the original page load took 90 seconds. After making the count update, we've reduced this to around 8 seconds.

Essentially instead of calculating an expensive distinct() of the delta and curated urls, we hard coded the value, which reduced the number of database operations.

.annotate(
    num_delta_urls=models.Value(100, output_field=models.IntegerField()),
    num_curated_urls=models.Value(50, output_field=models.IntegerField()),
)

However, long term, we still want to have these counts. Thankfully, it is not necessary to query the database to get the counts. Instead we can create two fields within the Collection model, num_delta_urls and num_curated_urls.

These are then populated at key points during the Collection lifecycle. First, when we make the migration we will populate all of them on a one time basis.

Within the django migration file itself, we will run an update command for these to fields.

Then, we will update these values for future collections at strategic times.

• num delta
  • this is updated at the promotion from dump to delta
  • it is updated again at promotion from delta to curated (set to 0)
• num curated: this is updated at the promotion from delta to curated

Also double check the collection list view file to ensure that it is updated to use these new fields.

Then, check the django admin and make sure it is used where ever necessary there as well.