Skip to content

perf3: importers should not save findings again#12863

Closed
valentijnscholten wants to merge 18 commits intoDefectDojo:bugfixfrom
valentijnscholten:perf3-reduce-saves
Closed

perf3: importers should not save findings again#12863
valentijnscholten wants to merge 18 commits intoDefectDojo:bugfixfrom
valentijnscholten:perf3-reduce-saves

Conversation

@valentijnscholten
Copy link
Copy Markdown
Member

@valentijnscholten valentijnscholten commented Jul 27, 2025

There'll be a sequence of import performance related PRs. They are separated into small PRs to simplify review and troubleshooting. As they all use test_importers_performance, they'll build on eachother. So they'll have to be merged by natural ordering.

At the end of the import or reimport process there was an call to finding.save() mainly to trigger post processing such as dedupe, pushing to JIRA and product grading.
But at that point all finding data was already saved in earlier steps of the import process. So instead of doing a save, we call the post processing directly. This saves a database UPDATE statement which is expensive and can trigger further processing/queries such as audit logging. Some may argue the code becomes less clear or harder to understand. Since this is the primary process of Defect Dojo and the main performance concern it is an acceptable downside.

@valentijnscholten valentijnscholten added this to the 2.49.0 milestone Jul 27, 2025
@valentijnscholten
Copy link
Copy Markdown
Member Author

This looked like a promising optimization possibility. However hash_code generation logic is depending on related objects like endpoints and vulnerability ids being present. This means there's not really a way around have two calls to save.
The PR here tried to opimize it to only set UPDATE the hash_code field, but still lots of tests are failing.
So for now we have to leave things as is as it seems to become too complicated.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant