perf3: importers should not save findings again#12863
Closed
valentijnscholten wants to merge 18 commits intoDefectDojo:bugfixfrom
Closed
perf3: importers should not save findings again#12863valentijnscholten wants to merge 18 commits intoDefectDojo:bugfixfrom
valentijnscholten wants to merge 18 commits intoDefectDojo:bugfixfrom
Conversation
Member
Author
|
This looked like a promising optimization possibility. However hash_code generation logic is depending on related objects like endpoints and vulnerability ids being present. This means there's not really a way around have two calls to save. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
There'll be a sequence of import performance related PRs. They are separated into small PRs to simplify review and troubleshooting. As they all use test_importers_performance, they'll build on eachother. So they'll have to be merged by natural ordering.
At the end of the import or reimport process there was an call to
finding.save()mainly to trigger post processing such as dedupe, pushing to JIRA and product grading.But at that point all finding data was already saved in earlier steps of the import process. So instead of doing a save, we call the post processing directly. This saves a database UPDATE statement which is expensive and can trigger further processing/queries such as audit logging. Some may argue the code becomes less clear or harder to understand. Since this is the primary process of Defect Dojo and the main performance concern it is an acceptable downside.