OpenVas re-import does not recognize same result

[LinoSino]

Bug Description

When importing or re-importing the same OpenVAS scan (or scans containing the same vulnerability), some findings are not recognized as already existing in the same test.

When I import the attached openvas.csv file for the first time, it correctly creates a finding. However, when I re-import the same file, it creates a duplicate finding instead of recognizing that it already exists.

The re-import was tested both manually via the GUI and through an API call.

Steps to reproduce

1. Create a new test and import the attached CSV file.
2. Re-import the same CSV file into the same test.
3. A second, duplicate finding is created.

Expected behaviour
The re-import should recognize existing findings based on the generated hash. This works for most findings, but not for all. An example of a finding that fails to be recognized is included in the attached CSV file.

Deployment method (select with an X)

• Docker Compose
• Kubernetes
• GoDojo

Environment information

• Operating System: [Ubuntu 24.04.2]
• Docker Compose version v2.35.1
• DefectDojo Community v. 2.45.3

Sample scan files
openvas.csv

Screenshots
Uploaded scan twice

Additional context
These are the hash_codes it generated:
second import:
Hash_code changed from "None" to "b1dccf748cfd23192d684069f99a237f0ed9201cfda912c29c9ddb89d5678eab"
first import:
Hash_code changed from "None" to "b1dccf748cfd23192d684069f99a237f0ed9201cfda912c29c9ddb89d5678eab"

[valentijnscholten]

Could this be the scenario from #3958 ?

[LinoSino]

I'm not entirely sure, as in my example I only have a single finding that I reimport so there sould not be an issue with just checking the first finding. In the file I uploaded, there's just one example finding where this occurs and there exist more but not every OpenVas finding behaves this way just some. Even with deduplication enabled (not just the reimport check), it's not detected as a duplicate.

As a further test, I modified the OpenVas CSV parser to use only the endpoint and Vulnerability-Name for checking, just to rule out anything strange with the description, but that didn’t work either.

[jostaub]

Hi,
I also encountered the same issue and believe I’ve found a potential workaround. It appears that the OpenVAS CSV parser falls back to the legacy reimport algorithm. A comment in the code (and a log warning) states that this part is pretty flawed and dos not match legacy algorithm for deduplication and suggests adding a deduplication config in case of problems.

My Workaround:
I configured the deduplication approach for the parser in local_settings.py like this:

HASHCODE_FIELDS_PER_SCANNER["OpenVAS Parser"] = ["title", "severity", "vulnerability_ids", "references"]
HASHCODE_ALLOWS_NULL_CWE["OpenVAS Parser"] = True
DEDUPLICATION_ALGORITHM_PER_PARSER["OpenVAS Parser"] = DEDUPE_ALGO_HASH_CODE

HASHCODE_ALLOWED_FIELDS.append("references")

Using this configuration, the problematic finding is not marked as a duplicate during a re-import (and not readded)—at least based on my testing. I used the "references" field instead of "description", since OpenVAS often includes the host IP in the description, which prevents the same findings on different hosts to match. However, I'm not sure if using references is ideal, as they can contain the NVT OID. I'm unsure whether the OID is stable and unique enough, especially in cases where vulnerability IDs aren't set.

@LinoSino I'm curious: What did you modify in the OpenVAS CSV parser? As I understand it, the "deduplication" part of this parser only prevents duplicated findings that are present in the currently processed report.

[jostaub]

The problems seems related to: #1163 and #2687

The method match_new_finding_to_existing_finding in default_reimporter.py uses a non-titlecased version of the title for searching when using the legacy deduplication algorithm. The title is only titlecased after the unsaved finding is saved.

Therefore, this issue may affect other parsers that use the legacy deduplication algorithm during reimport.

[valentijnscholten]

@jostaub Thanks, this is actually a nice find! Long ago we made sure everything around hash codes was case-insensitive. But this case looks like an oversight. I have created #12487 and in my testing it fixes the problem with the provided Openvas CSV report above.

Nevertheless it's always better to have a hash code field configuration present. We're curious about what you would suggest as a good field configuration based on your real-world testing.

[jostaub]

(sorry for the long response)

First, I want to correct a couple of mistakes I made in my earlier comment:

1. I stated that the IP address of the host is included in the OpenVAS description. That’s incorrect. The IP address is actually added during the parsing process by the parser, not by OpenVAS itself.
2. I also mentioned that the references necessarily contain the OID. That’s not always the case.

---

Goals for the New Hash Strategy

When considering how to compute the hash for deduplication, I believe the following two goals should be prioritized:

1. Support for DEDUPLICATION_ALGO_ENDPOINT_FIELDS
2. Avoid marking all findings with the same title and severity as duplicates

---

1. Support for DEDUPLICATION_ALGO_ENDPOINT_FIELDS

If DefectDojo supports this feature, the parser should align with it—especially if we’re updating it anyway.
To comply with DEDUPLICATION_ALGO_ENDPOINT_FIELDS, we must not hash any field that includes the host’s IP address or hostname.

---

2. Same Title and Severity ≠ Always a Duplicate

OpenVAS may report the same vulnerability multiple times on the same host. These are not necessarily duplicates.

Example:
If a CVE affects Java, OpenVAS might find multiple vulnerable Java binaries installed across different paths or services on the same host. Each instance potentially requires separate patching. Automatically marking them as duplicates could cause patching teams to miss one of them.

---

Recommended Fields for Hashing

Based on these goals and some testing in my environment, I propose the following fields for inclusion in the hash:

• Title or NVT OID
• Vulnerability IDs
• Severity
• Specific result
• (Port – optional)

---

Title or NVT OID

In my environment, each title has a 1:1 match with an NVT OID, so either can be used as a stable identifier. This makes it a good base for the hash.

---

Vulnerability IDs

Helpful if OpenVAS/Greenbone accidentally identical titles to different findings (can be really generic some thimes). In thoses cases, . the OID could help to distinguish them, as it must be unique (however as later explained not nessesarily present or consistent between parsers, including this field).

---

Severity

OpenVAS/Greenbone allows overriding CVSS scores per finding. The same vulnerability could have different severities depending on the context (e.g., network segment). Including severity ensures a high-severity finding isn’t mistakenly deduplicated against a lower-severity one.
It also ensures new findings are created if a CVSS score update changes the severity.

---

Specific Result

This field shows how the vulnerability was detected on the host. It’s especially useful when multiple instances of the same issue exist, such as different Java binary paths.
However, this field can be problematic in network-based tests—e.g., if the response includes a hostname, which would conflict with deduplication goals. While I haven't seen this in my environment, it’s possible depending on the vulnerability.

---

(Port – Optional)

I’m unsure whether the port is already included in the default configuration of DEDUPLICATION_ALGO_ENDPOINT_FIELDS.
Some tests (like SSL cipher checks) may produce identical results across different ports, so including the port might help.

---

A More General Note on Parser Inconsistency

Using specific result and NVT OID can be tricky due to differences between the CSV and XML parsers:

• In the CSV parser, specific result appears under finding references.
• In the XML parser, it’s missing.
• The NVT OID is not parsed in the CSV parser but is included in the XML parser as the vulnerability ID (unlike CVEs in the CSV parser).

I may open a feature request to improve consistency between the CSV and XML parsers—this isn’t the only inconsistency I’ve come across.

[valentijnscholten]

Thanks for the elaborate response. Please note that hash code fields can only contain fields of the finding model, are Specific Result and Port stored as field value in the finding model?

[jostaub]

The specific result field is included as finding.references in the CSV parser but not in the XML parser (my point to inconsistent behavior).

The port is something i have not fully researched. It is not present in the finding object but in the endpoint object. However i am currently unsure if this is per default included in DEDUPLICATION_ALGO_ENDPOINT_FIELDS (under path) or supported there at all. If not this could be addressed by using DEDUPE_ALGO_UNIQUE_ID_FROM_TOOL_OR_HASH_CODE for deduplication. This however would require the 2 parsers to be modified in order to support it.

[valentijnscholten]

The port is by default not used in the endpoints comparison function, but it could be added. But that would be a global setting affecting the Defect Dojo instance as whole.

[jostaub]

I’ve refactored and updated the parser – see the current WIP branch.

TL;DR:

• Updated the CSV implementation to behave more like the XML parser.
• Increased behavior consistency between the CSV and XML parsers.
• Added a config option to set severity based on CVSSv3.
  (Note: Currently, the maximum severity is "High" even when CVSSv3 is used. This appears to be a limitation with OpenVAS.)
• Introduced deduplication using unique_id_from_tool to OpenVAS parser.
• Combined findings where the only differences are in fields that can’t be reliably hashed due to inconsistent values between scanns e.g timestamps.

---

Remaining Questions (and some cleanup) before opening a PR:

1. Is it acceptable to introduce a config option specifically for configuring a parser?
2. Are there any guidelines on how parsers should be configurable?
3. I find some of the test files for the OpenVAS CSV and XML parsers a bit confusing.
   For example, some elements in the XML tests (like the nvt tag) seem inconsistent with what would appear in a regular OpenVAS export.
   Shouldn’t the nvt.name tag always resemble the name tag? How where these files generated?

[valentijnscholten]

1. Is it acceptable to introduce a config option specifically for configuring a parser?

It would be the first parser to have that. But sometimes we run into situation where you may want a feature toggle like this for a parser. My proposal would be to have a generic key/value mechanism to pass flags during import/reimport. The challenge there would be the UI as it would need to know which flags are available. In this particular case I think we should just switch to the new severity logic unless severity is/becomes part of the hash_code calculation.

2. Are there any guidelines on how parsers should be configurable?

No because currently they aren't.

4. I find some of the test files for the OpenVAS CSV and XML parsers a bit confusing.> For example, some elements in the XML tests (like the nvt tag) seem inconsistent with what would appear in a regular OpenVAS export.
   Shouldn’t the nvt.name tag always resemble the name tag? How where these files generated?

Usually they are supplied by the person who added the parser in the past. This could someone from DD Inc, but most parser were implemented by the community. If you have newer/recent samples you're welcome to add them. Often we keep old samples as well to ensure backwards compatibility.

[jostaub]

@valentijnscholten this might be a stupid question but how do i run linting (formatting) checks without using the pre commit hook? I can remember i used a simple check last time but i cant find the command again. (I use ruff auto format per default but i am not sure it works correctly)