Skip to content

[qualys_vmdr] request 4Gi of memory for agentless deployments - #20579

Open
igcybers wants to merge 1 commit into
elastic:mainfrom
cyber-smart:qualys_vmdr-agentless-memory
Open

[qualys_vmdr] request 4Gi of memory for agentless deployments#20579
igcybers wants to merge 1 commit into
elastic:mainfrom
cyber-smart:qualys_vmdr-agentless-memory

Conversation

@igcybers

@igcybers igcybers commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Proposed commit message:

qualys_vmdr: request 4Gi of memory for agentless deployments

The asset_host_detection input previously inherited the default agentless
memory allocation, which is not sufficient for the default batch size of 1000
hosts.

At publish time the CEL program holds the full host batch, the knowledge base
map accumulated across every paginated request for that batch, and the
flattened event array in memory simultaneously. Building the events copies the
entire host record once per detection, so peak usage scales with hosts x
detections-per-host rather than with the batch size alone.

Request 4Gi to match the other vulnerability management and security packages
 (rapid7_insightvm, m365_defender, microsoft_defender_endpoint, ti_crowdstrike).

Why 4Gi

asset_host_detection accumulates several large structures in CEL state at the same time before it can publish. In
[input.yml.hbs] https://github.com/elastic/integrations/blob/main/packages/qualys_vmdr/data_stream/asset_host_detection/agent/stream/input.yml.hbs#L206-L223):

  • state.worklist.AHD_LIST holds the full host batch - truncation_limit is set from batch_size, which defaults to 1000. In practice ~700 batch size is the limit with the current memory setup.
  • state.worklist.KB_MAP grows monotonically across every paginated /api/4.0/fo/knowledge_base/vuln/ request for that batch (host_qid_list_query_limit, default 500 QIDs per request).
  • The events array is then built with AHD_LIST.map(h, h.DETECTION_LIST.DETECTION.map(v, h.with({...}))).flatten().

That last step is the amplifier: h.with({...}) copies the entire host record once per detection and inlines its knowledge base entry, so peak usage scales with hosts × detections-per-host, not with batch_size alone and it happens while both source structures are still live.

This matches the four packages already owned by security-service-integrations that request 4Gi: rapid7_insightvm (same vulnerability-management and CDR workload shape),m365_defender, microsoft_defender_endpoint and ti_crowdstrike.

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have added an entry to my package's changelog.yml file.

How to test this PR locally

This change only adds the agentless resource request to the package manifest; there is no ingest, mapping or dashboard change, so the existing tests are unaffected.

To validate the manifest against the spec:

cd packages/qualys_vmdr
elastic-package check

@igcybers
igcybers requested review from a team as code owners August 6, 2026 16:22
@igcybers
igcybers force-pushed the qualys_vmdr-agentless-memory branch from cf2a6f4 to cc07a64 Compare August 6, 2026 16:22
The asset_host_detection input previously inherited the default agentless
memory allocation, which is not sufficient for the default batch size of 1000
hosts.

At publish time the CEL program holds the full host batch, the knowledge base
map accumulated across every paginated request for that batch, and the
flattened event array in memory simultaneously. Building the events copies the
entire host record once per detection, so peak usage scales with hosts x
detections-per-host rather than with the batch size alone.

Request 4Gi to match the other vulnerability management and security packages
owned by this team (rapid7_insightvm, m365_defender,
microsoft_defender_endpoint, ti_crowdstrike).
@igcybers
igcybers force-pushed the qualys_vmdr-agentless-memory branch from cc07a64 to 56cef1c Compare August 6, 2026 16:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant