UWSGI container high memory usage on Kubernetes with 2.53.1

[goat-ssh]

Bug description
We've upgraded v2.50.4 -> 2.53.1. Our usage pattern have not changed and the container has quite some spikes

Expected behavior
Memory usage is stable

Deployment method (select with an X)

• Docker Compose
• Kubernetes
• GoDojo

Environment information

• DefectDojo version 2.53.1
• k8s 1.33 AKS

Screenshots

Additional context (optional)

helm values (chart 1.9.1):

celery:
  beat:
    resources:
      limits:
        cpu: 500m
      requests:
        cpu: 100m
  worker:
    resources:
      limits:
        cpu: 500m
      requests:
        cpu: 100m
createPostgresqlSecret: true
createSecret: true
createValkeySecret: true
django:
  ingress:
    annotations:
      nginx.ingress.kubernetes.io/proxy-body-size: 50M
    ingressClassName: nginx
  nginx:
    resources:
      limits:
        cpu: 200m
      requests:
        cpu: 100m
  uwsgi:
    enableDebug: true
    resources:
      limits:
        cpu: 500m
        memory: 1Gi
      requests:
        cpu: 100m
global:
  postgresql:
    auth:
      password: ddddd
host: dojo.xyz

[valentijnscholten]

Quite a lot has changed between 2.50.x and 2.53.x. Do you have any idea which version introduced it?

The upgrade notes lists the biggest changes:

• https://docs.defectdojo.com/en/open_source/upgrading/2.51/
• https://docs.defectdojo.com/en/open_source/upgrading/2.52/
• https://docs.defectdojo.com/en/open_source/upgrading/2.53/
• and docs: add dedupe batching note to 2.53 upgrade notes #13914
• python 3.12 -> 3.13

Is there anything in there that rings a bell or is typically used a lot in your instance?

Biggest changes in the uwsgi container code in these releases have been about improving performance. But some of them could lead to a little more memory use. The graph however doesn't show "massive" memory use. More like a tradtional saw tooth pattern I know from Java applications. We don't get many memory issue reports and I have never looked into how to debug them.

Is the memory usage causing issues, or is it all recovering by itself and not hurting performance?

[goat-ssh]

We've 1Gi limit set, this should cause the saw tooth pattern. I'll increase it again

[b3ndill]

Just wanted to chime in on this issue. We are seeing a similar pattern on the memory usage of the UWSGI container with a fresh Defect Dojo setup:

• Defect Dojo version: 2.52.3
• Deployment type: GKE via Helm chart
• Helm chart version: 1.8.3
• Database type: External PostgreSQL via Cloud SQL

After the initial setup we noticed that the Django pod kept restarting which was due to the UWSGI container hitting the memory limit and Kubernetes killing it. We raised the memory limit to 2GB but it only prolonged the lifetime to around two days as can be seen on the graph below. After some research into the issue, we suspected that this was related to "unbound" file descriptors on the host systems. We tried setting the django.uwsgi.appSettings.maxFd parameter to 1024 but it didn't change the memory usage trajectory.

We haven't started using Defect Dojo apart from setting up external authentication and logging in.

We're currently running the UWSGI container without any resource limits to see if it balances out somewhere but it's approaching 2.5GB and isn't showing any signs of stopping... 😛

[valentijnscholten]

I don't see this behaviour in my local docker compose based instance. The DD Pro team also doesn't see it in the Pro SAAS instances. But those instances user tcpPorts based health checks which might confirm the HTTP health checks are causing the memory issues as mentioned by others on Slack: https://owasp.slack.com/archives/C2P5BA8MN/p1763049449585229

[b3ndill]

@valentijnscholten that might be the cause of the memory "leak".

I tested this by converting the uwsgi container liveness probe to a TCP probe as well as removing the nginx container readiness probe.
The memory utilization is now stable at around 500MiB (with no usage), before it would grow by 20-30MiB per hour under the same circumstances.

[goat-ssh]

I can also confirm the observations from @b3ndill .
What should be the proper solution? Change in helm templates to allow more customizations on the liveness probe settings?

[schewara]

Same here, there is definitely some memory leak somewhere.

I was cleaning up old closed findings over the last 2 days and after a while notified by the OOM killer about it being killed.

The memory graph tells a clear story that something is seriously off and increasing it just delays the inevitable memory exhaustion.

On a side node, I would really ❤️ to see the bulk delete API calls coming sometime soon, as I was unable to get more then 2 DELETE calls per second in and cleaning up 150k of old closed findings was quite painful. see also #13900 #8090

[valentijnscholten]

Since the health check is basically a call to the login page, can some of you describe more of you authentication setup? Is it local users? SAML? OIDC? Do you have the login banner enabled/configured?

[AndreVirtimo]

Since the health check is basically a call to the login page, can some of you describe more of you authentication setup? Is it local users? SAML? OIDC? Do you have the login banner enabled/configured?

We have OIDC configured. No configuration changed for the login banner.

[valentijnscholten]

#14117 has two fixes

• bleach was used in a way where for every request it was adding things to the allow list which made the allowlist grow over time (confirmed small memory leak)
• the git commit hash is being checked and displayed on every page using a subprocess that literally executes git. there were some log lines intermittently about processes and files not being closed properly. some memory analysis showed the top 3 allocations where all from the git python module. this PR stops executing git and just looks at the .git folder to get the commit hash (possible memory leak)
  Although I don't think these two issues solve the reported "ever increasing memory usage", it will definitely help.

I also looked at including some memory diagnostics functionality using tracemalloc. But this slows down the application a lot so it's not suited to enable or even bundle in the production images. If someone is able to do some analysis on the memory usage using tracemalloc or something like memray that could help. It would also be interesting to see if these two fixes, especially the git commit checker, has any noticable positive effect.

[menksshure]

Hello. I have also faced the same issue on 2.52.1. Is there any resolution on this issue? I see @valentijnscholten made some changes but wasn't sure if that fixes memory issues. Is there a higher version to upgrade to that might resolve this issue?

[valentijnscholten]

2.54.2 has the two fixes mentioned above. I am curious if they help.

[b3ndill]

Preliminary results with 2.54.3 are looking promising. I updated the Helm chart and reverted my health probe changes and it looks more stable on memory usage:

For context, the drop is where I updated (around 16:30).

Just to reiterate, I'm running this in GKE, external database, OIDC configured but no usage (yet) so the system is sitting idle.