Use system uWSGI packages instead of pip installation#13999
Closed
valentijnscholten wants to merge 8 commits intoDefectDojo:devfrom
Closed
Use system uWSGI packages instead of pip installation#13999valentijnscholten wants to merge 8 commits intoDefectDojo:devfrom
valentijnscholten wants to merge 8 commits intoDefectDojo:devfrom
Conversation
valentijnscholten
force-pushed
the
uwsgi-system-packages
branch
from
c81af86 to
3af77fd
Compare
github-actions
Bot
added
New Migration
valentijnscholten
force-pushed
the
uwsgi-system-packages
branch
from
16f16e3 to
c7c3c56
Compare
Switch from psycopg[c] to psycopg[binary] to use pre-compiled wheels instead of compiling the C extension from source. This significantly speeds up Docker image builds. Changes: - requirements.txt: psycopg[c] -> psycopg[binary] - Remove libpq-dev from all Dockerfiles (build and release stages) The psycopg[binary] package bundles its own libpq, so system libpq-dev is no longer needed for either compilation or runtime. Both options provide the same performance (fast C implementation). Reference: https://www.psycopg.org/psycopg3/docs/basic/install.html
Replace pip-installed uWSGI with OS-provided system packages to eliminate compilation during Docker builds. Changes: - Remove uWSGI from requirements.txt - Debian: Install uwsgi + uwsgi-plugin-python3 from apt - Alpine: Install uwsgi + uwsgi-python3 from apk - Remove build dependencies (gcc, build-essential, python3-dev, libffi-dev) - Remove CPUCOUNT=1 workaround (no longer needed) Benefits: - Eliminates uWSGI compilation (major build time improvement) - Removes need for build toolchain in Docker images - Smaller build stage (no gcc, build-essential, etc.) - Faster Docker image builds Version notes: - Debian Trixie: uwsgi 2.0.28 (vs pip 2.0.31) - Alpine 3.22: uwsgi 2.0.30 (vs pip 2.0.31) System packages are maintained by official distribution teams and receive security backports within their release cycle.
Remove pycurl from requirements.txt and use OS-provided packages instead: - Debian: python3-pycurl (7.45.6) - Alpine: py3-curl (7.45.6) Benefits: - Eliminates pycurl compilation during Docker builds - Removes need for curl-dev/libcurl4-openssl-dev build dependencies - Faster Docker image builds - Consistent pycurl version across Debian and Alpine images pycurl is only needed for Celery SQS broker support. It is not directly imported by DefectDojo code - it's a transitive dependency that Celery uses when configured with an SQS broker (DD_CELERY_BROKER_SCHEME=sqs). The system packages provide the same functionality without requiring compilation toolchains in the Docker build stage.
github-actions
Bot
removed
New Migration
Member
Author
|
Turns out it doesn't work on Alpine because alpine comes with Python 3.12 by default, so the uwsgi plugin apks is compiled against Python 3.12 while we use Python 3.13. In Debian it works (for now) as Trixie uses Python 3.13 out of the box. Two options:
I think option 2 is not really what we want, so this leaves option 1. |
Contributor
|
This pull request has conflicts, please resolve those before we can evaluate the pull request. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Builds on #13993 and #13993 so merge those first.
Description
Replace pip-installed uWSGI with OS-provided system packages to eliminate compilation during Docker builds. This removes the need for build toolchains (gcc, build-essential, python3-dev) in Docker images.
Why Use System Packages?
1. Eliminates Compilation
uWSGI requires compilation from source when installed via pip, which:
2. System Packages Are Well-Maintained
uwsgi+uwsgi-plugin-python3uwsgi+uwsgi-python3These are maintained by official distribution teams and receive security backports.
3. Version Considerations
The downside is that we no longer get Renovate / Dependabot PRs for updates of uwsgi.
The patch-level differences are typically minor bug fixes. Security patches are backported to system packages within their release cycle. I can't remember any time that these small patches broke Defect Dojo. In general it might be better to rely on official binaries instead of compiling our own possibly with suboptimal settings or small changes depending on where/how they are built. But I'm open for discussion/happy to close if we don't get consensus.