Add system-OpenSSL build variant for the bundled Python 3.10.13 Linux package

[nickschuetz]

Summary

Adds a parallel Linux build variant for the bundled Python 3.10.13 package that links dynamically against the building distro's system OpenSSL 3.x rather than the bundled OpenSSL-1.1.1t-rev1-linux static dependency.

Motivation

The bundled OpenSSL-1.1.1t is end-of-life since 2023-09-11. For downstream distribution packagers (Fedora, Debian, Arch, Alpine) targeting their distribution's review process, the EOL crypto in the bundled Python is a meaningful inclusion blocker. This variant gives those packagers a clean path: configure Python to find OpenSSL 3.x in the system at build time, link dynamically, let the OS handle CVE updates.

What this PR does

The existing linux_x64/ variant is unchanged. A new Linux-system-openssl sub-variant in build_config.json triggers an env-var (PYTHON_USE_SYSTEM_OPENSSL=1) that the modified docker_build_linux.sh reads. In that mode:

• --with-openssl=/usr --with-openssl-rpath=auto replaces --with-openssl=<bundled path>
• The Ubuntu base image has libssl-dev available (added to the Dockerfile)
• _ssl.cpython-310-x86_64-linux-gnu.so ends up dynamically linking against the OS's libssl.so.3 + libcrypto.so.3

Output package: python-3.10.13-rev1-linux-system-openssl. Drop-in replacement for downstream consumers whose target OS provides OpenSSL 3.x; Python ABI is 3.10.13 stable.

Compatibility

The new variant requires the target system to have OpenSSL 3.x in the base OS at runtime (libssl.so.3 + libcrypto.so.3). Target distros with OpenSSL 3.x in base: Ubuntu 22.04 LTS and newer, Fedora 39 and newer, RHEL 9 and newer, Debian 12 and newer. Existing consumers on older OSes (Ubuntu 20.04 LTS, RHEL 8, Debian 11) ship OpenSSL 1.1.x and would not work with the system-openssl variant; they keep using the existing linux_x64/ variant unchanged. Offline-install scenarios on older targets are unaffected because the default bundled variant is fully self-contained.

Validation

Spike validation completed locally before this PR. Net: 78 individual checks across the integration tier suite, 0 failures.

Build: PASS. Full pull_and_build_from_git.py package-system/python --platform-name Linux-system-openssl --clean ran clean inside an Ubuntu 22.04 container. Configure detected system OpenSSL (checking for openssl/ssl.h in /usr... yes), and the Building Python against system OpenSSL (dynamic link) env-var-driven branch executed correctly. Final PYTHON WAS BUILT FROM SOURCE marker reached.

Artifact:

• ldd _ssl.cpython-310-x86_64-linux-gnu.so shows libssl.so.3 => /lib/x86_64-linux-gnu/libssl.so.3 + libcrypto.so.3 => /lib/x86_64-linux-gnu/libcrypto.so.3 (Ubuntu container). On a Fedora 44 host, resolves to /lib64/libssl.so.3 + /lib64/libcrypto.so.3 (OpenSSL 3.x stable ABI holds across the build-host / runtime-host boundary).
• import ssl; print(ssl.OPENSSL_VERSION) returns OpenSSL 3.0.2 15 Mar 2022 (Ubuntu 22.04 container) and OpenSSL 3.5.5 27 Jan 2026 (Fedora 44 host).
• Non-OpenSSL Python C extensions still carry Ubuntu SONAMEs (unchanged from the existing bundled Python; this PR doesn't regress that).

Engine smoke (downstream Fedora packaging effort):

• Swapped the rebuilt Python into the engine install path, wiped ~/.o3de/Python/venv to force re-bootstrap via get_python.sh, ran the integration tier suite.
• Tier 1 (RPM integrity): PASS
• Tier 2 (install integrity + system-swap auto-Requires): PASS
• Tier 3 (cold-cache first-run setup via get_python.sh, manifest registration via o3de-cli register --this-engine): PASS. This is the critical OpenSSL-path canary: pip install runs HTTPS through the rebuilt _ssl module; manifest setup and registry work correctly.
• Tier 4 (engine binary smoke, Project Manager Python init): PASS
• Tier 5 (project create via Python-driven o3de create-project + cmake configure): PASS
• Tier 9 (MultiplayerSample warm-cache build + bake + GameLauncher load): 13/13 PASS
• Tier 10 (NewspaperDeliveryGame warm-cache build + bake + launcher loaded CharacterSample): 7/7 PASS

The engine works identically with the rebuilt Python. AssetProcessor's embedded Python interpreter executes builders cleanly. PySide2 (separate bundled package, links against libpython3.10.so.1.0) loads without ABI issues; Python's stable patch-level ABI held.

Open questions for sig-build

1. Default Dockerfile base image: Ubuntu 20.04 ships OpenSSL 1.1.1f. For the system-openssl variant the build needs Ubuntu 22.04 (OpenSSL 3.0.2) or 24.04 (OpenSSL 3.0.13). Two options: (a) parameterize the base image via build_config.json so each variant picks its own (preserves existing artifact-byte-equivalence for the bundled-OpenSSL build), or (b) bump the default base to 22.04 for both variants (simpler maintenance). Happy to refactor either way.
2. Naming: python-3.10.13-rev1-linux-system-openssl is verbose. Alternatives: python-3.10.13-rev1-linux-dynamic-ssl, python-3.10.13-system-ssl-rev1-linux, etc. Whatever convention sig-build prefers.
3. Should the dynamic variant eventually become the default? Long-term, dynamic-link-against-system-OpenSSL reduces O3DE maintainers' CVE-response burden (system updates flow through apt-get update / dnf update instead of requiring a Python rebuild every time OpenSSL ships a CVE). The bundled variant remains useful for environments where system OpenSSL can't be assumed (older containers, embedded targets, build-isolated CI, air-gapped offline-install on older OSes). Suggest both variants coexist long-term.

Notes

• aarch64 variant follows the same shape; happy to add in a follow-up once x86_64 is validated and the naming / base-image questions are settled.
• Filed in coordination with the downstream Fedora packaging effort at https://github.com/nickschuetz/o3de-rpm. See that repo's upstream-drafts/pr-python-system-openssl.md for the full motivation and validation evidence.

[jhanca-robotecai]

We might need to go deep in the history, but the question first. Any idea why OpenSSL was bundled into O3DE?
I found this PR while working on... bumping bundled OpenSSL to 3.x.y. The question now can be rephrased: do we want to bundle OpenSSL in O3DE or should we always go for the system?

I started working on bundling new OpenSSL for a simple reason. OpenSSL 1.1.1 is burned into AzNetworking, which links statically to AzFramework. It could be easily split (I have a PoC with AzNetworking being shipped as a shared object), but it does not help if we build monolithic. Since we need OpenSSL 3.x.y in our Gems, we get segfaults due to colliding OpenSSL linkage...

[jhanca-robotecai]

Up: #387

[nickschuetz]

Thanks for grabbing this, Jan, and #387 is the right call. Bumping the bundle to 3.6.3 kills the EOL 1.1.1 problem for everyone, which is the bigger win of the two.

Why it was bundled in the first place: no idea for certain, I never found a decision record. My guess is it just rode the general "bundle everything so builds are hermetic and work offline everywhere" convention rather than anyone choosing OpenSSL specifically. Whatever the original reason, your AzNetworking catch is what pins it in place now it seems. It highlights that ripping the bundle out might not be as simple as it looks.

On bundle vs. system: I'd keep bundled-by-default and just offer a system-link variant next to it, which is all #376 is. So the two PRs don't conflict: #387 makes the default bundle fine again by getting it off dead crypto, and #376 lets distro builds (Fedora and friends) link the OS OpenSSL so CVEs come through dnf/apt instead of a rebuild. Even a 3.6.3 bundle is still a bundled lib, which Fedora frowns on regardless of version, so #376 stays useful to us after yours lands.

Happy to rebase #376 on top of #387 and re-validate against the 3.6.3 bundle once it settles, or fold it into your sweep if that's less review churn. Thanks for driving this.

[nickschuetz]

Since the question of what actually happens when 1.1.1 and 3.x end up in the same process came up, I went and measured it on our Fedora packages rather than guessing. Posting in case it's useful.

What's in there today: libAzFramework.so statically links OpenSSL 1.1.1t and exports about 1600 of its symbols from the dynamic table. The bundled Qt5 doesn't link OpenSSL at all, it resolves it at runtime, so on a Fedora host it picks up the system OpenSSL 3.5.7 (QSslSocket::supportsSsl() is true and sslLibraryVersionNumber comes back 0x30500070). So 1.1.1t and 3.5.7 are already sitting in the same process today, before anything I'm proposing here.

The good news is it doesn't actually cross-bind. I traced it with LD_DEBUG=bindings and zero of the 26497 recorded bindings resolved from libssl/libcrypto 3.5.7 into AzFramework's copy. The reason is symbol versioning: OpenSSL 3.x exports versioned symbols like EVP_DigestInit@@OPENSSL_3.0.0 and libssl's references carry those version tags, while the statically linked copy inside AzFramework is unversioned, so a versioned reference can't bind to it. The one gap is unversioned lookups, a dlsym(RTLD_DEFAULT, "EVP_DigestInit") does land on AzFramework's 1.1.1t.

So the "as long as they all match" instinct is right, and it turns out symbol versioning is what quietly enforces it. Worth knowing that the mixed state isn't automatically fatal, but also that nothing is checking it for you.

On the "we may have to ship a new python" point from the #387 review, that's confirmed from this side: the stock python-3.10.13-rev2-linux _ssl module has OpenSSL 1.1.1t statically inside it with no dynamic libssl, so it'll still be carrying 1.1.1 after #387 lands until it gets rebuilt. That's the gap this PR is aimed at, either by rebuilding against the new 3.6.3 package or by linking the system one.

Offer from my earlier comment still stands, happy to rebase this onto #387 and re-validate once that merges.

[jhanca-robotecai]

I have a Gem that depends on OpenSSL 3.x and I was getting a segfault every time I added the Gem to the project due to the collisions between the embedded (in AzFramework via AzNetworking) OpenSSL 1.1.1 and OpenSSL 3.x that was required by my Gem. In particular, layout/struct mismatches between the two versions' internal EVP_PKEY_CTX representation caused the segfault.

[nickschuetz]

I dug into this properly and reproduced it, and I have to correct something I said earlier in the thread: I claimed the two OpenSSLs don't cross-bind and that symbol versioning keeps them apart. That's wrong. It does cross-bind, and versioning does not save you. Here's what actually happens.

Repro: a small "gem" .so linked against system OpenSSL 3.x doing an EC keygen. Standalone it works. Load AzFramework into the process first (it exports its statically-linked 1.1.1t symbols into global scope, unhidden) and the same keygen fails with pkey_ec_keygen: no parameters set. I got a clean error rather than your segfault, but it's the same collision landing on a state check instead of a pointer deref.

The mechanism is nastier than I first thought. It's not that libcrypto's internals get hijacked, they don't. It's that the gem's own calls get split across the two versions. glibc will happily bind a versioned reference (EVP_PKEY_keygen@OPENSSL_3.0.0) to an unversioned definition if that definition is first in scope, and AzFramework's 1.1.1t exports are exactly that. So in my repro EVP_PKEY_CTX_new_id, EVP_PKEY_keygen_init and EVP_PKEY_keygen all bound into AzFramework's 1.1.1t, but EVP_PKEY_CTX_set_ec_paramgen_curve_nid fell through to real 3.x, because 1.1.1 never exported that one (it was a macro back then, became a real symbol in 3.0). So the curve gets set on the ctx by 3.x, then keygen reads the ctx as 1.1.1t and doesn't see it. Half your call chain is in one OpenSSL, half in the other, on the same object. That's your struct mismatch, and depending on which call lands where you get either a clean-ish error like mine or a deref into a wrong-layout struct like your segfault.

So versioning is not the safety net I said it was, an unversioned export shadows a versioned one. Which also means #387 alone won't fully fix it: getting everyone on 3.x removes the version mismatch, but AzFramework still exports a whole second copy of OpenSSL globally, so a gem's libcrypto and AzFramework's still fight over the common symbols. The fix is to stop exporting them, hidden visibility on AzNetworking's OpenSSL. I confirmed the direction: loading AzFramework with local scope (RTLD_LOCAL) instead of global makes the exact same keygen pass, no other change. Same unhidden-export problem you flagged for Python's _ssl/_hashlib.

If you can share a minimal repro of your gem crash I'll run it against our Fedora build and get you the precise per-symbol trace for your case.

[jhanca-robotecai]

So versioning is not the safety net I said it was, an unversioned export shadows a versioned one. Which also means #387 alone won't fully fix it: getting everyone on 3.x removes the version mismatch, but AzFramework still exports a whole second copy of OpenSSL globally, so a gem's libcrypto and AzFramework's still fight over the common symbols. The fix is to stop exporting them, hidden visibility on AzNetworking's OpenSSL. I confirmed the direction: loading AzFramework with local scope (RTLD_LOCAL) instead of global makes the exact same keygen pass, no other change. Same unhidden-export problem you flagged for Python's _ssl/_hashlib.

I have some work in progress for O3DE, I will publish it when I get some time to work on it. There are two things that should be solved.

There are different applications. For me, #387 is a good step forward. I can easily link my gem against AzNetworking and get the access to OpenSSL 3.x, which is required by the rest of my infrastructure (1.1.1 is deprecated).