Fix several OpenSSL compatibility API gaps

[julek-wolfssl]

• Define WOLFSSL_SIGNER_DER_CERT for OPENSSL_ALL builds so the CA table keeps DER data, fixing X509_STORE_get0_objects() returning an empty stack after X509_STORE_add_cert(). WOLFSSL_NO_SIGNER_DER_CERT can opt back out.
• Accept FFDHE group names in SSL_set1_groups_list() by adding FFDHE entries to kNistCurves, so OpenSSL-style lists mixing curves and FFDHE groups no longer fail as a whole. SSL_set1_curves_list() remains EC-only.
• Add SSL_get_negotiated_group() and SSL_group_to_name(), reporting TLS group registry names (OpenSSL spelling), distinct from both kNistCurves names and wolfSSL_get_curve_name() output.
• Move the leaf domain/IP checks driven by X509_VERIFY_PARAM into ProcessPeerCerts(), alongside the existing wolfSSL_check_domain_name() check, instead of running them in DoVerifyCallback() after the app callback already ran. This lets the application see and react to name-mismatch errors via the verify callback instead of the handshake just failing afterward. As a side effect, SSL_VERIFY_NONE now suppresses these checks, matching the non-OpenSSL path.
• Fix SSL_get_peer_tmp_key() for TLS 1.3: the key share code freed the peer's ephemeral key right after deriving the shared secret regardless of wolfSSL_KeepHandshakeResources(), breaking the call for every TLS 1.3 group. Now the peer key is kept when requested, X25519/X448 keys are also returned, and the kept keys are freed with the other peer keys on teardown.

[Copilot]

Pull request overview

This PR closes several OpenSSL-compatibility gaps in wolfSSL by adjusting default feature macros for OPENSSL_ALL, improving supported-groups parsing/naming behavior, moving hostname/IP checks into the main certificate-processing path, and fixing TLS 1.3 temporary key retention so OpenSSL-style APIs can query negotiated/ephemeral key material post-handshake.

Changes:

• Enable signer DER retention by default under OPENSSL_ALL (with an opt-out macro) to support X509_STORE_get0_objects() enumeration.
• Extend groups-list handling and add OpenSSL-compatible group query/name APIs (SSL_get_negotiated_group(), SSL_group_to_name()), including FFDHE group-name acceptance in *_groups_list().
• Move OpenSSL-driven leaf hostname/IP verification into ProcessPeerCerts() and fix TLS 1.3 peer-tmp-key retention and exposure (including X25519/X448).

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
wolfssl/wolfcrypt/settings.h	Defines WOLFSSL_SIGNER_DER_CERT by default under OPENSSL_ALL unless opted out.
wolfssl/ssl.h	Adds public declarations for negotiated-group query and group-name mapping APIs.
wolfssl/openssl/ssl.h	Maps OpenSSL API names to the new wolfSSL compatibility implementations.
src/ssl.c	Implements wolfSSL_get_negotiated_group() / wolfSSL_group_to_name() and extends peer tmp-key extraction for X25519/X448.
src/tls.c	Adjusts TLS 1.3 key share cleanup to retain peer ephemeral keys when handshake resources are kept.
src/internal.c	Moves OpenSSL X509_VERIFY_PARAM host/IP checks into ProcessPeerCerts() (leaf cert processing) instead of DoVerifyCallback().
tests/api/test_tls.h	Registers new TLS API tests for the added/changed behaviors.
tests/api/test_tls.c	Adds tests for name-mismatch propagation via verify callback, peer tmp-key retrieval, and negotiated-group naming.
tests/api/test_ssl_ext.c	Adds coverage for accepting FFDHE names in *_groups_list() while keeping *_curves_list() EC-only.
.wolfssl_known_macro_extras	Registers WOLFSSL_NO_SIGNER_DER_CERT as a known macro extra.

Suppressed comments (3)

src/ssl.c:4861

• On X448 public-key export failure this function returns without writing to the output parameter (*pkey), which can leave callers with a stale pointer value after a failure. Set *pkey to NULL before returning failure.

        if (wc_curve448_export_public_ex(ssl->peerX448Key, pub, &pubSz,
                EC448_LITTLE_ENDIAN) != 0) {
            WOLFSSL_MSG("get curve448 public key failed");
            return WOLFSSL_FAILURE;
        }

src/tls.c:9768

• When keepResources is enabled, ssl->peerX25519Key is no longer freed here, and will later be released via FreeKey(..., DYNAMIC_TYPE_CURVE25519, ...) on teardown (see src/internal.c). However the key is allocated with XMALLOC(..., DYNAMIC_TYPE_TLSX) earlier in this function (tls.c:9690), so the eventual XFREE() type will not match the allocation type. This can break custom allocators/pool routing that rely on the dynamic type argument. Consider allocating the stored peer key with DYNAMIC_TYPE_CURVE25519 (or using AllocKey/FreeKey consistently) when it may outlive this function.

This issue also appears on line 9879 of the same file.

    if ((ssl->peerX25519Key != NULL) && !ssl->options.keepResources) {
        wc_curve25519_free(ssl->peerX25519Key);
        XFREE(ssl->peerX25519Key, ssl->heap, DYNAMIC_TYPE_TLSX);
        ssl->peerX25519Key = NULL;
        ssl->peerX25519KeyPresent = 0;

src/tls.c:9883

• When keepResources is enabled, peerX448Key is saved into ssl->peerX448Key and will later be freed via FreeKey(..., DYNAMIC_TYPE_CURVE448, ...) on teardown, but the key is allocated with XMALLOC(..., DYNAMIC_TYPE_TLSX) (tls.c:9838). That means the eventual XFREE() type will not match the allocation type, which can break custom allocators that use the type for pool selection/accounting. Consider allocating the stored peer key with DYNAMIC_TYPE_CURVE448 (or use AllocKey/FreeKey consistently) when it may outlive this function.

    if ((ret == 0) && ssl->options.keepResources) {
        FreeKey(ssl, DYNAMIC_TYPE_CURVE448, (void**)&ssl->peerX448Key);
        ssl->peerX448Key = peerX448Key;
        ssl->peerX448KeyPresent = 1;
    }

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.