[BUG] OCSP cache reload causes extremely slow (70+ min) EJBCA startup at high CA counts

[charlesday99]

Describe the Bug

On startup, StartupSingletonBean.startup() unconditionally calls ocspResponseGeneratorSession.initTimers(), which (when the OCSP signing cache has no entries, e.g. on a fresh boot) calls reloadOcspSigningCache(). This method sequentially activates the crypto token for every single CA in the system in order to build the OCSP signing cache entries, even for CAs that are never actually used to serve OCSP responses (no OcspKeyBinding/end-entity OCSP traffic configured).

In an environment with 10,000+ CAs, this causes each CA's SoftCryptoToken to be activated one-by-one during startup, resulting in a full application startup time of approximately 70+ minutes. This delay occurs on every restart, upgrade, or failover, significantly impacting operational recovery time.

This reload is not conditional on whether OCSP is actually used by the deployment, it runs regardless of whether any OCSP requests are ever served.

To Reproduce

1. Configure an EJBCA instance with a large number of CAs (10,000+) using software crypto tokens.
2. Restart the EJBCA application (systemctl restart <wildfly-service> or equivalent).
3. Observe the WildFly/application server log during startup, a SoftCryptoToken activation log line is emitted sequentially for each CA before the application becomes available.
4. Time from process start to the application accepting requests.

Expected Behaviour

Startup time should not scale linearly with the total number of configured CAs when those CAs are not actively used for OCSP responding. CA crypto tokens needed only for OCSP signing should ideally be activated lazily/on-demand (i.e., the first time an OCSP request actually requires that CA's signing key), rather than eagerly for every CA at boot.

Screenshots and Logs

Initial service start:

Aug 24 12:40:35 zynops systemd[1]: Started zynstra-ejbca.service - EJBCA Community Edition via Wildfly.

Start of the Crypto token activation which takes ~70 minutes to complete:

Aug 24 12:41:58 zynops standalone.sh[2075]: 12:41:58,021 INFO  [org.cesecore.certificates.certificate.CertificateStoreSessionBean] (ServerService Thread Pool -- 79) Setting isUniqueCertificateSerialNumberIndex to: true
Aug 24 12:41:58 zynops standalone.sh[2075]: 12:41:58,036 INFO  [org.cesecore.audit.impl.log4j.Log4jDevice] (ServerService Thread Pool -- 79) 2026-08-24 12:41:58+00:00;ACCESS_CONTROL;SUCCESS;ACCESSCONTROL;CORE;Internal upgrade;;;;resource0=/system_functionality/edit_systemconfiguration
Aug 24 12:41:58 zynops standalone.sh[2075]: 12:41:58,049 INFO  [org.cesecore.audit.impl.log4j.Log4jDevice] (ServerService Thread Pool -- 79) 2026-08-24 12:41:58+00:00;SYSTEMCONF_EDIT;SUCCESS;GLOBALCONF;CORE;Internal upgrade;;;;msg=Saved global configuration with id UPGRADE.
Aug 24 12:41:58 zynops standalone.sh[2075]: 12:41:58,060 INFO  [org.ejbca.core.ejb.upgrade.UpgradeSessionBean] (ServerService Thread Pool -- 79) Database content version: 9.1.1, current application version: 9.1.1 -> Upgrade is not needed.
Aug 24 12:41:58 zynops standalone.sh[2075]: 12:41:58,482 INFO  [org.cesecore.keys.token.SoftCryptoToken] (ServerService Thread Pool -- 79) Activated Crypto Token with ID 855846612.
Aug 24 12:41:58 zynops standalone.sh[2075]: 12:41:58,765 INFO  [org.cesecore.keys.token.SoftCryptoToken] (ServerService Thread Pool -- 79) Activated Crypto Token with ID 425602324.
Aug 24 12:41:59 zynops standalone.sh[2075]: 12:41:59,022 INFO  [org.cesecore.keys.token.SoftCryptoToken] (ServerService Thread Pool -- 79) Activated Crypto Token with ID 927518098.
...
Aug 24 13:50:41 zynops standalone.sh[2075]: 13:50:41,666 INFO  [org.cesecore.keys.token.SoftCryptoToken] (ServerService Thread Pool -- 79) Activated Crypto Token with ID 263898837.
Aug 24 13:50:41 zynops standalone.sh[2075]: 13:50:41,908 INFO  [org.cesecore.keys.token.SoftCryptoToken] (ServerService Thread Pool -- 79) Activated Crypto Token with ID 1177961987.
Aug 24 13:50:44 zynops standalone.sh[2075]: 13:50:44,440 INFO  [org.cesecore.certificates.ocsp.cache.OcspSigningCache] (ServerService Thread Pool -- 79) No default responder was defined. OCSP requests for certificates issued by unknown CAs will return "unauthorized" as per RFC6960, Section 2.3

Final log line indicating the application is fully started:

Aug 24 13:51:13 zynops standalone.sh[2075]: 13:51:13,729 INFO  [org.jboss.as] (Controller Boot Thread) WFLYSRV0025: WildFly 33.0.2.Final (WildFly Core 25.0.2.Final) started in 4237451ms - Started 5124 of 5335 services (396 services are lazy, passive or on-demand)

Product Deployment

Deployment format: software (WildFly-deployed ejbca.ear)
Version: 9.1.1 (confirmed present from EJBCA 9.x onward; confirmed absent in 8.2.0)
Application server: WildFly 33.0.2.Final (WildFly Core 25.0.2.Final)

Server

OS: Ubuntu
Version: 24.04.4 LTS

Additional Context

• We do not believe our OCSP endpoint is under meaningful load, the majority of our CAs are never queried for OCSP status. Despite this, all CAs' crypto tokens are activated unconditionally at every startup.
• As a possible way forward, we'd suggest either (a) allowing the OCSP responder web service to be fully disabled for deployments that don't use it, so this startup work can be skipped entirely, or (b) having CAs activate their crypto tokens lazily, only when actually needed to answer an OCSP request, rather than all CAs being activated upfront on every restart. We'd be happy to discuss either approach further and help test a fix.