Unable to create native thread: possibly out of memory or process/resource limits reached

[ValiDrv]

Hello

Is there a way to debug this error message?

> docker run -p 8080:8080 ghcr.io/observabilitystack/geoip-api:latest
java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
    at com.oracle.svm.core.thread.PlatformThreads.startThread(PlatformThreads.java:741)
    at java.lang.Thread.start0(Thread.java:386)
    at java.lang.Thread.start(Thread.java:802)
    at com.oracle.svm.core.heap.ReferenceHandlerThread.start(ReferenceHandlerThread.java:54)
    at com.oracle.svm.core.graal.snippets.CEntryPointSnippets.initializeIsolate(CEntryPointSnippets.java:292)
    at com.oracle.svm.core.JavaMainWrapper$EnterCreateIsolateWithCArgumentsPrologue.enter(JavaMainWrapper.java:383)

I get this on some (not all) servers, and they pretty powerful 20 core / 256gb ram servers, with identical hardware/os settings.

> ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 768246
max locked memory       (kbytes, -l) 16384
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 768246
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

> docker run --rm debian sh -c "ulimit -a"
time(seconds)        unlimited
file(blocks)         unlimited
data(kbytes)         unlimited
stack(kbytes)        8192
coredump(blocks)     unlimited
memory(kbytes)       unlimited
locked memory(kbytes) 16384
process              unlimited
nofiles              1048576
vmemory(kbytes)      unlimited
locks                unlimited
rtprio               0

[ValiDrv]

More info, in case someone can take a look at it:
The last image that works is 2022-52

docker run -p 8080:8080 observabilitystack/geoip-api:2022-52
--------------------------------------------------------------------------
 ,----.                    ,--. ,------.            ,---.   ,------.  ,--.
'  .-./     ,---.   ,---.  |  | |  .--. ' ,-----.  /  O  \  |  .--. ' |  |
|  | .---. | .-. : | .-. | |  | |  '--' | '-----' |  .-.  | |  '--' | |  |
'  '--'  | \   --. ' '-' ' |  | |  | --'          |  | |  | |  | --'  |  |
 `------'   `----'  `---'  `--' `--'              `--' `--' `--'      `--'
--------------------------------------------------------------------------
GeoIP-API (version 2022-52)
--------------------------------------------------------------------------


2023-05-13 14:12:36.564  INFO 1 --- [           main] com.s24.geoip.GeoIpApi                   : Starting GeoIpApi v2022-52 using Java 17.0.2 on ac921d052d6b with PID 1 (/srv/geoip-api.jar started by root in /srv)
2023-05-13 14:12:36.567  INFO 1 --- [           main] com.s24.geoip.GeoIpApi                   : No active profile set, falling back to 1 default profile: "default"
2023-05-13 14:12:37.757  INFO 1 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat initialized with port(s): 8080 (http)
2023-05-13 14:12:37.768  INFO 1 --- [           main] o.apache.catalina.core.StandardService   : Starting service [Tomcat]
2023-05-13 14:12:37.769  INFO 1 --- [           main] org.apache.catalina.core.StandardEngine  : Starting Servlet engine: [Apache Tomcat/9.0.62]
2023-05-13 14:12:37.858  INFO 1 --- [           main] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring embedded WebApplicationContext
2023-05-13 14:12:37.859  INFO 1 --- [           main] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 1216 ms
2023-05-13 14:12:38.127  INFO 1 --- [           main] com.s24.geoip.GeoIpApi                   : Loaded database file /srv/GeoLite2-City.mmdb (build date: Tue Dec 27 22:20:40 UTC 2022)
2023-05-13 14:12:38.133  INFO 1 --- [           main] com.s24.geoip.GeoIpApi                   : Loaded database file /srv/GeoLite2-ASN.mmdb (build date: Tue Dec 27 21:37:39 UTC 2022)
2023-05-13 14:12:38.462  INFO 1 --- [           main] o.s.b.a.e.web.EndpointLinksResolver      : Exposing 3 endpoint(s) beneath base path '/actuator'
2023-05-13 14:12:38.501  INFO 1 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat started on port(s): 8080 (http) with context path ''
2023-05-13 14:12:38.520  INFO 1 --- [           main] com.s24.geoip.GeoIpApi                   : Started GeoIpApi in 2.397 seconds (JVM running for 2.825)
2023-05-13 14:12:40.886  INFO 1 --- [nio-8080-exec-1] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring DispatcherServlet 'dispatcherServlet'
2023-05-13 14:12:40.886  INFO 1 --- [nio-8080-exec-1] o.s.web.servlet.DispatcherServlet        : Initializing Servlet 'dispatcherServlet'
2023-05-13 14:12:40.887  INFO 1 --- [nio-8080-exec-1] o.s.web.servlet.DispatcherServlet        : Completed initialization in 1 ms

And first image that does not work is: 2023-01

docker run -p 8080:8080 observabilitystack/geoip-api:2023-01
java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
    at com.oracle.svm.core.thread.PlatformThreads.startThread(PlatformThreads.java:741)
    at java.lang.Thread.start0(Thread.java:386)
    at java.lang.Thread.start(Thread.java:802)
    at com.oracle.svm.core.heap.ReferenceHandlerThread.start(ReferenceHandlerThread.java:54)
    at com.oracle.svm.core.graal.snippets.CEntryPointSnippets.initializeIsolate(CEntryPointSnippets.java:292)
    at com.oracle.svm.core.JavaMainWrapper$EnterCreateIsolateWithCArgumentsPrologue.enter(JavaMainWrapper.java:383)

[tboeghk]

This is most likely an issue with the operating systems and/or the limits the operating system supplies for the Java process. Starting with version 2023-01 the application is delivered as a compiled binary and does not need a Java JRE/JDK for running.

Maybe you need to raise your open files from 1024. Linux treats threads as files, maybe thats a hint.

So some quick googleing found this entry: Have you configured your DefaultTasksMax? https://stackoverflow.com/a/72551832

[ValiDrv]

I was thinking the same, but the OS limits seem fine.

BUT, I found out that it does not work on Ubuntu 18.04.4 LTS but works on Ubuntu 18.04.5 LTS and up, all with the same limits.
(tried on 8 servers + 2 workstations)
So I'm thinking it might be some defaults that changed, which Java takes for granted.

On this system it does not work. even if i have higher limits.

# lsb_release -a && systemctl show --property DefaultTasksMax && ulimit -a && docker run -p 8080:8080 observabilitystack/geoip-api:2023-01
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	**Ubuntu 18.04.4 LTS**
Release:	18.04
Codename:	bionic
DefaultTasksMax=231923
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 768246
max locked memory       (kbytes, -l) 16384
max memory size         (kbytes, -m) unlimited
open files                      (-n) 4096
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 768246
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
    at com.oracle.svm.core.thread.PlatformThreads.startThread(PlatformThreads.java:741)
    at java.lang.Thread.start0(Thread.java:386)
    at java.lang.Thread.start(Thread.java:802)
    at com.oracle.svm.core.heap.ReferenceHandlerThread.start(ReferenceHandlerThread.java:54)
    at com.oracle.svm.core.graal.snippets.CEntryPointSnippets.initializeIsolate(CEntryPointSnippets.java:292)
    at com.oracle.svm.core.JavaMainWrapper$EnterCreateIsolateWithCArgumentsPrologue.enter(JavaMainWrapper.java:383)

On this one it does:

lsb_release -a && systemctl show --property DefaultTasksMax && ulimit -a && docker run -p 8080:8080 observabilitystack/geoip-api:2023-01
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	**Ubuntu 18.04.5 LTS**
Release:	18.04
Codename:	bionic
DefaultTasksMax=154631
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 515438
max locked memory       (kbytes, -l) 65536
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 515438
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
--------------------------------------------------------------------------
 ,----.                    ,--. ,------.            ,---.   ,------.  ,--.
'  .-./     ,---.   ,---.  |  | |  .--. ' ,-----.  /  O  \  |  .--. ' |  |
|  | .---. | .-. : | .-. | |  | |  '--' | '-----' |  .-.  | |  '--' | |  |
'  '--'  | \   --. ' '-' ' |  | |  | --'          |  | |  | |  | --'  |  |
 `------'   `----'  `---'  `--' `--'              `--' `--' `--'      `--'
--------------------------------------------------------------------------
 (version )

Also works on Ubuntu 18.04.6 LTS and Ubuntu 22.04.2 LTS