Skip to content

fix: SIGSEGV on dedicated server when HTTP requests are in flight during shutdown - #299

Open
coredmp95 wants to merge 2 commits into
porisius:devfrom
coredmp95:fix/uws-thread-config-read-sigsegv
Open

fix: SIGSEGV on dedicated server when HTTP requests are in flight during shutdown#299
coredmp95 wants to merge 2 commits into
porisius:devfrom
coredmp95:fix/uws-thread-config-read-sigsegv

Conversation

@coredmp95

@coredmp95 coredmp95 commented Jul 20, 2026

Copy link
Copy Markdown

Summary

The dedicated server can SIGSEGV when an HTTP client is polling while the engine shuts down. Two independent faults are involved; this PR fixes both. Verified against a real Linux dedicated server: every attempt crashed before the fix, 3 clean shutdowns out of 3 after.

Two commits, two files, +74 / −8. No behaviour change for a running server — the only observable difference is that requests arriving during shutdown now get 503 instead of taking the server down.


Symptom

Any HTTP client polling FRM (a dashboard, an Arduino integration, a monitoring loop) can kill the dedicated server when the server is stopped. Exit code 139, Segmentation fault (core dumped), no shutdown save.

An idle server shuts down cleanly, which is why this is easy to miss — it only reproduces when requests are actually in flight.


Root cause

Fault 1 — config read reaches GEngine from the uWS worker thread

The uWS request handlers read uWS.AuthenticationToken from config on every request. That call chain ends inside the engine:

UEngine::GetGameUserSettings()                 <- SIGSEGV, reads 0x2a0
UFGGameUserSettings::GetFGGameUserSettings()
UFRMConfigManager::GetConfig<FString>()
  <- AFicsitRemoteMonitoring::StartWebSocketServer lambda
  <- uWS::HttpRouter::executeHandlers
  <- uWS::HttpParser::fenceAndConsumePostPadded

GetFGGameUserSettings() dereferences GEngine internally. This runs on the uWS worker thread, where nothing guarantees GEngine is still alive. During teardown it is destroyed while requests are still being served.

GetConfig does null-check its UserSettings pointer — but that check is downstream of the call that faults, so it never gets the chance to help.

Fault 2 — IsValid() cannot guard against a freed pointer

Fixing fault 1 did not fix the crash; it moved deeper, into endpoint execution:

AFicsitRemoteMonitoring::CallEndpoint()
AFicsitRemoteMonitoring::HandleEndpoint()
AFicsitRemoteMonitoring::HandleApiRequest()
  <- StartWebSocketServer lambda
  <- uWS::HttpRouter::executeHandlers

The faulting instruction is CallEndpoint's own guard:

if (!IsValid(WorldContext) || !IsValid(WorldContext->GetWorld()))

That guard cannot work in this situation. During teardown WorldContext is not null — it is freed. IsValid() has to dereference the object to read its flags, so the guard is reached and then faults on the very check meant to prevent the fault.

The same reasoning applies to this: the handler lambdas capture the subsystem actor, which is also being destroyed. So no UObject-based check is safe in this window, including checking a member variable.


The fix

Commit 1 — fix: don't read game user settings from the uWS thread

  • Cache the token. BeginPlay already reads it on the game thread. It now stores it in the existing AuthenticationToken member — which was declared but never assigned — and the three handlers read that member instead of hitting config per request. This also removes a redundant settings lookup from the hot path on every single request.
  • Assign the member on the freshly-generated branch too. This one is load-bearing, not tidy-up: when no token exists yet, BeginPlay generates one and writes it to config, but previously left the member empty. Once the handlers read the member instead of config, an unassigned member would mean IsAuthorizedRequest compares every header against "" — so a first-boot server would reject every authenticated request until restart. The assignment is what keeps the caching change from regressing auth on a fresh install.
  • Guard GetConfig. Check GEngine before the call that produces the settings pointer, and return false so callers fall back to their default. This also covers the other background-thread readers: uWS.PushCycle in the push loop and Debug.JSONDebug during JSON serialization.
  • Also logs the generated token instead of the local that is empty by definition on that branch (cosmetic, one line).

Commit 2 — fix: reject uWS requests once teardown has begun

Adds a barrier that consults no UObject at all:

  • GFRMShuttingDown, a file-scope std::atomic<bool>. It is deliberately not a member, because this may already be freed when a handler runs.
  • StopWebSocketServer() raises it first, before anything is torn down. That function is already wired to EndPlay and FCoreDelegates::OnExit, so both normal shutdown and exit paths are covered.
  • StartWebSocketServer() lowers it, so a server restarted within the same session (/frm http start after a stop) does not answer 503 forever.
  • The four request handlers that reach the world check it first and answer 503 Service Unavailable before touching the subsystem, the world, or any UObject.

Reproduction

# 1. start the dedicated server
# 2. poll any endpoint in a loop
while true; do curl -s -o /dev/null http://localhost:8080/getPlayer; done
# 3. stop the server
kill -TERM <FactoryServer pid>
Build Poller at SIGTERM Result
unpatched yes crashed, exit 139 (2 / 2 attempts)
unpatched no clean, exit 143 — control
commit 1 only yes still crashed, exit 139 — fault 2 surfaced here
commit 1 + 2 yes clean, exit 143 (3 / 3 attempts)

The "no poller" row is the control: same binary, same signal, no in-flight requests, no crash. That is what isolates the request path as the cause rather than shutdown in general.

The "commit 1 only" row is worth calling out — it is how fault 2 was found. Commit 1 removed every UFRMConfigManager frame from the stack, and the crash reappeared one layer deeper. If you review the commits separately, commit 1 alone will not stop the crash.


Scope

Changed: the auth-token read path, one guard in GetConfig, and a shutdown barrier in the four world-touching request handlers.

Not changed: endpoint logic, response formats, the WebSocket push path, or how a token is validated once read. A running server serves the same responses with the same fields.

Behavioural delta 1 — shutdown. Requests arriving after shutdown has begun now receive 503 {"error":"Server is shutting down"} instead of crashing the process. Previously there was no defined behaviour for that window.

Behavioural delta 2 — the auth token is now read once, not per request. This follows directly from the caching in commit 1 and is worth stating plainly, because it is the one place a running server does not behave identically:

  • Before, the handlers re-read uWS.AuthenticationToken from config on every request, so a mid-session change to that key took effect on the very next request.
  • Now the value is captured in BeginPlay and a mid-session change does not take effect until the world restarts.

The path that can hit this is the admin chat command in Commands/multi.cpp, which forwards any key/value to SetConfigFromInput. Point it at uWS.AuthenticationToken and it will still report Configured … in green while the server keeps authenticating against the old token — a silent no-op rather than an error.

I have deliberately not fixed that here, to keep this PR to the crash. It is worth a follow-up (have the setter refresh the cached member, or reject that key from the chat command). Two things bound the practical impact in the meantime: the token is normally set once at first boot and left alone, and that command already lowercases its argument (Arguments[2].ToLower()) while GenerateAuthToken produces mixed case — so setting a token by chat was already unreliable before this PR. Happy to fold a fix in here instead if you would rather not merge the regression, however small.


Notes for review

  • The std::atomic is file-scope by necessity, not by preference — a member would be unsafe for exactly the reason fault 2 describes. Same rationale as the existing file-scope SocketRunning / SocketListener in that file.
  • memory_order_release on the store and memory_order_acquire on the load: the barrier must be visible to the uWS thread before teardown proceeds.
  • The GEngine guard in GetConfig returns false rather than logging an error, because during teardown the log subsystem is itself going away; callers already handle false by using their default.
  • Each commit stands alone and is individually revertible. Commit 1 is a real improvement on its own (it removes a per-request settings lookup) but does not fix the crash by itself — commit 2 is the one that does.

coredmp95 and others added 2 commits July 20, 2026 21:25
The uWS HTTP handlers read uWS.AuthenticationToken from config on every
request. UFRMConfigManager::GetConfig reaches UFGGameUserSettings::
GetFGGameUserSettings(), which dereferences GEngine internally. That runs on
the uWS worker thread, where nothing guarantees GEngine is still alive, and it
faults during engine teardown:

    UEngine::GetGameUserSettings()                 <- SIGSEGV
    UFGGameUserSettings::GetFGGameUserSettings()
    UFRMConfigManager::GetConfig<FString>()
      <- AFicsitRemoteMonitoring::StartWebSocketServer lambda
      <- uWS::HttpRouter::executeHandlers

Reproduced on a Linux dedicated server: poll any endpoint in a loop, then send
SIGTERM. With the poll loop running the server died with SIGSEGV twice out of
two; with the poll loop stopped, the same signal on the same binary exited
cleanly (143).

Two changes:

- Cache the token. BeginPlay already reads it on the game thread; it now stores
  it in the existing AuthenticationToken member (previously assigned nowhere),
  and the three handlers read that member instead of hitting config per request.
  This also drops a redundant settings lookup from the hot path.

- Guard GetConfig. The existing UserSettings null-check cannot help, because the
  fault happens inside the call that produces that pointer. Check GEngine first
  and return false so callers fall back to their default rather than crashing.
  This covers the other background-thread readers too, notably uWS.PushCycle in
  the push loop and Debug.JSONDebug during serialization.

Also log the generated token rather than the local that is empty by definition
on that branch.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Caching the auth token removed one faulting path, but the crash simply moved
deeper — into endpoint execution itself:

    AFicsitRemoteMonitoring::CallEndpoint()     FicsitRemoteMonitoring.cpp
    AFicsitRemoteMonitoring::HandleEndpoint()
    AFicsitRemoteMonitoring::HandleApiRequest()
      <- StartWebSocketServer lambda
      <- uWS::HttpRouter::executeHandlers

The faulting instruction is CallEndpoint's own guard:

    if (!IsValid(WorldContext) || !IsValid(WorldContext->GetWorld()))

That guard cannot work here. During teardown WorldContext is not null, it is
freed, and IsValid() must dereference the object to read its flags — so the
guard is reached and then faults. The same applies to `this`: the handler
lambdas capture the subsystem actor, which is being destroyed too.

Add a barrier that consults no UObject at all:

- GFRMShuttingDown, a file-scope std::atomic<bool>. It must not be a member,
  because `this` may already be freed when a handler runs.
- StopWebSocketServer() raises it first, before anything is torn down. That
  function is already wired to EndPlay and FCoreDelegates::OnExit.
- StartWebSocketServer() lowers it, so a server restarted in the same session
  (`/frm http start` after a stop) does not answer 503 forever.
- The four request handlers that reach the world check it first and answer
  503 before touching the subsystem, the world, or any UObject.

Verified on a Linux dedicated server with the documented repro — poll endpoints
in a loop, then SIGTERM. Before: 3 crashes out of 3 attempts (exit 139). After:
3 clean shutdowns out of 3 (exit 143).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant