fix: SIGSEGV on dedicated server when HTTP requests are in flight during shutdown - #299
Open
coredmp95 wants to merge 2 commits into
Open
fix: SIGSEGV on dedicated server when HTTP requests are in flight during shutdown#299coredmp95 wants to merge 2 commits into
coredmp95 wants to merge 2 commits into
Conversation
The uWS HTTP handlers read uWS.AuthenticationToken from config on every
request. UFRMConfigManager::GetConfig reaches UFGGameUserSettings::
GetFGGameUserSettings(), which dereferences GEngine internally. That runs on
the uWS worker thread, where nothing guarantees GEngine is still alive, and it
faults during engine teardown:
UEngine::GetGameUserSettings() <- SIGSEGV
UFGGameUserSettings::GetFGGameUserSettings()
UFRMConfigManager::GetConfig<FString>()
<- AFicsitRemoteMonitoring::StartWebSocketServer lambda
<- uWS::HttpRouter::executeHandlers
Reproduced on a Linux dedicated server: poll any endpoint in a loop, then send
SIGTERM. With the poll loop running the server died with SIGSEGV twice out of
two; with the poll loop stopped, the same signal on the same binary exited
cleanly (143).
Two changes:
- Cache the token. BeginPlay already reads it on the game thread; it now stores
it in the existing AuthenticationToken member (previously assigned nowhere),
and the three handlers read that member instead of hitting config per request.
This also drops a redundant settings lookup from the hot path.
- Guard GetConfig. The existing UserSettings null-check cannot help, because the
fault happens inside the call that produces that pointer. Check GEngine first
and return false so callers fall back to their default rather than crashing.
This covers the other background-thread readers too, notably uWS.PushCycle in
the push loop and Debug.JSONDebug during serialization.
Also log the generated token rather than the local that is empty by definition
on that branch.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Caching the auth token removed one faulting path, but the crash simply moved
deeper — into endpoint execution itself:
AFicsitRemoteMonitoring::CallEndpoint() FicsitRemoteMonitoring.cpp
AFicsitRemoteMonitoring::HandleEndpoint()
AFicsitRemoteMonitoring::HandleApiRequest()
<- StartWebSocketServer lambda
<- uWS::HttpRouter::executeHandlers
The faulting instruction is CallEndpoint's own guard:
if (!IsValid(WorldContext) || !IsValid(WorldContext->GetWorld()))
That guard cannot work here. During teardown WorldContext is not null, it is
freed, and IsValid() must dereference the object to read its flags — so the
guard is reached and then faults. The same applies to `this`: the handler
lambdas capture the subsystem actor, which is being destroyed too.
Add a barrier that consults no UObject at all:
- GFRMShuttingDown, a file-scope std::atomic<bool>. It must not be a member,
because `this` may already be freed when a handler runs.
- StopWebSocketServer() raises it first, before anything is torn down. That
function is already wired to EndPlay and FCoreDelegates::OnExit.
- StartWebSocketServer() lowers it, so a server restarted in the same session
(`/frm http start` after a stop) does not answer 503 forever.
- The four request handlers that reach the world check it first and answer
503 before touching the subsystem, the world, or any UObject.
Verified on a Linux dedicated server with the documented repro — poll endpoints
in a loop, then SIGTERM. Before: 3 crashes out of 3 attempts (exit 139). After:
3 clean shutdowns out of 3 (exit 143).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The dedicated server can
SIGSEGVwhen an HTTP client is polling while the engine shuts down. Two independent faults are involved; this PR fixes both. Verified against a real Linux dedicated server: every attempt crashed before the fix, 3 clean shutdowns out of 3 after.Two commits, two files, +74 / −8. No behaviour change for a running server — the only observable difference is that requests arriving during shutdown now get
503instead of taking the server down.Symptom
Any HTTP client polling FRM (a dashboard, an Arduino integration, a monitoring loop) can kill the dedicated server when the server is stopped. Exit code
139,Segmentation fault (core dumped), no shutdown save.An idle server shuts down cleanly, which is why this is easy to miss — it only reproduces when requests are actually in flight.
Root cause
Fault 1 — config read reaches
GEnginefrom the uWS worker threadThe uWS request handlers read
uWS.AuthenticationTokenfrom config on every request. That call chain ends inside the engine:GetFGGameUserSettings()dereferencesGEngineinternally. This runs on the uWS worker thread, where nothing guaranteesGEngineis still alive. During teardown it is destroyed while requests are still being served.GetConfigdoes null-check itsUserSettingspointer — but that check is downstream of the call that faults, so it never gets the chance to help.Fault 2 —
IsValid()cannot guard against a freed pointerFixing fault 1 did not fix the crash; it moved deeper, into endpoint execution:
The faulting instruction is
CallEndpoint's own guard:That guard cannot work in this situation. During teardown
WorldContextis not null — it is freed.IsValid()has to dereference the object to read its flags, so the guard is reached and then faults on the very check meant to prevent the fault.The same reasoning applies to
this: the handler lambdas capture the subsystem actor, which is also being destroyed. So noUObject-based check is safe in this window, including checking a member variable.The fix
Commit 1 —
fix: don't read game user settings from the uWS threadBeginPlayalready reads it on the game thread. It now stores it in the existingAuthenticationTokenmember — which was declared but never assigned — and the three handlers read that member instead of hitting config per request. This also removes a redundant settings lookup from the hot path on every single request.BeginPlaygenerates one and writes it to config, but previously left the member empty. Once the handlers read the member instead of config, an unassigned member would meanIsAuthorizedRequestcompares every header against""— so a first-boot server would reject every authenticated request until restart. The assignment is what keeps the caching change from regressing auth on a fresh install.GetConfig. CheckGEnginebefore the call that produces the settings pointer, and returnfalseso callers fall back to their default. This also covers the other background-thread readers:uWS.PushCyclein the push loop andDebug.JSONDebugduring JSON serialization.Commit 2 —
fix: reject uWS requests once teardown has begunAdds a barrier that consults no
UObjectat all:GFRMShuttingDown, a file-scopestd::atomic<bool>. It is deliberately not a member, becausethismay already be freed when a handler runs.StopWebSocketServer()raises it first, before anything is torn down. That function is already wired toEndPlayandFCoreDelegates::OnExit, so both normal shutdown and exit paths are covered.StartWebSocketServer()lowers it, so a server restarted within the same session (/frm http startafter a stop) does not answer503forever.503 Service Unavailablebefore touching the subsystem, the world, or anyUObject.Reproduction
The "no poller" row is the control: same binary, same signal, no in-flight requests, no crash. That is what isolates the request path as the cause rather than shutdown in general.
The "commit 1 only" row is worth calling out — it is how fault 2 was found. Commit 1 removed every
UFRMConfigManagerframe from the stack, and the crash reappeared one layer deeper. If you review the commits separately, commit 1 alone will not stop the crash.Scope
Changed: the auth-token read path, one guard in
GetConfig, and a shutdown barrier in the four world-touching request handlers.Not changed: endpoint logic, response formats, the WebSocket push path, or how a token is validated once read. A running server serves the same responses with the same fields.
Behavioural delta 1 — shutdown. Requests arriving after shutdown has begun now receive
503 {"error":"Server is shutting down"}instead of crashing the process. Previously there was no defined behaviour for that window.Behavioural delta 2 — the auth token is now read once, not per request. This follows directly from the caching in commit 1 and is worth stating plainly, because it is the one place a running server does not behave identically:
uWS.AuthenticationTokenfrom config on every request, so a mid-session change to that key took effect on the very next request.BeginPlayand a mid-session change does not take effect until the world restarts.The path that can hit this is the admin chat command in
Commands/multi.cpp, which forwards any key/value toSetConfigFromInput. Point it atuWS.AuthenticationTokenand it will still reportConfigured …in green while the server keeps authenticating against the old token — a silent no-op rather than an error.I have deliberately not fixed that here, to keep this PR to the crash. It is worth a follow-up (have the setter refresh the cached member, or reject that key from the chat command). Two things bound the practical impact in the meantime: the token is normally set once at first boot and left alone, and that command already lowercases its argument (
Arguments[2].ToLower()) whileGenerateAuthTokenproduces mixed case — so setting a token by chat was already unreliable before this PR. Happy to fold a fix in here instead if you would rather not merge the regression, however small.Notes for review
std::atomicis file-scope by necessity, not by preference — a member would be unsafe for exactly the reason fault 2 describes. Same rationale as the existing file-scopeSocketRunning/SocketListenerin that file.memory_order_releaseon the store andmemory_order_acquireon the load: the barrier must be visible to the uWS thread before teardown proceeds.GEngineguard inGetConfigreturnsfalserather than logging an error, because during teardown the log subsystem is itself going away; callers already handlefalseby using their default.