Fix panic and lost EOS when kill races pipeline startup - #473
Open
pksgit wants to merge 1 commit into
Open
Conversation
pksgit
force-pushed
the
pradeepsharma/cs-1376-ingress-panic-on-sendeos
branch
2 times, most recently
from
August 28, 2026 16:05
d65b28a to
fd23c03
Compare
Handler.HandleIngress starts Pipeline.Run on a goroutine and calls SendEOS from its kill watcher, so a DeleteIngress arriving during startup runs both against the same Pipeline. Two things went wrong: Run created p.loop, so a SendEOS that got there first dereferenced a nil loop. That panic is raised on a goroutine SendEOS spawns, where nothing can recover it, so it took the whole handler process down. Once the loop existed, a quit issued before Run reached loop.Run() was still lost: g_main_loop_run sets is_running=TRUE on entry, overwriting the FALSE that g_main_loop_quit wrote. The flag cannot distinguish "not started yet" from "asked to stop", so the request was never representable rather than discarded. The loop then ran with nobody left to stop it. Nothing reaps a handler in that state -- the process manager only cleans up once cmd.Run returns, its SIGKILL backstop is guarded on a fuse already broken by then, and the handler traps the SIGINT that killAll sends -- so the process outlives its ingress, holds the room participant open, and blocks the instance from draining. That failure is silent, and its window spans pipeline.Start and input.Start, so it is likely more common than the panic that got reported. Build the loop in New so it is never nil, and queue quits as idle sources, which live on the context rather than in a flag Run overwrites and are dispatched as soon as the loop starts. The fix also makes the race invisible: a kill during startup now tears down cleanly and leaves no trace. Run warns when it finds the fuse already broken on its way to the loop, so the race stays observable after it stops being fatal. The tests drive Run itself through a stub Source, which is what lets them cover the loop moving back out of New, and assert on real log output from a child process. Each half of the fix was reverted to confirm the suite fails without it: a direct quit fails four tests, and recreating the loop inside Run fails two. Fixes CS-1376 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
pksgit
force-pushed
the
pradeepsharma/cs-1376-ingress-panic-on-sendeos
branch
from
August 28, 2026 16:17
fd23c03 to
98b4528
Compare
pksgit
marked this pull request as ready for review
August 28, 2026 16:22
Contributor
we launch handler per ingress - handler process going down is actually causing failing egress |
| pipeline: pipeline, | ||
| input: input, | ||
| // Built here rather than in Run so that it is never nil: SendEOS can | ||
| // run before Run starts, and used to dereference a nil loop and take |
Contributor
There was a problem hiding this comment.
Please update the comment to mention what the code does, not reviewer only relevant info about what the code used to do.
| // ones all want a network peer. | ||
| type stubSource struct{} | ||
|
|
||
| func (stubSource) GetSources() []*gst.Element { return nil } |
Contributor
There was a problem hiding this comment.
Would using counterfeiter on the Source interface work here instead?
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Handler.HandleIngressstartsPipeline.Runon a goroutine and callsSendEOSfrom its kill watcher, so aDeleteIngressarriving during startup runs both against the samePipeline. Two things went wrong.The reported panic
Runcreatedp.loop, so aSendEOSthat got there first dereferenced a nil loop:It is raised on a goroutine
SendEOSspawns, where nothing can recover it, so it took the whole handler process down rather than failing one ingress. Theif p.loop != nilguard already sitting in the sink's close callback is the fingerprint of someone hitting this earlier on that path.The failure underneath it
Fixing the nil is not enough. Once the loop exists, a quit issued before
Runreachesloop.Run()is still lost, and this half is silent.g_main_loop_quithas no running check — it unconditionally setsis_running = FALSE.g_main_loop_runthen setsTRUEon entry, unconditionally, and only afterwards reads it:So the quit is not rejected, it is overwritten.
FALSEis also the loop's initial value, so the flag cannot distinguish "not started yet" from "asked to stop" — the request was never representable. The only reader of that flag is thewhileinsideRun.Nothing reaps a handler stuck there. The process manager only cleans up once
cmd.Runreturns; its SIGKILL backstop is guarded on a fuse already broken by then; and the handler traps the SIGINT thatkillAllsends. The process outlives its ingress, holds the room participant open (neitherinput.Closenorsink.Closeis reached), fails theDeleteIngress, and blocks the instance from draining —for !s.sm.IsIdle()never exits.Its window spans
pipeline.Startandinput.Start, so it is considerably wider than the panic window and likely more common than what got reported.The change
Newso it is never nil.quitLoopqueues quits as idle sources, which live on the context rather than in a flagRunoverwrites, and are dispatched as soon as the loop starts.Quit()calls inmessageWatchare untouched: those run on the loop's own thread while it is already spinning, where a direct quit is correct.Measured on GStreamer 1.28, all four timings:
Quitbefore the loop startsIdleAdd(Quit)before the loop startsQuitwhile runningIdleAdd(Quit)while runningThe change converts A into B. C and D are unaffected, so steady-state shutdown behaviour does not change.
Tests
Five tests, verified to fail without the fix rather than merely to pass with it. Reverting
quitLoopto a direct quit failsTestSendEOSBeforeRunIsHonored,TestQuitLoopBeforeRunIsHonoredandTestQuitLoopIsSafeConcurrently.TestDirectQuitBeforeRunIsLostpins the GStreamer behaviour this fix exists for: if a future version honours an early quit, it fails and says the indirection may no longer be needed.Green under
-raceand-count=3.Reviewer notes
Pipelinevalue, so moving the loop back intoRundoes not fail anything. Covering it needs eitherInput/WebRTCSinkbehind interfaces so a test can callRun, or a test that asserts on the source. Neither is in this PR.IdleAddtargets the default main context. That coupling is load bearing: a second main loop in this process would share the context, dispatch the source before our loop starts, and lose the quit again — silently. Nothing else in ingress creates a main loop today and the handler builds one pipeline, so this is latent, not live. A private context is not practical here, sincegst_bus_add_watchalso targets the default one.Quitcannot help in the race.IdleAddfails only on a bad callback type or a failed allocation, so it is a can't-happen branch, and if it ever fired during the window it is exactly the call that does not work. Happy to drop it to log-only.Runwhen EOS was already requested. Correctness does not need it, but without it an ingress killed during startup still connects to the room before tearing down.The same shape exists in egress at
pkg/gstreamer/pipeline.go(Stopquits a loopRunhas not started yet), thoughStopdoesOnStopplus a full state change to NULL before quitting, so the window is far narrower there. Egress already builds its loop in the constructor, so it has no nil exposure. Raised separately; not addressed here.🤖 Generated with Claude Code