Recover stuck graphs by continuing queued next_nodes when no interrupt exists - #12
Open
MikaAK wants to merge 1 commit into
Open
Recover stuck graphs by continuing queued next_nodes when no interrupt exists#12MikaAK wants to merge 1 commit into
MikaAK wants to merge 1 commit into
Conversation
…t exists
When a graph stops mid-execution at a non-interrupt node (e.g. the
calling process crashes after a checkpoint is written but before the
queued node finishes), the latest checkpoint has next_nodes populated
but pending_interrupts: nil. Subsequent calls to LangEx.invoke with
%Command{resume: _} previously returned {:error, :no_pending_interrupt}
and left the graph permanently stuck.
This change adds a fallback in resume_from_checkpoint: when no pending
interrupt is found anywhere but the latest checkpoint has runnable
next_nodes (excluding :__end__), continue execution from those nodes.
The resume value is discarded since there is no interrupt to feed it
to — callers using a dummy resume_val purely as a "nudge" trigger
(e.g. cron-driven re-observation of external state) get automatic
recovery without needing to detect the stuck state themselves.
Pregel.run_graph/3 gains a new clause that handles next_nodes-driven
start as a peer to the existing :__start__ and resume paths.
Strictly additive: previously-erroring no-pending-interrupt cases
either now recover (when next_nodes is non-empty) or still error.
Collaborator
|
@claude review |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
When a graph stops mid-execution at a non-interrupt node — for example, the calling process crashes after a checkpoint is written but before the queued node finishes — the latest checkpoint has
next_nodespopulated butpending_interrupts: nil. Subsequent calls toLangEx.invoke(graph, %Command{resume: _}, ...)previously returned{:error, :no_pending_interrupt}and left the graph permanently stuck with no automatic recovery path.This PR adds a fallback in
resume_from_checkpoint: when no pending interrupt is found but the latest checkpoint has runnablenext_nodes(excluding:__end__), continue execution from those nodes. The resume value is discarded since there is no interrupt to feed it to — callers using a dummy resume_val as a "nudge" trigger (e.g. cron-driven re-observation of external state) get automatic recovery without having to detect the stuck state themselves.Pregel.run_graph/3gains a new clause that handlesnext_nodes-driven start as a peer to the existing:__start__and resume paths.The change is strictly additive: previously-erroring no-pending-interrupt cases either now recover (when
next_nodesis non-empty) or still error.How this stuck state happens in practice
Real-world reproduction: a Pregel super-step writes a checkpoint with
next_nodes: [:create_pr], then the worker process is killed (crash, code reload, deploy) before:create_prcan execute. The next cron-drivennudgecall sees no pending interrupt anywhere in the last N checkpoints and bails out. The graph is dead.After this PR: the same nudge call notices the queued non-interrupt nodes and resumes execution from them.
Test plan
test/lang_ex/checkpoint/resume_stuck_at_next_nodes_test.exswith three cases:next_nodeswhen no pending interrupt exists anywhere (was failing, now passing){:error, :no_pending_interrupt}whennext_nodesis empty{:error, :no_pending_interrupt}whennext_nodesonly has:__end__mix format --check-formattedclean