Skip to content

fix(dfir_lang): order loop-ingress senders before the whole loop block, fix #3048#3049

Open
MingweiSamuel wants to merge 3 commits into
mainfrom
mingwei/loop-contig
Open

fix(dfir_lang): order loop-ingress senders before the whole loop block, fix #3048#3049
MingweiSamuel wants to merge 3 commits into
mainfrom
mingwei/loop-contig

Conversation

@MingweiSamuel

@MingweiSamuel MingweiSamuel commented Jul 20, 2026

Copy link
Copy Markdown
Member

make_loops_contiguous gathers every subgraph of a loop to the flat-order
position of the loop's first subgraph. This is only sound if every subgraph
feeding the loop from outside (loop ingress) is already ordered before the
loop's earliest subgraph. Otherwise gathering hoists an ingress receiver
inside the loop ahead of its external sender, violating topological order.
Downstream this produced a handoff-buffer "use before declaration" compile
error (and, logically, a drained-but-never-filled batch).

Fix: in find_subgraph_unionfind, add loop-ingress ordering constraints to the
predecessor set that feeds the (existing, unified) topological sort. For each
forward (non-tick) edge src -> dst where dst lies inside a loop that does
not contain src, require src to precede every node inside the outermost
such loop — i.e. the whole top-level loop block that will be gathered. Because
this reuses the same toposort, all other ordering constraints (handoff
references, access groups) are respected automatically, and non-loop programs
are entirely unaffected (no constraints are added). A genuine
loop -> external -> loop dependency (which cannot be made contiguous) now
surfaces as the usual intra-tick cycle error.

Also adds a defensive assertion after make_loops_contiguous that verifies the
toposort is still valid.

Tests:

  • dfir_lang: test_make_loops_contiguous_ingress_toposort (previously
    #[ignore]d as a repro) now asserts a valid toposort and passes.
  • dfir_rs surface_loop: adds test_loop_ingress_forward_ref_sources, an
    end-to-end surface-syntax repro (two ingress points with sources declared
    after the loop) that both compiles and delivers all data. Both tests were
    confirmed to fail without the fix (the integration test fails at macro
    expansion via the new debug assertion).

`make_loops_contiguous` gathers every subgraph of a loop to the flat-order
position of the loop's *first* subgraph. This is only sound if every subgraph
feeding the loop from outside (loop ingress) is already ordered before the
loop's earliest subgraph. Otherwise gathering hoists an ingress *receiver*
inside the loop ahead of its external *sender*, violating topological order.
Downstream this produced a handoff-buffer "use before declaration" compile
error (and, logically, a drained-but-never-filled batch).

Fix: in `find_subgraph_unionfind`, add loop-ingress ordering constraints to the
predecessor set that feeds the (existing, unified) topological sort. For each
forward (non-tick) edge `src -> dst` where `dst` lies inside a loop that does
not contain `src`, require `src` to precede *every* node inside the outermost
such loop — i.e. the whole top-level loop block that will be gathered. Because
this reuses the same toposort, all other ordering constraints (handoff
references, access groups) are respected automatically, and non-loop programs
are entirely unaffected (no constraints are added). A genuine
`loop -> external -> loop` dependency (which cannot be made contiguous) now
surfaces as the usual intra-tick cycle error.

Also adds a defensive `debug_assert` (`subgraph_toposort_respects_handoffs`)
after `make_loops_contiguous` that verifies every forward handoff's producer
subgraph precedes its consumer subgraph, catching any future regression at
compile time.

Tests:
- `dfir_lang`: `test_make_loops_contiguous_ingress_toposort` (previously
  `#[ignore]`d as a repro) now asserts a valid toposort and passes.
- `dfir_rs` `surface_loop`: adds `test_loop_ingress_forward_ref_sources`, an
  end-to-end surface-syntax repro (two ingress points with sources declared
  after the loop) that both compiles and delivers all data. Both tests were
  confirmed to fail without the fix (the integration test fails at macro
  expansion via the new debug assertion).

Co-authored-by: Infinity 🤖 <infinity@hydro.run>
PR: #3049
@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Jul 20, 2026

Copy link
Copy Markdown

Deploying hydro with  Cloudflare Pages  Cloudflare Pages

Latest commit: 41f9871
Status: ✅  Deploy successful!
Preview URL: https://3eb60c15.hydroflow.pages.dev
Branch Preview URL: https://mingwei-loop-contig.hydroflow.pages.dev

View logs

@MingweiSamuel MingweiSamuel changed the title fix(dfir_lang): order loop-ingress senders before the whole loop block fix(dfir_lang): order loop-ingress senders before the whole loop block, fix #3048 Jul 20, 2026
MingweiSamuel added a commit that referenced this pull request Jul 21, 2026
MingweiSamuel added a commit that referenced this pull request Jul 21, 2026
@MingweiSamuel
MingweiSamuel marked this pull request as ready for review July 21, 2026 17:26
@MingweiSamuel
MingweiSamuel requested review from a team, Copilot and shadaj July 21, 2026 17:26

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes a DFIR graph ordering bug where make_loops_contiguous could hoist loop-ingress receivers ahead of their external senders, breaking topological order (issue #3048). It does so by adding loop-ingress ordering constraints into the existing predecessor set that feeds the unified toposort, and adds regression coverage plus a defensive post-condition check.

Changes:

  • Add loop-ingress ordering constraints during subgraph union-find/toposort construction to keep gathered loop blocks topologically valid.
  • Add a defensive post-make_loops_contiguous topo validation check (plus a reusable validate_topo_sort helper).
  • Add regression tests in both dfir_lang (unit) and dfir_rs (end-to-end surface syntax).

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 4 comments.

File Description
dfir_rs/tests/surface_loop.rs Adds an end-to-end regression test covering forward-referenced loop ingress sources.
dfir_lang/src/graph/meta_graph.rs Exposes loop member nodes via a new loop_nodes(loop_id) accessor.
dfir_lang/src/graph/graph_algorithms.rs Adds validate_topo_sort utility and unit tests for it.
dfir_lang/src/graph/flat_to_partitioned.rs Adds loop-ingress ordering constraints and a post-contiguity topo validation check; includes regression test.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread dfir_lang/src/graph/flat_to_partitioned.rs
Comment thread dfir_lang/src/graph/flat_to_partitioned.rs
Comment thread dfir_lang/src/graph/flat_to_partitioned.rs Outdated
Comment thread dfir_lang/src/graph/flat_to_partitioned.rs Outdated
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
MingweiSamuel added a commit that referenced this pull request Jul 21, 2026
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
PR: #3049
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants