feat: extend Leiden/Louvain with multi-label graphs and stable incremental updates#712
Draft
BingqingLyu wants to merge 23 commits into
Draft
feat: extend Leiden/Louvain with multi-label graphs and stable incremental updates#712BingqingLyu wants to merge 23 commits into
BingqingLyu wants to merge 23 commits into
Conversation
Add two new GDS algorithms that support: - Multiple vertex labels and edge triplets (heterogeneous subgraphs) - Optional initial_community_property for incremental/warm-start community detection - Fixed seed (42) for deterministic results New files: - include/multi_label_leiden.h, include/impl/multi_label_leiden_impl.h - src/multi_label_leiden.cc, src/impl/multi_label_leiden_impl.cc - include/multi_label_louvain.h, include/impl/multi_label_louvain_impl.h - src/multi_label_louvain.cc, src/impl/multi_label_louvain_impl.cc Modified: - gds_algo_function_collection.h (include new headers) - gds_algo_extension.cc (register new functions) - tools/python_bind/tests/test_gds.py (add test cases) Existing algorithms (leiden, louvain, wcc, etc.) are unchanged.
When vertex_labels.size()==1 && edge_triplets.size()==1 (simple graph), automatically fall back to parallel preprocessing and cached views, eliminating performance overhead vs standard leiden/louvain: - Parallel degree/edge-count/modularity computation via ParallelUtils - Pre-cached oe_view/ie_view in local_moving_phase and refine (leiden) / one_level (louvain) to avoid repeated view creation - Direct vid access (base_offset=0 => gid==vid) skipping global ID mapping This allows multi_label_leiden/louvain to fully replace standard leiden/louvain on simple graphs with zero extra cost while retaining incremental (initial_community_property) and multi-label capabilities.
…vain When initial_community_property is provided, the sink() phase now inherits old community IDs via majority vote instead of reassigning sequential 0-based IDs: - Store initial community assignments in initial_community_ array - For each new community after compute(), find the old community ID that the majority of its members belonged to (majority vote) - Larger communities get priority in ID assignment to avoid conflicts - Unmatched new communities receive fresh IDs starting from max+1 This ensures: - Stable case (no movement): community IDs are identical to input - Changed case: communities inherit their closest ancestor's ID, only truly new communities get new IDs - Zero extra cost when initial_community_property is not used
Make CALL leiden(...) and CALL louvain(...) reuse the MultiLabelLeiden /
MultiLabelLouvain implementations, keeping the public interface unchanged
while gaining multi-label graph support and initial_community_property
warm-start capability.
Changes:
- leiden.cc / louvain.cc: parse multiple vertex labels and edge triplets
(drop the simple-graph restriction), add initial_community_property
option, keep directed option for signature compatibility (ignored, same
as before), and construct MultiLabel* in exec().
- Delete the now-unused standalone implementations: leiden_impl.{h,cc}
and louvain_impl.{h,cc}.
- Existing queries need zero changes; single-label single-triplet graphs
automatically take the in-impl simple-graph fast path (no perf loss).
Tests: add leiden/louvain multi-edge and incremental alias tests.
All 47 test_gds.py tests pass.
…iden/louvain
Following the alias unification, remove the now-redundant public wrappers
and unify all naming under leiden/louvain:
- Remove public functions multi_label_leiden / multi_label_louvain
(delete their .cc/.h and registrations); leiden/louvain already carry
the full capability (multi-label + initial_community_property).
- Rename implementation files and classes to match the public names:
impl/multi_label_leiden_impl.{h,cc} -> impl/leiden_impl.{h,cc} (MultiLabelLeiden -> Leiden)
impl/multi_label_louvain_impl.{h,cc} -> impl/louvain_impl.{h,cc} (MultiLabelLouvain -> Louvain)
- Update leiden.cc / louvain.cc includes and construction accordingly.
- Update gds_algo_function_collection.h and gds_algo_extension.cc.
- test_gds.py: switch all CALL sites to leiden/louvain.
Algorithm logic is unchanged (fast path, incremental, stable IDs).
Build passes; all 47 test_gds.py tests pass.
… tests Critical fixes: - Validate initial_community_property values (non-negative, within array_size) to prevent heap buffer overflow; fallback to singleton on invalid values - Replace fixed-size sc_to_new[64] stack array with unordered_map to prevent stack buffer overflow in refine() when sub-community count exceeds 64 Performance & safety: - Pre-fetch CsrView once per triplet in generic path (eliminates O(V*T) repeated construction in hot loops of compute/local_moving_phase/refine) - Replace concurrent label_to_index_[].operator[] with pre-computed global_to_label_idx_ array and triplet_src_base_/dst_base_ arrays - Parallel stot_ init for non-warm-start simple-graph path - Defensive is_simple_graph_ label consistency check - Fix max_old_id when no valid old community exists (next_fresh starts at 0) Tests: - Add test_leiden/louvain_two_vertex_labels (multi-vertex-label + multi-edge) - Add test_leiden_incremental_data_changed (majority-vote ID preservation) - Add test_leiden_warmstart_multi_edge (warm-start stability on multi-edge) - Remove 4 duplicate test function definitions - Rename test_multi_label_leiden_incremental -> test_leiden_deterministic_simple Restore Apache-2.0 license headers in louvain_impl.h/cc and leiden_impl.cc
Upstream moved ValueColumnBuilder, MSVertexColumnBuilder, LabelTriplet
from neug::execution:: to neug:: namespace, and relocated headers
from neug/execution/common/columns/ to neug/common/columns/.
- Update include paths in leiden_impl.{h,cc} and louvain_impl.{h,cc}
- Replace execution::LabelTriplet with LabelTriplet
- Replace execution::MSVertexColumnBuilder/ValueColumnBuilder with
unqualified names (resolves via neug:: namespace)
BingqingLyu
changed the title
…/louvain - Add initial_community_property option to both Louvain and Leiden - Add multi-label graph support description and examples - Add incremental warm-start usage examples - Update Note, Limitations, and Algorithm Summary sections
BingqingLyu
force-pushed
the
feature/multi-label-leiden-louvain
branch
from
July 9, 2026 07:50
6e04f3e to
0323838
Compare
Replace placeholder comments with actual ALTER TABLE + SET statements showing how to persist community IDs between runs.
BingqingLyu
force-pushed
the
feature/multi-label-leiden-louvain
branch
from
July 9, 2026 09:02
0323838 to
a30d94b
Compare
Contributor
There was a problem hiding this comment.
Pull request overview
Note
Copilot couldn't run its full agentic review because no GitHub Actions runner was available. Make sure your repository has a runner available to run Copilot's review, or add a copilot-setup-steps.yml file specifying one with the runs-on attribute. See the docs for more details.
Extends Leiden/Louvain community detection to support heterogeneous (multi-vertex-label / multi-edge-triplet) projected graphs and adds an incremental warm-start option that preserves stable community IDs across runs.
Changes:
- Updated Cypher functions to accept multi-label/multi-triplet subgraphs and added
initial_community_propertyoption. - Reworked Leiden/Louvain implementations to handle global vertex indexing, a simple-graph fast path, and majority-vote remapping for stable community IDs.
- Added Python binding tests for multi-label graphs, determinism, and incremental warm-start behavior; updated docs to reflect new capabilities/options.
Reviewed changes
Copilot reviewed 8 out of 10 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| tools/python_bind/tests/test_gds.py | Adds tests for multi-label graphs, multi-edge graphs, determinism, and warm-start stability. |
| extension/gds/src/louvain.cc | Extends input parsing to accept multiple labels/triplets and passes through initial_community_property. |
| extension/gds/src/leiden.cc | Extends input parsing to accept multiple labels/triplets and passes through initial_community_property. |
| extension/gds/src/impl/louvain_impl.cc | Implements multi-label/multi-triplet support, simple-graph fast path, and stable ID inheritance in Louvain. |
| extension/gds/src/impl/leiden_impl.cc | Implements multi-label/multi-triplet support, simple-graph fast path, and stable ID inheritance in Leiden. |
| extension/gds/src/gds_algo_extension.cc | Minor formatting/whitespace fix. |
| extension/gds/include/impl/louvain_impl.h | Updates Louvain API and internal state for multi-label graphs and warm-start support. |
| extension/gds/include/impl/leiden_impl.h | Updates Leiden API and internal state for multi-label graphs and warm-start support. |
| doc/source/extensions/load_gds.md | Documents multi-label support and initial_community_property option for Leiden/Louvain. |
| .gitignore | Minor formatting tweak. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+204
to
+212
| // Parallel stot_ init for non-warm-start; serial accumulate for warm-start | ||
| if (!initial_community_) { | ||
| ParallelUtils::parallel_for( | ||
| valid_vertices_.data(), valid_vertices_.size(), | ||
| [&](vid_t v, int /*tid*/) { stot_[v] = degree_[v]; }, num_threads_); | ||
| } else { | ||
| for (uint32_t gid : valid_vertices_) | ||
| stot_[community_[gid]] += degree_[gid]; | ||
| } |
Comment on lines
+196
to
+204
| // Parallel stot_ init for non-warm-start; serial accumulate for warm-start | ||
| if (!initial_community_) { | ||
| ParallelUtils::parallel_for( | ||
| valid_vertices_.data(), valid_vertices_.size(), | ||
| [&](vid_t v, int /*tid*/) { stot_[v] = degree_[v]; }, num_threads_); | ||
| } else { | ||
| for (uint32_t gid : valid_vertices_) | ||
| stot_[community_[gid]] += degree_[gid]; | ||
| } |
Comment on lines
+243
to
+268
| for (uint32_t gid : valid_vertices_) { | ||
| size_t li = global_to_label_idx_[gid]; | ||
| vid_t lv = global_to_vid_[gid]; | ||
| double deg = 0; | ||
| for (size_t ti : label_out_triplets_[li]) { | ||
| auto e = out_views[ti].get_edges(lv); | ||
| for (auto it = e.begin(); it != e.end(); ++it) | ||
| deg += 1.0; | ||
| } | ||
| for (size_t ti : label_in_triplets_[li]) { | ||
| auto e = in_views[ti].get_edges(lv); | ||
| for (auto it = e.begin(); it != e.end(); ++it) | ||
| deg += 1.0; | ||
| } | ||
| degree_[gid] = deg; | ||
| } | ||
| m_ = 0; | ||
| for (uint32_t gid : valid_vertices_) { | ||
| size_t li = global_to_label_idx_[gid]; | ||
| vid_t lv = global_to_vid_[gid]; | ||
| for (size_t ti : label_out_triplets_[li]) { | ||
| auto e = out_views[ti].get_edges(lv); | ||
| for (auto it = e.begin(); it != e.end(); ++it) | ||
| m_ += 1.0; | ||
| } | ||
| } |
| ``` | ||
|
|
||
| **Example (multi-label graph):** | ||
|
|
Comment on lines
+967
to
+978
| def test_multi_label_leiden_basic(tmp_path): | ||
| """multi_label_leiden on a single-label graph (backward compat).""" | ||
| with tinysnb_simple_connection(tmp_path) as conn: | ||
| rows = list( | ||
| conn.execute( | ||
| """ | ||
| CALL leiden('person_knows', {concurrency: 4}) | ||
| YIELD node, community | ||
| RETURN node.id, community; | ||
| """ | ||
| ) | ||
| ) |
Comment on lines
+1131
to
+1135
| # Step 4: Write back communities to vertex property via SET | ||
| for node_id, comm in r1_map.items(): | ||
| conn.execute( | ||
| f"MATCH (n:person) WHERE n.id = {node_id} SET n.leiden_comm = {comm};" | ||
| ) |
liulx20
reviewed
Jul 10, 2026
| ) | ||
|
|
||
|
|
||
| # ─── Multi-Label Leiden / Louvain ──────────────────────────────────────────── |
Collaborator
There was a problem hiding this comment.
Should we add a correctness test case?
1. Zero stot_ before accumulation (simple-graph + generic path in both leiden_impl and louvain_impl) — community IDs may differ from GIDs after warm-start, so stale values must be cleared. 2. Parallelize degree/m_ computation in generic path using parallel_for + per-thread local accumulators (both algorithms). 3. Fix typo in docs: studyat → studyAt (4 occurrences in load_gds.md). 4. Rename test functions/docstrings: drop multi_label_ prefix and alias references for consistency with consolidated interface. 5. Batch writeback: replace per-vertex for-loop SET with single CASE WHEN query to reduce round-trips. 6. Inline expected results in tests: each test now asserts concrete expected vertex IDs, non-negative community IDs, >= 2 communities, and non-singleton partition — no shared helper or internal modularity/edge-ratio checks exposed.
BingqingLyu
marked this pull request as draft
July 13, 2026 07:00
Add an optional output column to both Leiden and Louvain community detection algorithms. When is set and the user YIELDs , the column outputs each vertex's previous community ID (NULL for vertices without a prior assignment). Community-level change analysis (migration matrix, split/merge detection) is delegated to Cypher aggregation on previous_community x community. C++ changes: - getFunctionSet: declare previous_community (kInt64) as 3rd output column - bind: use kInt64 counter to distinguish community vs previous_community - sink: build nullable INT64 column from initial_community_ array (UINT32_MAX -> NULL); zero overhead when not YIELDed (alias = -1) Tests (7 cases in test_gds.py): - unchanged data: previous_community == community - data change: previous_community reflects old IDs - new vertex simulation: previous_community = None - backward compat: not YIELDing produces only 2 columns - Louvain versions of above - Cypher migration matrix aggregation Docs: update load_gds.md with previous_community column, delta analysis examples, and algorithm summary table.
…vain Upstream PR alibaba#692 changed call_output_columns from DataTypeId to DataType(DataTypeId). Merge our previous_community column addition with the new DataType wrapper format.
When initial_community_property is set, existing vertices are now frozen by default — their community assignments cannot change. Only new vertices (without a previous community) are clustered: assigned to existing communities or forming new ones. Add allow_relocation parameter (default: false) to opt-in the previous warm-start behavior where all vertices can be re-optimized. Changes: - leiden_impl.h/louvain_impl.h: add allow_relocation constructor param - leiden.cc/louvain.cc: parse allow_relocation option from Cypher - leiden_impl.cc: skip frozen nodes in local_moving_phase, skip refine() in freeze mode - louvain_impl.cc: skip frozen nodes in one_level()
Add a 'weight' option to both Leiden and Louvain algorithms that allows
users to specify an edge property name to use as edge weight for weighted
modularity computation.
Key changes:
- Leiden/Louvain constructors accept an optional weight_property parameter
- When set, EdgeDataAccessor reads the specified DOUBLE edge property
- All degree, total edge weight (m), and modularity calculations use the
edge weight instead of hardcoded 1.0
- Both simple-graph and multi-label code paths are covered
- When the property is missing or not specified, falls back to weight 1.0
(full backward compatibility)
The option name 'weight' follows the same convention as SSSP.
Usage example:
CALL leiden('social', {weight: 'strength', concurrency: 8})
CALL louvain('graph', {weight: 'weight'})
Add test_gds_weighted.py covering: - Louvain/Leiden with valid edge weight property - Weighted vs unweighted comparison - Missing property fallback to default weight 1.0 - Backward compatibility (no weight option)
BingqingLyu
force-pushed
the
feature/multi-label-leiden-louvain
branch
from
July 16, 2026 06:07
73b214f to
ee3abd9
Compare
The modularity computation used for convergence checking was hardcoded to γ=1.0, ignoring the resolution parameter. This caused the algorithm to stop prematurely when resolution != 1.0, since the convergence metric didn't match the actual objective function being optimized. Fix: multiply the null-model term by resolution_ in all 4 modularity computation sites (leiden/louvain × simple-graph/multi-label paths). Add resolution sensitivity tests verifying that γ=100 produces more communities than γ=0.001 on a two-cluster graph.
BingqingLyu
force-pushed
the
feature/multi-label-leiden-louvain
branch
from
July 16, 2026 09:56
e9b2d5c to
27dd98e
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related Issues
Fixes #709
What does this PR do?
Extends the
leidenandlouvaincommunity detection algorithms to support two new capabilities while keeping the existing public Cypher interface backward compatible:temp_person+personvertices,temp_knows+knowsedges) is now treated as one unified graph in a single run.initial_community_propertyoption allows seeding a run from a previously written-back community property. When data is unchanged, results are identical; when data changes, unchanged communities inherit their original IDs via majority-vote mapping.What changes in this PR?
Core algorithm rewrite (
extension/gds/src/impl/&include/impl/)Leiden/Louvainclasses now acceptvector<label_t> vertex_labelsandvector<LabelTriplet> edge_tripletsinstead of a singlelabel_t. A global vertex indexing scheme (label_base_offsets_) handles overlapping local vertex IDs across labels.is_simple_graph_), the algorithm skips multi-label indirection entirely — pre-computes a singleCsrView, parallel-initializesstot_by degree, and uses direct array indices. Zero overhead vs. the original simple-graph path.sink()performs majority-vote mapping from new clusters back to old community IDs (viainitial_community_property), so unchanged regions keep their original IDs and only changed regions receive new IDs.initial_community_propertyvalues (raw >= 0 && raw < array_size_)sc_to_newfixed-size stack array →unordered_mapto prevent overflow on high-degree graphsCsrViewvectors to avoid repeated construction in hot loopslabel_to_index_map lookups replaced with pre-computedglobal_to_label_idx_array for O(1) accessmax_old_idinitialization guarded withhas_valid_oldflagis_simple_graph_defensive check on label equalityInterface consolidation (
extension/gds/src/leiden.cc&louvain.cc)CALL leiden(...)/CALL louvain(...)withYIELD node, community) is unchanged — existing queries need no modifications.RunLeiden/RunLouvainfree functions andLeidenResult/LouvainResultstructs removed; replaced byLeiden/Louvainclasses withcompute()+sink()pattern (consistent with PageRank/WCC/SSSP).initial_community_property(string, default empty).Tests (
tools/python_bind/tests/test_gds.py)test_leiden_two_vertex_labels/test_louvain_two_vertex_labels— verify algorithms run on 2 vertex labels + 2 edge triplets.test_leiden_incremental_data_changed— add edge, verify majority-vote preserves some community IDs.test_leiden_warmstart_multi_edge— warm-start on multi-edge graph, verify >=80% stability.test_leiden_deterministic_simple— verify deterministic results on simple graph.