Symptom
tests/models/test_glm52_fullparam_frozen_trunk_backward.py::test_cuda_frozen_bank_value_bytes_identical_with_and_without_grad_engagement fails in the GPU models CI shard (first surfaced on #82, which made sglang importable in the trainer env so this sglang-gated test runs in CI at all). It passes standalone and within its own file.
Minimal reproducer (H100, torch 2.11.0, sgl-kernel 0.4.5, triton 3.6.0)
Run in one process, in this order:
tests/models/test_batch_invariance_dense.py::test_dense_batch_composition_invariance_policytests/models/test_glm52_exact_routed_experts_qlora.py::test_routed_experts_literal_sampler_and_gradient_policy- the frozen-bank byte test
After steps 1–2, Glm52NativeBlockFP8Experts forward (_sglang_ep_native_routed_value → fused_experts_impl_strided → sglang fused_experts_impl, topk=1, EP-native local_ids with one -1 sentinel, routed_scaling_factor=1.5) returns structurally wrong and run-to-run nondeterministic outputs for bit-identical inputs: clean row norms [0.268, 0.536, 0.144, 0.287, 0.107] become [0.0, 0.0, 0.761, 0.761, 0.025] (± garbage up to 1e37 in some runs).
Evidence trail (all verified on H100)
Instrumented every Python-visible stage boundary, clean vs polluted process, identical fixture inputs:
moe_align_block_size: inputs identical; sorted_ids/num_tokens_post_padded identical; expert_ids differs only in its uninitialized tail slot (block 5 of 5, beyond post-pad; torch.empty residue: 0x08080808 fresh page vs 0x3F800000 freed-float) — forcing that slot to -1 does not fix the failure.- GEMM1 (
invoke_fused_moe_kernel): input and output row norms bit-identical.
- Activation:
intermediate_cache2 differs only in the filtered row (uninitialized by design).
- GEMM2: output (
intermediate_cache3) row norms bit-identical.
- Final returned tensor: differs. The divergence enters between the last kernel invoke and the returned tensor (combine stage).
Ruled out empirically: torch backend flags (tf32/reduced-precision/deterministic — unchanged), xorl batch-invariant mode leak (off), bi_families_v2 selection (None), sglang exec-context (deterministic inference enabled in both), triton in-process JIT caches (cleared, no change), _down_moe_use_tma (forced off, no change), CUDA allocator state (empty_cache(), no change; NaN-priming a clean process does not reproduce), sgl_kernel.moe_sum_reduce primitive on fixed input (identical clean vs polluted).
Where this points
Persistent library state inside the pinned sglang fused-MoE stack's combine/epilogue path (sgl_kernel 0.4.5 + the vendored fused_moe.py orchestration), toggled by whatever the two polluter tests initialize. Needs debugging at the compiled-kernel level in xorl-sglang.
Interim state
The test is quarantined with xfail(strict=False) referencing this issue (PR #82). The frozen-bank byte contract itself holds when the process history is clean — verified 20/20 deterministic standalone runs.
Symptom
tests/models/test_glm52_fullparam_frozen_trunk_backward.py::test_cuda_frozen_bank_value_bytes_identical_with_and_without_grad_engagementfails in the GPUmodelsCI shard (first surfaced on #82, which made sglang importable in the trainer env so this sglang-gated test runs in CI at all). It passes standalone and within its own file.Minimal reproducer (H100, torch 2.11.0, sgl-kernel 0.4.5, triton 3.6.0)
Run in one process, in this order:
tests/models/test_batch_invariance_dense.py::test_dense_batch_composition_invariance_policytests/models/test_glm52_exact_routed_experts_qlora.py::test_routed_experts_literal_sampler_and_gradient_policyAfter steps 1–2,
Glm52NativeBlockFP8Expertsforward (_sglang_ep_native_routed_value→fused_experts_impl_strided→ sglangfused_experts_impl, topk=1, EP-native local_ids with one-1sentinel,routed_scaling_factor=1.5) returns structurally wrong and run-to-run nondeterministic outputs for bit-identical inputs: clean row norms[0.268, 0.536, 0.144, 0.287, 0.107]become[0.0, 0.0, 0.761, 0.761, 0.025](± garbage up to 1e37 in some runs).Evidence trail (all verified on H100)
Instrumented every Python-visible stage boundary, clean vs polluted process, identical fixture inputs:
moe_align_block_size: inputs identical;sorted_ids/num_tokens_post_paddedidentical;expert_idsdiffers only in its uninitialized tail slot (block 5 of 5, beyond post-pad;torch.emptyresidue:0x08080808fresh page vs0x3F800000freed-float) — forcing that slot to-1does not fix the failure.invoke_fused_moe_kernel): input and output row norms bit-identical.intermediate_cache2differs only in the filtered row (uninitialized by design).intermediate_cache3) row norms bit-identical.Ruled out empirically: torch backend flags (tf32/reduced-precision/deterministic — unchanged), xorl batch-invariant mode leak (off),
bi_families_v2selection (None), sglang exec-context (deterministic inference enabled in both), triton in-process JIT caches (cleared, no change),_down_moe_use_tma(forced off, no change), CUDA allocator state (empty_cache(), no change; NaN-priming a clean process does not reproduce),sgl_kernel.moe_sum_reduceprimitive on fixed input (identical clean vs polluted).Where this points
Persistent library state inside the pinned sglang fused-MoE stack's combine/epilogue path (sgl_kernel 0.4.5 + the vendored
fused_moe.pyorchestration), toggled by whatever the two polluter tests initialize. Needs debugging at the compiled-kernel level in xorl-sglang.Interim state
The test is quarantined with
xfail(strict=False)referencing this issue (PR #82). The frozen-bank byte contract itself holds when the process history is clean — verified 20/20 deterministic standalone runs.