JIT: regression in compare simplification rewriting `~x op k` without inverting the comparison

[AndyAyersMS]

Note

This issue was authored by GitHub Copilot CLI on @AndyAyersMS's machine,
based on a bug surfaced by an experimental in-development fuzzer.
The C# repro and disassembly are verified; the hypothesis about the
offending phase is informed speculation.

Repro

using System;
using System.Runtime.CompilerServices;

public static class Program
{
    private static volatile int Input_p0 = unchecked((int)0x800335C5);
    private static volatile int Input_p1 = 0;
    private static volatile int Sink;

    [MethodImpl(MethodImplOptions.NoInlining)]
    public static int Func(int p0, int p1)
    {
        unchecked
        {
            int v1 = unchecked((int)0x000335C5) & p1;
            int v3 = (v1 != p0) ? 1 : 0;
            if (v3 == 0) return 99;
            int v4 = ~v3;
            return (v4 >= -1) ? 1 : 0;
        }
    }
    public static int Main()
    {
        Console.WriteLine($"{(uint)Func(Input_p0, Input_p1):X8}");
        return 0;
    }
}

Manual trace: p0 = 0x800335C5, p1 = 0 ⇒ v1 = 0, v3 = (0 != p0) ? 1 : 0 = 1, branch not taken, v4 = ~1 = -2, (-2 >= -1) ? 1 : 0 = 0. Expected: 0.

Observed

Configuration	Output	Correct?
DOTNET_JITMinOpts=1	00000000	✅
DOTNET_TieredCompilation=1 DOTNET_TC_QuickJitForLoops=1 (Tier1 via tiering)	00000000	✅
DOTNET_TieredCompilation=0 (FullOpts direct)	00000001	❌
Shipping dotnet 11.0.100-preview.5.26227.104 (any config)	00000000	✅

Repros on dotnet/runtime at HEAD a8b2c92ce21 (2026-06-05) on osx-arm64 Checked. Does not repro on preview-5 — appears to be a regression introduced since then.

Disassembly (osx-arm64 Checked, DOTNET_TieredCompilation=0)

; Assembly listing for method Program:Func(int,int):int (FullOpts)
G_M26094_IG02:
    movz    w2, #0x35C5
    movk    w2, #3 LSL #16     ;; w2 = 0x000335C5
    and     w1, w1, w2         ;; v1 = 0x335C5 & p1
    cmp     w1, w0             ;; (v1 == p0)?
    cset    x0, ne             ;; v3 = (v1 != p0) ? 1 : 0
    mov     w1, #1
    mov     w2, #99
    cmp     w0, #0             ;; v3 == 0?
    csel    w0, w1, w2, ne     ;; return (v3 != 0) ? 1 : 99      <-- WRONG

The whole post-branch return has been folded to (v3 != 0) ? 1 : 99. The correct fold would be (v3 != 0) ? 0 : 99 — i.e., the THEN constant should be 0, not 1.

For comparison, MinOpts emits the actual mvn/cmn/cset ge sequence and returns the correct value.

Hypothesis

After if (v3 == 0) return 99, the JIT knows v3 ∈ {1} (since v3 is a compare result with range {0, 1} and the branch narrowed it). It then evaluates (~v3 >= -1):

• Correct: (~v3 >= -1) ⇔ (v3 <= 0) (complementing both sides reverses the comparison). With v3 ∈ {1} this is false, so the THEN constant is 0.
• What the JIT seems to do: (~v3 >= -1) ⇔ (v3 >= 0) (forgot to invert the comparison direction). With v3 ∈ {1} (or any non-negative range) this is true, so the THEN constant is 1.

Looks like a peephole / fgOptimizeRelationalComparisonWithCasts-style rewrite of ~x op k to x op ~k that missed reversing the comparison sign for ordered (non-EQ/NE) compares.

Candidate offending PRs

All merged between preview-5 and HEAD a8b2c92ce21:

• JIT: Enable fgOptimizeRelationalComparisonWithCasts for EQ/NE #128091 — JIT: Enable fgOptimizeRelationalComparisonWithCasts for EQ/NE (2026-06-01)
• JIT: Unify post-order cmp transforms into fgOptimizeRelationalComparison #127883 — JIT: Unify post-order cmp transforms into fgOptimizeRelationalCompari… (2026-05-21)
• Update fgMorphSmpOp to canonicalize commutative and compare ops at the start of POST-ORDER #127661 — Update fgMorphSmpOp to canonicalize commutative and compare ops… (2026-05-02)
• JIT: relop simplification during redundant branch opts #127181 — JIT: relop simplification during redundant branch opts (2026-04-24)

How it was found

ReifyCs (a from-scratch implementation of the PLDI 2026 semantic-reification
technique adapted to C#) ran a 500-trial, 4-type, 7-config differential
campaign and reported exactly one mismatch. Manual reduction took the
generated method down to the 5-line body above.

[dotnet-policy-service]

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

[AndyAyersMS]

Note

This comment was authored by GitHub Copilot CLI on @AndyAyersMS's machine,
based on continued fuzzing with the same in-development ReifyCs fuzzer
that surfaced the original issue. The C# repro is verified.

Continued fuzzing surfaced a second instance of the same peephole bug, with two new pieces of evidence:

1. The RHS of the broken comparison is a symbolic expression (-v1), not a literal -1. So the buggy rewrite isn't literal-specific — it's a general ~x op y → x op ~y (without inverting the comparison direction).
2. Trigger conditions differ from the original: this one fails in tier0_min (because the function contains a loop and TC_QuickJitForLoops=0 routes loopy methods straight to FullOpts) but passes in tier1_pgo (the profile-guided Tier1 codepath avoids whatever sequence triggers the buggy peephole).

Repro

using System;
using System.Runtime.CompilerServices;

public static class Program
{
    private static volatile int Input_p0 = unchecked((int)0x80000000);
    private static volatile int Input_p1 = unchecked((int)0xFFF76B55);

    [MethodImpl(MethodImplOptions.NoInlining)]
    public static int Fn(int p0, int p1)
    {
        unchecked
        {
            int v1 = 0, v2 = 0, v3 = 0, v4 = 0, v5 = 0, v6 = 0, v8 = 0, v9 = 0, v11 = 0, v13 = 0, v16 = 0, v20 = 0, v31 = 0;
            b0:
                v1 = (p1 >= unchecked((int)0xFFF76B55)) ? 1 : 0;
                v2 = -v1;
                v3 = ~v1;
                v4 = (v3 < v2) ? 1 : 0;     // ~v1 < -v1 is ALWAYS TRUE for any v1
                if (v4 != 0) goto b1;
                goto b7;
            b1:
                v5 = v3 + v3;
                v6 = p0 & unchecked((int)0xFFFFFFFF);
                v8 = (v8 >= v6) ? 1 : 0;
                goto b2;
            b2:
                v9 = v5 - v3;
                v11 = (p0 >= v1) ? 1 : 0;
                if (v11 != 0) goto b4;
                goto b5;
            b4:
                v20 = (v16 >= v9) ? 1 : 0;
                if (v20 != 0) goto b7;
                goto b2;
            b5:
                return 0;
            b7:
                v31 = ~v13;
                return v31;
        }
    }
    public static int Main()
    {
        Console.WriteLine($"{(uint)Fn(Input_p0, Input_p1):X8}");
        return 0;
    }
}

The buggy expression chain at b0:

v1 = (p1 >= 0xFFF76B55) ? 1 : 0;   // v1 = 1 (p1 = -562347, lit = -562347)
v2 = -v1;                           // v2 = -1
v3 = ~v1;                           // v3 = -2
v4 = (v3 < v2) ? 1 : 0;             // -2 < -1 ⇒ true ⇒ v4 should be 1

~x < -x is always true for any x (~x = -x - 1 < -x). With the buggy peephole ~x < y → x < ~y (no direction flip), the JIT rewrites this to x < ~(-x) = x < (x - 1) which is always false — so v4 = 0, the goto b7 branch is taken, and Fn returns ~0 = -1 instead of 0.

Observed

Configuration	Output	Correct?
DOTNET_TieredCompilation=1 DOTNET_TC_QuickJitForLoops=0 (loops bypass Tier0, go straight to FullOpts)	FFFFFFFF	❌
DOTNET_TieredCompilation=0 (FullOpts direct)	FFFFFFFF	❌
DOTNET_TieredCompilation=1 DOTNET_TC_QuickJitForLoops=1 (no warmup, stays Tier0)	00000000	✅
DOTNET_TieredCompilation=1 DOTNET_TC_QuickJitForLoops=1 + 100 warmup calls (PGO Tier1)	00000000	✅

Repros on dotnet/runtime at HEAD a8b2c92ce21 (2026-06-05) on osx-arm64 Checked.

I haven't reduced this further because the auto-reducer stopped at 11 stmts — removing any of the surrounding blocks/stmts loses the bug, suggesting the trigger needs the specific surrounding redundant-branch shape (which is consistent with #127181 being one of the candidate offending PRs).

[AndyAyersMS]

Note

This comment was authored by GitHub Copilot CLI on @AndyAyersMS's machine.

Bisect result: the first bad commit is 19f4dfc70e7 (PR #127181 — "JIT: relop simplification during redundant branch opts"), merged 2026-04-24. This was one of the four candidate PRs in the original report; the bisect now confirms it as the culprit.

Bisect details

• Range: a28fe52ce3f (preview-4 branch base, GOOD) → e6f711f4669 (preview-5 branch base, BAD) = 440 commits
• Steps: 9 build-and-test iterations
• Build time per step: ~1.5 min (./build.sh -s clr.runtime+clr.corelib -rc checked, copying libclrjit.dylib + libcoreclr.dylib + System.Private.CoreLib.dll into the existing Core_Root)
• Test: the 5-line minimized repro from the issue body, no warmup, DOTNET_TieredCompilation=0. GOOD = 00000000, BAD = 00000001.

af478f70bf3 GOOD  Normalize native compiler prop to CppCompilerAndLinker (#127371)
19f4dfc70e7 BAD   JIT: relop simplification during redundant branch opts (#127181)  <-- first bad

Mapping to the hypothesis

The original issue's hypothesis was a peephole that rewrites ~x op k → x op ~k without inverting the comparison direction. #127181's commit message describes the new optimization as:

Extend that optimization to handle some cases where pB does not imply pA, by forming (AND pB pA) (in VN space) and seeing if we can simplify it to a relop over the same operands as pB, or to a constant.

So the buggy fold lives in the new value-number-space (AND pB pA) simplification added in src/coreclr/jit/valuenum.cpp (or its caller in redundantbranchopts.cpp) — the rewrite that simplifies the conjoined relop is missing the comparison-direction flip when one of the operands is the result of ~x. The PR's added regression test (RedundantBranchSimplify.cs) does not cover this ~x op k shape.

How it was bisected

Found by ReifyCs, the same fuzzer that surfaced the original bug. The bisect itself was driven by GitHub Copilot CLI running on @AndyAyersMS's laptop using a tight build-test loop (clr.runtime+clr.corelib subset rebuild only, ~1.5 min per step). Full transcript and the repro program are at the corresponding ReifyCs finding directory.

[AndyAyersMS]

Note

This comment was authored by GitHub Copilot CLI on @AndyAyersMS's machine.

Diagnosis

The buggy rewrite is at src/coreclr/jit/valuenum.cpp lines 2578-2583 (added in #127181):

else if (func == VNF_NOT)
{
    VNFuncApp funcApp;
    if (GetVNFunc(arg0VN, &funcApp))
    {
        // NOT(NOT(x)) ==> x
        //
        if (funcApp.FuncIs(VNF_NOT))
        {
            *resultVN = funcApp.GetArg(0);
        }
        // NOT(relop(x,y)) ==> Reverse(relop)(x,y)            <-- WRONG
        //
        else if (VNFuncIsComparison(funcApp.GetFunc()))
        {
            *resultVN = GetRelatedRelop(arg0VN, VN_RELATION_KIND::VRK_Reverse);
        }
    }
}

VNF_NOT corresponds to GT_NOT, which is bitwise complement (~, lowered to mvn on arm64 / not on x64) — not logical negation (!). For a relop's int result in {0, 1}:

Expression	Possible values
(x < y)	0 or 1
~(x < y) (bitwise complement of the above)	-1 or -2
(x >= y) (the reversed relop)	0 or 1

So ~(x < y) and (x >= y) are NOT value-equivalent — they don't even share any bit pattern in common. The VN-equivalence is incorrect, and downstream uses that compare the complemented value arithmetically (e.g. ~(x < y) >= -1 in the original repro, or ~(x < y) < -1 in the seed-3001017 repro) get folded as if the complemented value were a 0/1 result.

This pattern is what made both the original repro and the seed 3001017 variant fail: both chain ~ over a 0/1 Compare result, then compare arithmetically against -1 (literal or -v).

Proposed fix

Remove the NOT(relop) → Reverse(relop) case entirely — the optimization is unsalvageable as a pure VN-equivalence because ~ of a 0/1 produces values outside {0, 1}. The NOT(NOT(x)) → x case above it is correct and should be kept.

         else if (func == VNF_NOT)
         {
             VNFuncApp funcApp;
             if (GetVNFunc(arg0VN, &funcApp))
             {
                 // NOT(NOT(x)) ==> x
                 //
                 if (funcApp.FuncIs(VNF_NOT))
                 {
                     *resultVN = funcApp.GetArg(0);
                 }
-                // NOT(relop(x,y)) ==> Reverse(relop)(x,y)
-                //
-                else if (VNFuncIsComparison(funcApp.GetFunc()))
-                {
-                    *resultVN = GetRelatedRelop(arg0VN, VN_RELATION_KIND::VRK_Reverse);
-                }
             }
         }

If preserving the optimization is desirable for some bool-typed code path, the rewrite would need to be guarded so that it only fires when the JIT can prove all downstream uses of the complemented value treat it as truthy/falsy (i.e., the ~ is followed only by a non-zero check, never by arithmetic). Detecting that reliably is non-trivial, so removal is the simplest correct fix.

Verification

Applied the above one-line removal locally, rebuilt JIT (./build.sh -s clr.jit -c checked, ~10s), and:

• Original repro (Func(0x800335C5, 0) from the issue body): now 00000000 ✅
• seed 3001017 variant: now 00000000 ✅
• 100-trial ReifyCs smoke campaign on the previously-bug-productive seed range: 0 mismatches across all 7 JIT configs (was 1 mismatch consistently) ✅

I haven't run the full JIT test suite or measured the diff impact — leaving that to whoever picks up the actual fix.