Be more liberal in allowing for index merges

[ben-thul]

Attempts to address this issue.

[ben-thul]

Here's a transcript of my "vibe debugging" session with Claude

---

MySQL Index Merge Sort Union Regression Analysis

Problem Statement

A performance regression was identified between MySQL 8.0.34 and 8.4.5 where queries that previously used index_merge_sort_union optimization no longer consider this strategy, even when explicitly hinted with /*+ INDEX_MERGE(...) */.

Example Query:

SELECT * FROM myTable WHERE col1 = 1 AND (col2 = 2 OR col3 = 3)

In MySQL 8.0.34, this would use index merge optimization. In 8.4.5, it falls back to less efficient single-index strategies.

Investigation Summary

Key Findings

1. Hypergraph optimizer is not the cause - Verified that hypergraph_optimizer=off and index_merge_sort_union=on
2. INDEX_MERGE hints don't work - Even explicit hints fail, indicating broken functionality rather than cost-based selection issues
3. Root cause identified - Logic replacement with failing assertions in the join optimizer

Root Cause Analysis

The regression is caused by changes in sql/join_optimizer/join_optimizer.cc where functional logic was replaced with assertions that fail for common query patterns.

MySQL 8.0.34 (Working):

// Set inexact flag based on conditions
merge.inexact = (new_tree->merges.size() > 1);
merge.inexact |= !new_tree->keys_map.is_clear_all();

MySQL 8.4.5 (Broken):

// Replaced logic with assertions that can FAIL
assert(merge.inexact || new_tree->merges.size() == 1);
assert(merge.inexact || new_tree->keys_map.is_clear_all());

When These Assertions Fail

Condition 1: new_tree->merges.size() > 1

Occurs with complex OR conditions creating multiple merge candidates:

SELECT * FROM table WHERE 
  (col1 = 'A' AND col2 = 'B') OR 
  (col1 = 'C' AND col2 = 'D') OR
  (col1 = 'E' AND col3 = 'F');

Condition 2: !new_tree->keys_map.is_clear_all()

Occurs when predicates can use both regular range scans AND index merges:

SELECT * FROM table WHERE 
  col1 BETWEEN 100 AND 200 AND  -- Can use range scan on col1
  col2 = 'value' AND             -- Can use range scan on col2  
  col3 IN (1,2,3);               -- Can use range scan on col3

User's Specific Case

The query SELECT * FROM myTable WHERE col1 = 1 AND (col2 = 2 OR col3 = 3) triggers Condition 2 because:

• Individual indexes can be used on col1, col2, or col3 (making keys_map non-empty)
• Index merges are possible between col1+col2 and col1+col3
• The inexact flag is not properly set by earlier logic
• The assertion fails and index merge consideration is abandoned

Related Commits

Key commits between versions that contributed to this issue:

1. WL#14808: Row-ID ordered scans in the hypergraph optimizer [3/9, needed_fields] (81e4576e5c3)

   • Removed needed_fields parameter handling
   • Changed cost calculation methodology

2. Logic changes in join optimizer

   • Replaced functional inexact flag setting with failing assertions
   • Changed from permissive to strict validation

3. Multiple "wrong result" fixes related to inexact handling suggest the 8.4.5 approach had correctness issues

Risk Assessment for Fix

Proposed Solution

Revert to the 8.0.34 behavior:

// Revert to 8.0.34 safe behavior  
merge.inexact = new_tree->inexact;
merge.inexact |= (new_tree->merges.size() > 1);
merge.inexact |= !new_tree->keys_map.is_clear_all();

Risk Analysis: LOW RISK ✅

Why the reversion is safe:

1. Conservative Approach: 8.0.34 behavior over-sets the inexact flag rather than under-setting it

   • Over-setting = Extra filtering (slower but correct results)
   • Under-setting = Missing filtering (faster but wrong results)

2. Correctness Priority: The inexact flag controls whether additional filtering is applied after index merge execution. Over-filtering cannot produce wrong results.

3. Performance vs Correctness Trade-off:

   • 8.0.34: Slightly slower (extra filtering) but always correct
   • 8.4.5: Potentially faster but completely broken (no index merge at all)

4. Well-tested Logic: 8.0.34 behavior was in production for years and is known to be stable

What the inexact Flag Controls

From code documentation:

// If true, the index merge does not faithfully represent the entire
// predicate (it could return more rows), and needs to be re-checked
// with a filter.
bool inexact;

When inexact = true, MySQL adds an additional filter step to ensure correctness. This is much faster than falling back to single-index scans.

Recommendation

PROCEED WITH REVERSION - The fix is low-risk because:

1. It fixes a clear regression without introducing new logic
2. It uses conservative, well-tested code from a stable release
3. It prioritizes correctness over micro-optimizations
4. The performance cost is minimal compared to the current broken state

This change will restore index_merge_sort_union functionality for affected query patterns while maintaining result correctness through conservative filtering.

Technical Details

Files Affected

• sql/join_optimizer/join_optimizer.cc - Main logic change needed
• Assertions at lines ~1500 and ~1506 need to be replaced with assignment logic

Code Location

The problematic assertions are in the FindIndexRangeScans() method in the join optimizer, specifically in the section that processes PossibleIndexMerge structures.

Testing

Queries with patterns like col1 = value AND (col2 = value OR col3 = value) should be tested to verify index merge optimization is properly selected after the fix.