Technique vector clustering for beam-search diversity + production scheduling refinements

[jiannanWang]

Summary

Builds on #139 (multi-LLM beam search) to add a diversity-preserving selection layer plus the scheduling and locking refinements that the 225-worker XL preset needs.

The motivation: with #139 's P × B × M × S fanout, the beam quickly fills with kernels that target the same optimization approach. PTX dedup catches byte-equivalents but can't tell apart kernels that differ structurally yet still pursue the same strategy. Without a diversity term, the beam collapses onto a single lineage and the search plateaus. This PR adds an LLM-classifier-driven technique-vector clustering layer that selects round-robin per cluster, preserving at least one representative of each distinct optimization approach.

It also includes the scheduling/locking changes that became necessary at 225-worker scale (dynamic GPU scheduler, manager-side lock-pool decouple, bottleneck-analysis caching), and an API cleanup that collapses two redundant knobs into one.

All new features are opt-in. Clustering is gated by technique_clustering.enabled: true; the existing #139 presets do not enable it.

Example commands

Two new presets ship with the PR:

# 90-worker clustered (2 parents × 3 bottlenecks × 3 models × 5 samples + clustering)                                                                                                                                                                                                               
python examples/run_opt_manager.py \                                                                                                                                                                                                                                                                
  --kernel-dir examples/optimize_01_matvec \                                                                                                                                                                                                                                                        
  --strategy beam_search_diverse_clustered \                                                                                                                                                                                                                                                        
  --max-rounds 8                                                                                                                                                                                                                                                                                    
                                                                                                                                                                                                                                                                                                    
# 225-worker clustered XL (5 parents × 3 bottlenecks × 3 models × 5 samples + clustering, 8 GPUs × 4 workers/GPU)                                                                                                                                                                                   
python examples/run_opt_manager.py \                                   
  --kernel-dir examples/optimize_01_matvec \                       
  --strategy beam_search_diverse_clustered_xl \                        
  --max-rounds 8                 

Both presets configure models: [claude-opus-4.6, gpt-5-4, gemini-2-5-pro] and use claude-opus-4.6 as the technique classifier. Clustering off by default in existing presets.

Experiment results

Two problems, 8 rounds each (full production runs):

┌───────────────┬────────────────────────────────────────────────────┬──────────────────────────────────────────────┐                            
│    Problem    │ Without clustering (PR-1, 90 workers concentrated) │    With clustering (PR-2, 225 workers XL)    │
├───────────────┼────────────────────────────────────────────────────┼──────────────────────────────────────────────┤                            
│ matvec        │ 1.8318 ms (1.12× PyTorch eager)                    │ 1.8323 ms (1.12× eager)                      │
├───────────────┼────────────────────────────────────────────────────┼──────────────────────────────────────────────┤
│ gemma3_swiglu │ 0.2716 ms (3.10× eager, 1.14× torch.compile)       │ 0.3107 ms (2.71× eager, 1.00× torch.compile) │
└───────────────┴────────────────────────────────────────────────────┴──────────────────────────────────────────────┘                            

Notes:

• Matvec is DRAM-saturated by 90 workers — clustering doesn't add headroom past that ceiling, but doesn't hurt.
• On gemma3_swiglu, clustering enables a late-round breakthrough. R7 of the XL run produced a winner kernel whose lineage would have been culled by pure top-K in round 1; clustering preserved the slow-but-diverse parent slot. (PR-1's 90-worker concentrated still beat XL clustering on this
  problem at fixed budget — clustering helps escape plateaus but isn't always the dominant intervention. Both configurations beat torch.compile.)

Verified live on this branch with a 2-round 90-worker clustered run on matvec:

• R1: 90/90 succeeded, PTX dedup 92→92, clusters=14, best 1.8753 ms (claude-opus-4.6)
• R2: 71/90 succeeded, PTX dedup 73→17 (77% collapse), clusters=7, best 1.8612 ms (claude-opus-4.6)
• Final: 1.8612 ms = 1.10× PyTorch eager
• Vector cache hit confirmed: R2 only classified 15 new kernels (not 73) — vectors for carry-over kernels were reused from R1's JSON-DB persistence.

What is changed

• LLM technique classifier — new searching/technique_vector.py (TechniqueDefinition, load_techniques, classify_kernel, classify_many) that emits a binary 18-bit vector per kernel via one LLM call, fanned out via a thread pool.
• 18-bit technique taxonomy — new examples/configs/techniques_default.yaml listing optimization techniques (tensor_cores, persistent_kernel, autotuned_config, etc.) with prompt hints for the classifier.
• Round-robin per-cluster beam selection — new select_diverse_top_k (depth-first per cluster); wired into BeamSearchStrategy.update_with_results behind a config flag.
• ProgramEntry.technique_vector field — JSON-DB serdes so vectors persist across rounds and runs; the manager only pays the LLM cost for newly-discovered kernels.
• Per-parent bottleneck-analysis caching — manager runs the LLM-driven NCU-to-bottleneck analysis once per unique parent per round and shares the result across siblings.
• Spawn-on-free-GPU-slot dynamic scheduler — replaces static "spawn N, wait for all" with a pool topped up to workers_per_gpu × len(gpu_ids) and refilled as workers exit. New workers_per_gpu config knob.
• Manager GPU lock pool decoupled from workers'; semaphore timeout reduced 900s → 60s so a stuck worker can't block the manager.
• candidate_pool_size API simplification — single knob replaces num_top_kernels + num_expanding_parents. All preset YAMLs updated. This is the only breaking config change.
• Two new strategy presets — beam_search_diverse_clustered.yaml, beam_search_diverse_clustered_xl.yaml. Existing presets unchanged (clustering off by default).