From af9b8f599503fac8b064362dfecf4b3272ca4283 Mon Sep 17 00:00:00 2001 From: bong-water-water-bong Date: Tue, 10 Nov 2026 11:03:50 -0400 Subject: [PATCH] moe_placement: Kernighan-Lin refinement, and a held-out share beside the in-sample one (Refs #1702) Run on the report's own traces the merged tool reached 34.8 % / 34.8 % / 24.7 % one-device share (Qwen3.6 chat, code; Flash-Next), where the report measured 46.2 % / 60.0 % / 44.0 %: it cut a greedy chain in G runs and skipped the Kernighan-Lin partition the report used. - plan_layer(): refine the chain's cut with Kernighan-Lin swaps (51.1 % / 55.6 % / 52.3 % in-sample, group sizes unchanged, under 2 s per trace) - held_out_share(): the table from the first half of each layer's steps scored on the second half (22.5 % / 26.3 % / 18.9 %); printed and written into the table beside the in-sample share - --score TABLE: score an existing table on another trace (a Qwen3.6 table built from chat scores 13.5 % on code, from code 7.0 % on chat) - self-test arms for each; docs/moe-streaming.md says which share is which Co-Authored-By: Claude Fable 5.2 --- docs/moe-streaming.md | 42 +++++++- tools/moe_placement.py | 222 +++++++++++++++++++++++++++++++++++++---- 2 files changed, 243 insertions(+), 21 deletions(-) diff --git a/docs/moe-streaming.md b/docs/moe-streaming.md index 817a2a6a7..058c83725 100644 --- a/docs/moe-streaming.md +++ b/docs/moe-streaming.md @@ -687,6 +687,24 @@ Co-activating experts cluster hard enough that a **per-layer** placement is wort (share of decoded tokens whose experts all land on ONE device, i.e. how often a token can skip the wire entirely). The middle row is why this is per layer and not one permutation for the model. +**Those shares are in-sample, and a consumer will not see them.** Each was scored on the same decode steps the plan was +built from. A placement is fixed in a model while prompts change, so the number that matters is the share on steps the +plan never saw. Measured with `tools/moe_placement.py` on the same traces (G=2, 2026-11-10): + +| table built from | scored on | Qwen3.6 chat | Qwen3.6 code | Flash-Next | +|---|---|---|---|---| +| the whole trace | the same trace (in-sample) | 51.1 % | 55.6 % | 52.3 % | +| the first half of each layer's steps | the second half (held out) | **22.5 %** | **26.3 %** | **18.9 %** | +| the other prompt's trace | this trace | **7.0 %** (built from code) | **13.5 %** (built from chat) | not measured | +| contiguous id split (control) | this trace | 0.6 % | 0.5 % | 0.2 % | + +So a per-layer placement is still 15 to 50 times the contiguous control on unseen steps, but it is a 7-26 % lever, not a +44-60 % one, and it is weakest across prompts. With a few hundred decode steps and 256 or 512 experts per layer, a +partition can keep many of exactly the steps it was fitted to: a table built from chat and code together scores 50 % on +both, in-sample. The Flash-Next cross-prompt cell is empty on purpose: the only other Flash-Next decode trace on the box +shares 18 % of its rows with the fitting trace, so it is not an independent sample. Any gate for a consumer of this +table (engine#1702) should be stated on a held-out or cross-prompt share, or on an end-to-end rate. + `tools/moe_placement.py` builds that table from a `tools/moe_trace.cpp` trace: ``` @@ -694,6 +712,14 @@ tools/moe_trace … trace.tsv # the per-step, per-layer exper tools/moe_placement.py trace.tsv -g 2 -e 256 -o placement.json ``` +**The partition is Kernighan-Lin, not just a chain.** The first version of the tool cut a greedy max-affinity chain into +G runs and said that was close enough to the report's balanced Kernighan-Lin plan. Measured on the report's traces it was +not: 34.8 % / 34.8 % / 24.7 % in-sample (chat, code, Flash-Next) against 51.1 % / 55.6 % / 52.3 % once the chain's cut is +refined by Kernighan-Lin swaps (the report measured 46.2 % / 60.0 % / 44.0 % with its own implementation). A chain follows +one neighbour at a time, so it strings clusters together and the cut lands wherever the count says. The refinement swaps +equal numbers of experts between groups, so a balanced split stays balanced; inside each group the order is still a +max-affinity chain. The whole run takes under 2 s per trace. + **`-e` is the model's expert count, and it matters.** A decode sample does not fire every expert in every layer, and the default derives the layer's size from the highest id the trace happens to visit — so without `-e` the table is a placement of the *sample*. With `-e` the table covers `0..N-1`, every expert appears exactly once, and each layer records `unseen`: @@ -711,6 +737,7 @@ splitter or the streamer may consume, and an unused table changes nothing. Its s ```json { "schema": "1bit/moe-placement/1", "groups": 2, + "share_in_sample": 0.5113, "share_held_out": 0.2252, "held_out_steps": 14400, "layers": { "0": { "decode_steps": 720, "experts": 256, "experts_seen": 243, "unseen": 13, "groups": [[…ids…], […ids…]] } } } ``` @@ -718,7 +745,20 @@ splitter or the streamer may consume, and an unused table changes nothing. Its s keyed by (layer, global expert id) — the same key `tools/build_expert_placement.py` uses upstream of us, which is why the schema carries its own name: a consumer can check `schema` rather than assume this tool's version. -`--self-test` pins the property that matters on synthetic traces with a known answer: two **interleaved** clusters (ids +The two shares travel in the file, so a table found later says which number it was built on. `share_held_out` is the +table built from the first half of each layer's steps scored on the second half; the run prints both. To score a table on +a different trace (another prompt, another session), which is the closest thing to what a consumer gets: + +``` +tools/moe_placement.py other-prompt.tsv --score placement.json +``` + +It refuses a file whose `schema` is not this one, and a table that lacks a layer the trace has. + +`--self-test` pins the property that matters on synthetic traces with a known answer: a bridge pair that fires more often +than any pair inside two clusters must not pull the clusters apart (the chain alone scores 37.5 % there, the refined plan +62.5 % with the group sizes unchanged), a table built from the first half of a layer whose clusters then change must score +0 % held out, and otherwise: two **interleaved** clusters (ids `0,2,4,6` / `1,3,5,7`, so a contiguous split cannot be right by accident) must reach a 100 % one-device share under `-g 2` while the contiguous control reaches 0 %, four clusters must do it under `-g 4`, a token spanning every group must never count as one device, `G=1` / `G=n_experts` must be exact, and a **partial** sample must still place every expert in range diff --git a/tools/moe_placement.py b/tools/moe_placement.py index 5c3f92830..0f38b7ef4 100755 --- a/tools/moe_placement.py +++ b/tools/moe_placement.py @@ -17,7 +17,9 @@ Why per layer, and why decode rows only: measured on 2026-11-09 (engine#1702), a **per-layer** co-activation placement puts all of a decoded token's experts on one device for 44-60 % of tokens, against 0.2-0.6 % for a contiguous id split and -2.7-6.9 % for **one global** permutation. The global row is what fixes the shape of this file's output: one permutation is +2.7-6.9 % for **one global** permutation. Those shares are IN-SAMPLE (scored on the steps the plan was built from). On +steps the plan never saw it is 19-26 % for the second half of the same conversation and 7-14 % for a different prompt +(held_out_share() below, and `--score`), still 15-50x the contiguous split. The global row is what fixes the shape of this file's output: one permutation is worth almost nothing, so the plan is keyed by layer. The same measurement shows placement adds 5.4-6.9 % on batched prefill while deduplication gives 90-94 %, so prefill rows ('P') are excluded: placement is a decode lever and this tool does not repackage a prefill result as one. @@ -26,7 +28,8 @@ choose to consume, and an unused table changes nothing. Usage: - tools/moe_placement.py [-g GROUPS] [-o out.json] [--min-pairs N] + tools/moe_placement.py [-g GROUPS] [-e EXPERTS] [-o out.json] [--min-pairs N] + tools/moe_placement.py --score table.json tools/moe_placement.py --self-test Trace format (tools/moe_trace.cpp, Writer::flush): `steplayerP|Did,id,...` and, on 'D' rows, two further @@ -87,31 +90,105 @@ def coactivation(steps, n_experts): return counts -def plan_layer(steps, n_experts, groups): - """A balanced G-group order for one layer: greedy max-affinity chain, then a balance-preserving split. - - Deliberately not a full Kernighan-Lin: the report's plan uses KL over the same measure, and for the sizes that matter - (256 or 512 experts) the greedy chain captures the clustering the share is computed from. The test pins the property - that matters (co-activating experts end up in one group) rather than the exact partition, so a KL pass can replace this - without changing the output's meaning. - """ - counts = coactivation(steps, n_experts) - seen = {e for row in steps for e in row} +def chain_order(counts, seen): + """A greedy max-affinity chain over the experts the sample fired: the starting order, and the order inside a group.""" order = sorted(seen) if not order: return [] - # affinity e -> sum of co-activation with everything placed before it, greedily extended placed, remaining = [order[0]], set(order[1:]) while remaining: last = placed[-1] nxt = max(remaining, key=lambda e: (counts[last][e], counts[e][last], -e)) placed.append(nxt) remaining.discard(nxt) + return placed + + +# How many of the best candidates on each side a Kernighan-Lin step compares. The pair with the largest gain is almost +# always among the few nodes with the largest D on each side (Kernighan & Lin's own shortcut); the full product is +# 256 x 256 per step on a 512-expert layer and buys nothing measurable. +KL_CANDIDATES = 8 + + +def kl_refine_pair(counts, ga, gb): + """One Kernighan-Lin refinement of two equal-role groups, in place. Returns the co-activation weight it moved inside. + + The weight between two experts is how often they fired in the same step (symmetric). A pass tentatively swaps pairs, + each node at most once, always taking the best available swap even when its gain is negative, and keeps the prefix of + swaps with the largest total gain -- that is what lets it leave a local optimum a plain "swap while it helps" loop + stops in. Passes repeat until one gains nothing. Sizes never change, so a balanced split stays balanced. + """ + total = 0 + while True: + in_a = set(ga) + # D[e] = (weight to the other group) - (weight to its own group): what moving e alone would gain + D = {} + for e in ga: + row = counts[e] + D[e] = sum(row[x] for x in gb) - sum(row[x] for x in ga) + for e in gb: + row = counts[e] + D[e] = sum(row[x] for x in ga) - sum(row[x] for x in gb) + free_a, free_b = set(ga), set(gb) + swaps, gains = [], [] + while free_a and free_b: + top_a = sorted(free_a, key=lambda e: (-D[e], e))[:KL_CANDIDATES] + top_b = sorted(free_b, key=lambda e: (-D[e], e))[:KL_CANDIDATES] + gain, a, b = max((D[x] + D[y] - 2 * counts[x][y], -x, -y) for x in top_a for y in top_b) + a, b = -a, -b + swaps.append((a, b)) + gains.append(gain) + free_a.discard(a) + free_b.discard(b) + # a goes to B and b goes to A: every free node's D moves by twice its weight to the pair + ra, rb = counts[a], counts[b] + for e in free_a: + D[e] += 2 * ra[e] - 2 * rb[e] + for e in free_b: + D[e] += 2 * rb[e] - 2 * ra[e] + best, best_k, run = 0, 0, 0 + for k, g in enumerate(gains, 1): + run += g + if run > best: + best, best_k = run, k + if best_k == 0: + return total + for a, b in swaps[:best_k]: + ga[ga.index(a)] = b + gb[gb.index(b)] = a + total += best + del in_a + + +def plan_layer(steps, n_experts, groups): + """A balanced G-group order for one layer: a greedy max-affinity chain cut into G runs, refined by Kernighan-Lin. + + The chain alone is not enough, and that is measured rather than argued (engine#1702, the report's own traces, G=2): + the chain split puts all of a decoded token's experts on one device for 34.8 % (Qwen3.6 code), 34.8 % (chat) and + 24.7 % (Flash-Next) of tokens, where the report's balanced Kernighan-Lin plan measured 60.0 %, 46.2 % and 44.0 %. + A chain follows one neighbour at a time, so it strings clusters together and the cut lands wherever the count says. + The refinement swaps experts between groups to move co-activation weight inside them; group sizes do not change. + + The returned order is group by group, and inside each group it is again a max-affinity chain, so neighbours in the + file are experts that fire together -- the order a streamer's layout would want. + """ + counts = coactivation(steps, n_experts) + seen = {e for row in steps for e in row} + placed = chain_order(counts, seen) + if not placed: + return [] if groups <= 1: return placed - # balance: chunk the chain, so a group is a contiguous run of strongly-affine experts - per = (len(placed) + groups - 1) // groups - return placed[: per * groups] if len(placed) > per * groups else placed + parts = group_of_plan(placed, groups) + if len(parts) > 1: + moved = True + while moved: # every pair of groups, until a whole round moves nothing + moved = False + for i in range(len(parts)): + for j in range(i + 1, len(parts)): + if kl_refine_pair(counts, parts[i], parts[j]) > 0: + moved = True + return [e for part in parts for e in chain_order(counts, part)] def group_of_plan(order, groups): @@ -162,6 +239,34 @@ def build(trace, groups, min_pairs=1, experts=None): return layers +def held_out_share(trace, groups, min_pairs=1, experts=None): + """(share, steps): the table built from the FIRST half of each layer's decode steps, scored on the SECOND half. + + The share build() reports for its own trace is in-sample: the same steps chose the groups and are then counted. With + a few hundred steps and 256 or 512 experts a partition can keep many of exactly those steps intact, so that number is + an upper bound on what a consumer sees, not an estimate of it. Measured on engine#1702's own traces (G=2): in-sample + 51.1 % / 55.6 % / 52.3 %, held out 22.5 % / 26.3 % / 18.9 % (Qwen3.6 chat, code; Flash-Next), and a Qwen3.6 table + built from the chat prompt scores 13.5 % on the code prompt. A placement is fixed in a model while prompts change, so + the held-out figure is the one to quote. + """ + first = {layer: steps[: len(steps) // 2] for layer, steps in trace.items() if len(steps) >= 2} + second = {layer: steps[len(steps) // 2:] for layer, steps in trace.items() if len(steps) >= 2} + if not first: + return 0.0, 0 + layers = build(first, groups, min_pairs, experts) + return one_device_share({int(l): v["groups"] for l, v in layers.items()}, second) + + +def load_table(path): + """{layer: groups} from a table this tool wrote; ValueError naming what is wrong with anything else.""" + with open(path, encoding="utf-8") as f: + table = json.load(f) + if not isinstance(table, dict) or table.get("schema") != "1bit/moe-placement/1": + raise ValueError(f"{path} is not a 1bit/moe-placement/1 table (schema {table.get('schema')!r})" + if isinstance(table, dict) else f"{path} is not a placement table") + return {int(layer): entry["groups"] for layer, entry in table.get("layers", {}).items()} + + def contiguous(trace, groups): """The control: a per-layer contiguous id split, which the report measured at 0.2-0.6 %.""" plan = {} @@ -236,6 +341,40 @@ def clusters(ids_per_cluster, reps=50): except ValueError as e: check("expert 7" in str(e) and "4" in str(e), f"... refused by naming the id and the count ({str(e)[:40]}...)") + # The partition, not just the chain (engine#1702). Two clusters of four plus a bridge pair (0, 1) that fires more + # often than any pair inside a cluster: the max-affinity chain leaves cluster {0,2,4,6} for 1 at its first step and + # then walks {3,5,7}, so cutting it in half splits BOTH clusters and keeps only the bridge steps (60 of 160). The + # right plan keeps the clusters and cuts the bridge (100 of 160). Kernighan-Lin gets there by swaps of equal size. + bridged = {0: [[0, 2, 4, 6]] * 50 + [[1, 3, 5, 7]] * 50 + [[0, 1]] * 60} + b_groups = build(bridged, 2)["0"]["groups"] + bshare, bn = one_device_share({0: b_groups}, bridged) + check(abs(bshare - 100 / 160) < 1e-9, + f"a bridge pair does not pull two clusters apart: share {bshare:.1%} of {bn} (the chain alone reaches 37.5%)") + check(sorted(map(sorted, b_groups)) == [[0, 2, 4, 6], [1, 3, 5, 7]], f"... and the groups are the clusters ({b_groups})") + check([len(g) for g in b_groups] == [4, 4], "... with the group sizes unchanged by the refinement") + # refinement must never make a plan worse than the chain it started from, on a trace with no structure to find + import random + rng = random.Random(1702) + noise = {0: [rng.sample(range(32), 4) for _ in range(300)]} + counts = coactivation(noise[0], 32) + chain_parts = group_of_plan(chain_order(counts, set(range(32))), 2) + + def inside(parts): + return sum(counts[a][b] for part in parts for a in part for b in part) + + refined = build(noise, 2)["0"]["groups"] + check(inside(refined) >= inside(chain_parts) and sorted(len(g) for g in refined) == [16, 16], + f"on noise the refined plan holds at least the chain's co-activation ({inside(refined)} >= {inside(chain_parts)})") + + # In-sample is not held out. A layer whose clusters CHANGE half way: the table built from the first half is right + # for the first half and wrong for the second, and held_out_share() must report the second, not the fit. + shifted = {0: ([[0, 2, 4, 6]] * 25 + [[1, 3, 5, 7]] * 25) + ([[0, 1, 2, 3]] * 25 + [[4, 5, 6, 7]] * 25)} + hs, hn = held_out_share(shifted, 2) + check(hs == 0.0 and hn == 50, f"a table from the first half scores {hs:.0%} on a second half it never saw ({hn} steps)") + steady = {0: ([[0, 2, 4, 6]] * 25 + [[1, 3, 5, 7]] * 25) * 2} + hs2, _ = held_out_share(steady, 2) + check(hs2 == 1.0, f"... and {hs2:.0%} when the second half fires the same clusters") + # Prefill rows must be ignored: the report says placement is not a prefill lever, so a tool that counted them would # inflate its own case. import tempfile, os @@ -248,6 +387,25 @@ def clusters(ids_per_cluster, reps=50): finally: os.unlink(path) + # --score reads a table back by its schema, and refuses a file that is not one + with tempfile.NamedTemporaryFile("w", suffix=".json", delete=False) as f: + json.dump({"schema": "1bit/moe-placement/1", "groups": 2, "layers": {"0": {"groups": [[0, 2, 4, 6], [1, 3, 5, 7]]}}}, f) + good = f.name + with tempfile.NamedTemporaryFile("w", suffix=".json", delete=False) as f: + json.dump({"schema": "something/else", "layers": {}}, f) + bad = f.name + try: + rshare, _ = one_device_share(load_table(good), two) + check(rshare == 1.0, f"a table read back from disk scores the trace it fits ({rshare:.0%})") + try: + load_table(bad) + check(False, "a file with another schema is refused") + except ValueError as e: + check("something/else" in str(e), "a file with another schema is refused, naming the schema it has") + finally: + os.unlink(good) + os.unlink(bad) + print(f"\nmoe_placement self-test: {fails} failure(s)") return 0 if fails == 0 else 1 @@ -261,6 +419,9 @@ def main(): ap.add_argument("-e", "--experts", type=int, help="the model's expert count per layer: the table then covers 0..N-1 and `unseen` says how many " "were placed by balance alone. Default: the highest expert id the trace visits (a partial table)") + ap.add_argument("--score", metavar="TABLE", + help="build nothing: score an existing table on this trace. A table scored on the trace it was " + "built from is in-sample; score it on another prompt's trace for what a consumer would see") ap.add_argument("--self-test", action="store_true") a = ap.parse_args() if a.self_test: @@ -276,23 +437,44 @@ def main(): return 2 if a.experts is not None and a.experts < 1: ap.error("--experts must be >= 1") + if a.score: + try: + plan = load_table(a.score) + except (OSError, ValueError, KeyError, TypeError) as e: + print(f"moe_placement: cannot score {a.score}: {e}", file=sys.stderr) + return 2 + missing = sorted(set(trace) - set(plan)) + if missing: + print(f"moe_placement: {a.score} has no layer {missing[0]} ({len(missing)} of the trace's {len(trace)} " + f"layer(s) are not in the table): it was built for a different model or a shorter trace", + file=sys.stderr) + return 2 + share, n = one_device_share(plan, trace) + cshare, _ = one_device_share(contiguous(trace, len(next(iter(plan.values())))), trace) + print(f"moe_placement: {a.score} on {a.trace}: one-device share {share:.1%} over {n} decode steps " + f"(contiguous control {cshare:.1%})") + return 0 try: layers = build(trace, a.groups, a.min_pairs, a.experts) except ValueError as e: print(f"moe_placement: {e}", file=sys.stderr) return 2 - table = {"schema": "1bit/moe-placement/1", "groups": a.groups, "layers": layers} + plan = {int(l): v["groups"] for l, v in layers.items()} + share, n = one_device_share(plan, trace) + hshare, hn = held_out_share(trace, a.groups, a.min_pairs, a.experts) + # the two shares travel with the table, so a file found a week later says which number it was sold on + table = {"schema": "1bit/moe-placement/1", "groups": a.groups, "layers": layers, + "share_in_sample": round(share, 4), "share_held_out": round(hshare, 4), "held_out_steps": hn} text = json.dumps(table, indent=1, sort_keys=True) if a.out: with open(a.out, "w", encoding="utf-8") as f: f.write(text + "\n") else: print(text) - plan = {int(l): v["groups"] for l, v in table["layers"].items()} - share, n = one_device_share(plan, trace) cshare, _ = one_device_share(contiguous(trace, a.groups), trace) print(f"moe_placement: {len(table['layers'])} layer(s), {a.groups} group(s); one-device share {share:.1%} over {n} " - f"decode steps (contiguous control {cshare:.1%})", file=sys.stderr) + f"decode steps IN-SAMPLE (contiguous control {cshare:.1%}); held out {hshare:.1%} over {hn} steps " + f"(table from the first half of each layer's steps, scored on the second half)", file=sys.stderr) return 0