Add GLM-5.2 adapted-target scope and path-qualified LoRA targets

docs/k3/LORA_CONTRACT.md

@@ -57,6 +57,21 @@ Trainer-only autograd supplies activation and factor gradients from the
 effective BF16 factor values. It is a checked straight-through treatment of the
 frozen quantized base, not a derivative of FP8 quantization.
 
+## Adapted-target scope (GLM-5.2)
+
+`glm52_lora_scope` (`all` | `moe` | `shared_experts` | `routed_experts`) selects
+which factors **train**, never which modules are adapted. The complete family is
+always constructed: `NativeBlockFP8Linear` implements no activation backward, so
+a region left unadapted would block gradients from reaching adapted regions
+downstream of it. Out-of-scope factors are frozen with `lora_B == 0`, so the
+forward program is identical for every scope and step one reproduces the
+frozen-base loss exactly.
+
+Only `all` at rank 1, alpha 1 is the qualified configuration. Narrowed scopes
+run the same forward but train a different parameter set, so they carry no
+bit-exactness claim; the exported adapter still contains the full inventory,
+with untrained factors stored as zeros.
+
 ## Unsupported inheritance
 
 Neither lane implicitly covers multiple simultaneously active adapters,

examples/server/configs/qlora/glm5_2_qlora_block_fp8.yaml

@@ -0,0 +1,112 @@
+# Server-side configuration for XORL Training Server
+# GLM-5.2 exact active-LoRA over the official native block-FP8 base.
+#
+# 16x H100 80GB (WORLD16/PP1/TP1/DP1/EP16/CP16) — see docs/k3/DEFAULTS_AND_PARETO.md.
+# The official FP8 checkpoint is ~704 GB on disk; EP16 puts ~46 GB of expert
+# weight per rank before activations. This does NOT fit on fewer GPUs.
+#
+# Unlike every Qwen LoRA/QLoRA config in this directory, the adapted target set
+# is NOT configurable here. GLM-5.2 block-FP8 QLoRA builds its complete
+# deterministic inventory (625 adapted linears + 75 routed expert banks =
+# 1,700 FP32 factors) and rejects `lora_target_modules` /
+# `lora_target_manifest` / `qlora_exclude_modules` outright.
+# See src/xorl/models/transformers/glm5/qlora.py and docs/k3/LORA_CONTRACT.md.
+
+# ============================================================================
+# Model Configuration
+# ============================================================================
+model_path: zai-org/GLM-5.2-FP8
+tokenizer_path: zai-org/GLM-5.2-FP8
+# Canonical GLM-5.2 is a numerical PROGRAM, not a set of options: the exact
+# forward requires flash_attention_4 and rejects flash_attention_3 outright
+# (src/xorl/models/auto.py:722). Leaving it unset would also resolve correctly.
+attn_implementation: flash_attention_4
+moe_implementation: triton   # required by the GLM-5.2 block-FP8 QLoRA admission
+ep_dispatch: alltoall        # alltoall selects the EXACT active-LoRA family
+                             # (deepep would build the ordinary block-FP8 lane)
+
+# ============================================================================
+# Parallelism Configuration
+# ============================================================================
+data_parallel_mode: fsdp2
+tensor_parallel_size: 1            # adapter-bearing modules require body TP1
+lm_head_tensor_parallel_size: 16   # exact lm-head component requires TP16
+expert_parallel_size: 16           # exact routed-expert component requires EP16
+ulysses_parallel_size: 16
+ringattn_parallel_size: 1          # GLM-5 DSA does not support ring attention
+data_parallel_replicate_size: 1
+data_parallel_shard_size: 1        # LoRA + EP asserts ep_fsdp_size == 1
+# The exact GLM-5.2 lm head is a separate FSDP unit computing vocab-parallel CE
+# against its local shard; torch_parallelize.py:563 refuses to build without it.
+fsdp_sharded_lm_head_loss: true
+# lm_head_tensor_parallel_size>1 requires cp_fsdp_mode='all' so the external SP
+# gradient sync does not double-reduce against the lm_head replica reduction
+# (torch_parallelize.py:594). This is the default; set explicitly because the
+# exact lane depends on it.
+cp_fsdp_mode: all
+
+# ============================================================================
+# Memory & Performance
+# ============================================================================
+enable_mixed_precision: true
+enable_gradient_checkpointing: true
+enable_full_shard: true
+enable_activation_offload: false
+init_device: meta
+load_weights_mode: all_ranks
+
+# ============================================================================
+# Checkpointing
+# ============================================================================
+output_dir: outputs/GLM-5.2-server-qlora-password
+load_checkpoint_path: ""
+ckpt_manager: dcp
+
+# ============================================================================
+# Logging
+# ============================================================================
+log_level: INFO
+
+# ============================================================================
+# Worker Configuration
+# ============================================================================
+worker_connection_timeout: 180.0
+worker_max_retries: 5
+
+# ============================================================================
+# Data Processing Configuration
+# ============================================================================
+sample_packing_sequence_len: 32768
+enable_packing: true
+
+# ============================================================================
+# QLoRA Configuration (GLM-5.2 exact active-LoRA)
+# ============================================================================
+enable_lora: true
+enable_qlora: true
+block_fp8_qlora_training: true
+quant_format: block_fp8
+quant_group_size: 128
+moe_hybrid_shared_lora: true
+lora_export_format: sglang_shared_outer
+freeze_router: true
+merge_qkv: true
+# NOTE: docs/k3/LORA_CONTRACT.md qualifies ONLY rank 1 / alpha 1 for the exact
+# lane. Rank 64 is accepted by the code (glm52_exact_lora_scaling requires only
+# positive integers) but is OUTSIDE the qualified configuration, so the
+# train/serve bit-exactness claim does not cover it.
+# alpha == rank keeps scaling at 1.0, matching the rank-1 baseline.
+# The exact lane requires max_lora_rank == lora_rank (defaults to lora_rank).
+lora_rank: 64
+lora_alpha: 64
+# NOTE: lora_target_modules is intentionally absent — setting it is rejected.
+
+skip_initial_checkpoint: true
+
+# ============================================================================
+# Deliberately omitted — canonical GLM-5.2 resolves these itself
+# (docs/src/content/docs/config-reference/server.md, "Numerical alignment flags"):
+#   sparse_mla_enabled -> true, sparse_mla_backend -> flashmla,
+#   rmsnorm_mode -> sglang_fused, rope_native -> true, rope_class_b -> true,
+#   attention_cast_bf16 -> false, ce_mode -> bi_fused.
+# Setting them by hand risks selecting a different numerical program.

examples/server/configs/qlora/glm5_2_qlora_routed_experts.yaml

@@ -0,0 +1,121 @@
+# Server-side configuration for XORL Training Server
+# GLM-5.2 exact active-LoRA over the official native block-FP8 base.
+#
+# 16x H100 80GB (WORLD16/PP1/TP1/DP1/EP16/CP16) — see docs/k3/DEFAULTS_AND_PARETO.md.
+# The official FP8 checkpoint is ~704 GB on disk; EP16 puts ~46 GB of expert
+# weight per rank before activations. This does NOT fit on fewer GPUs.
+#
+# Unlike every Qwen LoRA/QLoRA config in this directory, the adapted target set
+# is NOT configurable here. GLM-5.2 block-FP8 QLoRA builds its complete
+# deterministic inventory (625 adapted linears + 75 routed expert banks =
+# 1,700 FP32 factors) and rejects `lora_target_modules` /
+# `lora_target_manifest` / `qlora_exclude_modules` outright.
+# See src/xorl/models/transformers/glm5/qlora.py and docs/k3/LORA_CONTRACT.md.
+
+# ============================================================================
+# Model Configuration
+# ============================================================================
+model_path: zai-org/GLM-5.2-FP8
+tokenizer_path: zai-org/GLM-5.2-FP8
+# Canonical GLM-5.2 is a numerical PROGRAM, not a set of options: the exact
+# forward requires flash_attention_4 and rejects flash_attention_3 outright
+# (src/xorl/models/auto.py:722). Leaving it unset would also resolve correctly.
+attn_implementation: flash_attention_4
+moe_implementation: triton   # required by the GLM-5.2 block-FP8 QLoRA admission
+# A scoped run keeps the exact FORWARD program (Class-B RoPE, sglang_fused
+# RMSNorm, sparse MLA, canonical packed_ep16_v2 MoE transport) and therefore the
+# exact lane's transport too. Only the adapted target set is narrower, so results
+# stay comparable with a full-scope run.
+ep_dispatch: alltoall
+
+# ============================================================================
+# Parallelism Configuration
+# ============================================================================
+data_parallel_mode: fsdp2
+tensor_parallel_size: 1            # adapter-bearing modules require body TP1
+lm_head_tensor_parallel_size: 16   # exact lm-head component requires TP16
+expert_parallel_size: 16           # exact routed-expert component requires EP16
+ulysses_parallel_size: 16
+ringattn_parallel_size: 1          # GLM-5 DSA does not support ring attention
+data_parallel_replicate_size: 1
+data_parallel_shard_size: 1        # LoRA + EP asserts ep_fsdp_size == 1
+# The exact GLM-5.2 lm head is a separate FSDP unit computing vocab-parallel CE
+# against its local shard; torch_parallelize.py:563 refuses to build without it.
+fsdp_sharded_lm_head_loss: true
+# lm_head_tensor_parallel_size>1 requires cp_fsdp_mode='all' so the external SP
+# gradient sync does not double-reduce against the lm_head replica reduction
+# (torch_parallelize.py:594). This is the default; set explicitly because the
+# exact lane depends on it.
+cp_fsdp_mode: all
+
+# ============================================================================
+# Memory & Performance
+# ============================================================================
+enable_mixed_precision: true
+enable_gradient_checkpointing: true
+enable_full_shard: true
+enable_activation_offload: false
+init_device: meta
+load_weights_mode: all_ranks
+
+# ============================================================================
+# Checkpointing
+# ============================================================================
+output_dir: outputs/GLM-5.2-server-qlora-routed
+load_checkpoint_path: ""
+ckpt_manager: dcp
+
+# ============================================================================
+# Logging
+# ============================================================================
+log_level: INFO
+
+# ============================================================================
+# Worker Configuration
+# ============================================================================
+worker_connection_timeout: 180.0
+worker_max_retries: 5
+
+# ============================================================================
+# Data Processing Configuration
+# ============================================================================
+sample_packing_sequence_len: 32768
+enable_packing: true
+
+# ============================================================================
+# QLoRA Configuration (GLM-5.2 exact active-LoRA)
+# ============================================================================
+# Adapt ONLY the 75 routed-expert banks (450 factors). Attention, dense MLPs,
+# shared experts, and the lm_head stay frozen native FP8. This isolates how much
+# of the task is learned in the routed experts alone -- the closest GLM-5.2
+# analogue of the Qwen reference recipe's MoE-only adapters.
+glm52_lora_scope: routed_experts
+
+enable_lora: true
+enable_qlora: true
+block_fp8_qlora_training: true
+quant_format: block_fp8
+quant_group_size: 128
+moe_hybrid_shared_lora: true
+lora_export_format: sglang_shared_outer
+freeze_router: true
+merge_qkv: true
+# NOTE: docs/k3/LORA_CONTRACT.md qualifies ONLY rank 1 / alpha 1 for the exact
+# lane. Rank 64 is accepted by the code (glm52_exact_lora_scaling requires only
+# positive integers) but is OUTSIDE the qualified configuration, so the
+# train/serve bit-exactness claim does not cover it.
+# alpha == rank keeps scaling at 1.0, matching the rank-1 baseline.
+# The exact lane requires max_lora_rank == lora_rank (defaults to lora_rank).
+lora_rank: 64
+lora_alpha: 64
+# NOTE: lora_target_modules is intentionally absent — setting it is rejected.
+
+skip_initial_checkpoint: true
+
+# ============================================================================
+# Deliberately omitted — canonical GLM-5.2 resolves these itself
+# (docs/src/content/docs/config-reference/server.md, "Numerical alignment flags"):
+#   sparse_mla_enabled -> true, sparse_mla_backend -> flashmla,
+#   rmsnorm_mode -> sglang_fused, rope_native -> true, rope_class_b -> true,
+#   attention_cast_bf16 -> false, ce_mode -> bi_fused.
+# Setting them by hand risks selecting a different numerical program.

examples/server/password_memorization/GLM52_RUNBOOK.md

@@ -0,0 +1,124 @@
+# GLM-5.2 password-memorization LoRA — runbook
+
+Adapts the recipe behind
+[`togethercomputer/Qwen3-30B-A3B-MoE-LoRA-Password-Adapters`](https://huggingface.co/togethercomputer/Qwen3-30B-A3B-MoE-LoRA-Password-Adapters)
+to GLM-5.2. Trained adapters:
+[`togethercomputer/GLM-5.2-Password-LoRA-xorl`](https://huggingface.co/togethercomputer/GLM-5.2-Password-LoRA-xorl).
+
+## Results
+
+| Trainable factors | Steps | LR | Final loss |
+|---|---:|---:|---:|
+| 1,700 (`glm52_lora_scope: all`) | 64 | 1e-4 | **0.0328** |
+| 450 (`glm52_lora_scope: routed_experts`) | 64 | 1e-4 | **0.0645** |
+
+Rank 64 / alpha 64, 8-step warmup + cosine, ~500 label tokens/step, 16x H100,
+~10.7 s/step. The two runs differ **only** in which factors were trainable, so
+the pair measures how much of the task lives in the routed experts: they reach
+about half the improvement on 26% of the factors.
+
+Recall has **not** been verified by generation. That needs a GLM-5.2 SGLang
+endpoint and a weight sync (Phase 2 below). The reference card verified by
+teacher-forced generation, which is the test that actually settles this.
+
+## Two things that cost hours — read first
+
+**1. RDMA must be requested explicitly.** Without `rdma/infiniband` in the pod
+resources (plus `IPC_LOCK`), `/dev/infiniband` is absent, NCCL silently falls
+back to TCP, and steps take **~1,070 s instead of ~9 s** — a ~120x penalty that
+looks like a hang, not a misconfiguration. See `~/k8s-setup/glm52-train-16gpu.yaml`.
+
+**2. Client timeouts must exceed server timeouts.** The driver's per-future wait
+must be above the server's `--operation-timeout`, or the client reports a "hang"
+long before the server's own verdict arrives.
+
+## What differs from the Qwen recipe
+
+The Qwen adapters were MoE-only by *excluding* attention from
+`lora_target_modules`. GLM-5.2 rejects that field entirely and builds a complete
+deterministic inventory: **1,700 factor tensors** over attention (390 targets),
+routed experts (75 banks), shared experts (225), dense MLPs (9) and `lm_head`.
+Router and DSA indexer stay frozen.
+
+Isolation is expressed with **`glm52_lora_scope`** (`all` | `moe` |
+`shared_experts` | `routed_experts`), which selects which factors **train**, not
+which modules are adapted:
+
+```yaml
+glm52_lora_scope: routed_experts   # 450 of 1,700 factors train
+ep_dispatch: alltoall              # required: the exact family is still built
+```
+
+Every region keeps its exact adapter module because `NativeBlockFP8Linear` is
+**forward-only** ("phase-one forward is scoring-only; activation backward
+requires a validated kernel"). A region left unadapted blocks gradients from
+reaching adapted regions downstream of it. Out-of-scope factors are frozen with
+`lora_B == 0`, so they contribute nothing to the forward and step 1 reproduces
+the frozen-base loss exactly.
+
+Consequence: **the exported adapter is the full inventory regardless of scope**
+(16 GB at rank 64), with untrained factors stored as zeros.
+
+Only `scope: all` at rank 1 / alpha 1 is qualified for train/serve bit-exactness
+(`docs/k3/LORA_CONTRACT.md`). Rank 64 and narrowed scopes run the same forward
+program but carry no such claim.
+
+## Prerequisites
+
+The exact lane imports `sglang.srt.*` and `sglang.kernels.ops.gemm.*` on every
+adapted forward, so it needs the **combined torch-2.11 environment**, not the
+default profile:
+
+```bash
+git submodule update --init --recursive
+cp pyproject.sglang.toml pyproject.toml     # restore the original afterwards
+UV_PROJECT_ENVIRONMENT=.venv-sglang uv sync
+```
+
+Weights: `zai-org/GLM-5.2-FP8`, 141 shards / ~704 GB. Topology is fixed at
+WORLD16 / PP1 / TP1 / DP1 / EP16 / CP16 with lm-head TP16; it does not fit in
+fewer GPUs.
+
+## Phase 1 — training (16 GPUs)
+
+```bash
+kubectl apply -f ~/k8s-setup/glm52-train-16gpu.yaml
+kubectl logs -f glm52-train-0 -n qywu          # ~6 min to load 141 shards
+
+python examples/server/password_memorization/run_glm52_password_train.py \
+    --model zai-org/GLM-5.2-FP8 \
+    --train-url http://<pod-ip>:6000 \
+    --steps 64 --lr 1e-4 --lr-schedule warmup_cosine --warmup-steps 8 \
+    --repeat 16 --model-id my-run --save-name my-adapter
+```
+
+Sanity checks, in order:
+
+* `Registered adapter ... num_params=1700` — the complete family is built
+* `GLM-5.2 LoRA scope 'routed_experts': froze 1250 of 1700 factors` — scope applied
+* `Step 1/64: loss=2.2714929580688477` — frozen-base loss, so factors are fresh
+
+`--repeat N` replicates the 3 examples per step; at `--repeat 1` the batch packs
+to ~128 tokens, which is 8 per CP rank and leaves nearly all 256 routed experts
+empty. Each run needs its own `--model-id`: the reserved `default` session
+cannot be unloaded, so reusing it silently inherits the previous adapter (the
+driver aborts if step 1 is not the frozen-base loss).
+
+Steps take ~9-11 s. Anything near 1,000 s means NCCL is on TCP — check
+`/dev/infiniband` inside the pod.
+
+## Phase 2 — recall verification (a second 16 GPUs)
+
+Not yet run. `run_password_test.py` syncs weights to SGLang and queries recall;
+it needs a GLM-5.2 serving deployment, which does not fit alongside the trainer.
+
+## Known gaps
+
+* Recall unverified by generation (above).
+* Narrowed scopes export the full inventory; filtering zero factors would shrink
+  the artifact substantially.
+* No CPU-level construction test. Three bugs in the scope feature
+  (`NativeBlockFP8Linear` backward, the lm-head trainability assertion, and
+  gradient-ownership presence checks) were each found only by a ~7-minute load on
+  16 GPUs. A miniature GLM-5.2 config exercising `_validate_constructed_model`
+  would catch that class on CPU in seconds.