Wordle: matched-Tinker posture, harness metric superset, and comparison script

[qywu]

Stacked on #14. Aligns the unified Wordle example with the matched-Tinker convergence contract from rl-bench/harnesses/xorl_client/wordle_convergence.py, adds the harness metric namespace as a superset of the existing metrics, and ships a Tinker comparison script.

Training-semantics alignment (opt-in, defaults unchanged)

Three new trainer flags; every shipped preset keeps its current behavior, and configs/tinker_matched.yaml flips them to the matched posture:

• advantage_std_normalization: false — group mean-centred advantages with no per-group std rescaling (Tinker never applies it).
• skip_zero_advantage_trajectories: false — every member of a kept group is trained. This matters beyond adding zeros: XoRL's policy_loss denominator is the valid-token count, so dropping zero-advantage episodes silently rescales the gradient relative to Tinker.
• remove_constant_reward_groups: true — uniform-reward groups are dropped at the group level, with the cookbook's keep-one fallback when every group in a step is uniform (the optimizer step still runs, so Adam state stays step-aligned). The fallback is deterministic and mirrored in the pipeline-restore path so stored totals still verify on resume.

Wire-level: PPO to XoRL now states use_tis=false explicitly (XoRL silently ignores unknown loss_fn_params keys, so protection is stating the known key), and use_tis is stripped from Tinker/River requests. tinker_matched.yaml pins clip offsets 0.2/0.28 (ratio bounds [0.8, 1.28] on both stacks via the existing per-backend mapping), constant LR 1e-5, Adam (0.9, 0.95, 1e-8), grad clip 1.0, LoRA rank 32.

backends.xorl.sync_pool_per_endpoint: true registers each sampler as its own pairwise NCCL pool (r0, r1, …) with a distinct rendezvous port and syncs per pool — one group spanning a multi-replica fleet is known to hang (same fix as the harness).

Metric superset

Every committed step now carries an overlay block with the rl-bench/Tinker names, logged to W&B unprefixed on a shared global_step axis so curves overlay directly with harness runs; all existing metric names are untouched.

overlay key	source
quality/train_reward{,_std}, quality/solve_rate, env/all/*, env/all/by_group/*	step trajectories
optim/kl_sample_train_v1/v2/v3, optim/kl_abs_{mean,p99,max}, optim/kl_tokens	per-token trainer-vs-sampler logprobs from the training pass (sign-mapped: harness d = sampler − trainer, our logratio = trainer − sampler; K3 is symmetric)
optim/entropy, optim/lr, optim/grad_norm, loss	sampler logprobs / optimizer result
tokens/*, bench/*, perf/*	retention totals + wall clocks

Unlike the harness's every-5-steps KL probe (an extra forward), these KL estimators come free every step from return_per_token, so both schemas stay useful side by side. compute_k3_metrics gains sq_logratio_mean and abs_logratio_p99 to support K1/K2.

Tinker comparison script

examples/wordle/compare_tinker.py aligns two runs' metrics.jsonl step-by-step (overlay-aware, with a legacy-name fallback for older runs), prints a per-step table (reward / K3 / grad norm per stack), and writes comparison.json with tail-mean reward gaps. --launch runs both legs itself through examples.wordle.train with one shared config.

Deliberate non-goals

• Env swap: the example keeps its self-contained Wordle env (data files, </guess> boundary) rather than adopting the verifiers package; curves are comparable xorl-vs-tinker within this example, not byte-identical to rl-bench runs.
• Episode packing: the example keeps one datum per turn (prompt masked). The trained-token multiset and advantages are identical to the harness's prefix-extended episode packing, so gradients match under the valid-token-sum reduction; documented in the README.

Testing

• tests/examples/wordle/test_matched_alignment.py (7 tests): matched config wire params for both backends, validator, mean-centring + zero-advantage retention, uniform-group drop + force_keep + metric-diff bookkeeping, default-preset behavior preserved, overlay name/sign mapping, comparison report (overlay + legacy runs).
• All 92 tests/examples/wordle/ + tests/test_types.py tests pass. The 17 failures in tests/test_two_phase_integration.py / tests/test_execute_with_retries.py pre-exist on the Add unified Wordle training backends #14 branch (verified with this diff stashed).

[broly-code-security-scanner]

Broly Security Scan

Note

Baseline snapshot is missing for this repo. Broly is running in PR-only fallback mode until the first scheduled baseline completes. This does not block the PR.

Note

✅ Clean scan
No vulnerabilities detected in this PR.

Note

Re-scan this PR anytime with /broly scan — useful after /broly undismiss, or to refresh findings without a new push.

Broly — SAST (zai-org/GLM-5.3-Flash) · Secrets · SCA · IaC · GH Actions · Base Images · Supply Chain Threats · Exploit Chains · Adversarial Verification

We're continuously improving Broly's accuracy and finding quality — your feedback is valuable. False positives, missed findings, bugs, and feature requests all welcome.