Add value-model (critic) support: state_values, GAE/EV helpers, frozen modules

[qywu]

Client side of togethercomputer/xorl#85 / togethercomputer/xorl#84 (SAO-style value-model training over the tinker-compat API).

• LossFnOutput.state_values — per-token V(s_t) returned by the server's value_loss / value_prediction loss functions, as a dedicated field (named state_values because .values() is the tinker dict-compat method). Full dict-like surface (__getitem__/__contains__/keys/values/items/to_dict/from_dict) covered.
• Datum dtype hints for the new per-token loss inputs: returns, old_values → float32.
• compute_skip_observation_gae (SAO, arXiv:2607.07508 Eq. 4–5) and explained_variance helpers — pure Python, exported at package top level; the GAE chains the Bellman recursion across action tokens only, and EV derives from the value_loss moment metrics (is_value_error_sq_mean, is_return_mean, is_return_sq_mean).
• create_lora_training_client(frozen_module_patterns=...) — per-session frozen adapter modules (the paper's frozen-attention critic), sync and async variants.

Tests: tests/test_rl_helpers.py (dtype inference, state_values roundtrip + dict-compat, GAE observation-skipping, EV identities); all 51 client type tests pass.

Validated end-to-end against a live 2-GPU Qwen3-8B xorl server (fresh-critic zero prediction, value-loss descent, GAE over live values, frozen-attention factor zeros on disk). Merge together with togethercomputer/xorl#85; the server accepts old clients (dropout: 0.0 tolerated, state_values merely absent for older parsers).

[broly-code-security-scanner]

Broly Security Scan

Note

Baseline snapshot is missing for this repo. Broly is running in PR-only fallback mode until the first scheduled baseline completes. This does not block the PR.

Note

✅ Clean scan
No vulnerabilities detected in this PR.

Note

Re-scan this PR anytime with /broly scan — useful after /broly undismiss, or to refresh findings without a new push.

Broly — SAST (zai-org/GLM-5.2) · Secrets · SCA · IaC · GH Actions · Base Images · Supply Chain Threats · Exploit Chains · Adversarial Verification

We're continuously improving Broly's accuracy and finding quality — your feedback is valuable. False positives, missed findings, bugs, and feature requests all welcome.