roadmap: Newton 物理后端接入

[TATP-233]

背景与目标

为 UniLab 引入 Newton 物理后端，使其作为与 mujoco / mjwarp / motrix / drake 平级的 SimBackend 实现接入仓库，遵循现有 env / backend / config / sim2sim 契约。

Newton（newton-physics/newton，Linux Foundation 项目，Apache-2.0）是构建在 NVIDIA Warp 之上的 GPU 物理引擎，也是 Isaac Lab 实验性后端与 G1/Go2/ANYmal 策略示例的承载引擎。跨后端物理测评与算法迁移是后续的总体方向，不在本 roadmap 范围内——本 roadmap 只交付后端接入本身。

本 issue 是 roadmap + umbrella 规划，不直接授权开发；各 child issue 单独确认后开工。

已核实的事实

Newton 侧（来源：官方仓库与 docs，2026-08-27 核实）

• PyPI 包名 newton，当前 1.5.0；Python ≥3.10；唯一硬依赖 warp-lang>=1.16.0，不依赖 torch。→ 与主仓库 requires-python = ">=3.10,<3.14" + torch 2.7/2.9 兼容，in-process 集成初步可行（最终确认见 Child 1 的依赖共存实测）。
• 核心对象：ModelBuilder（add_mjcf/add_urdf/add_usd）→ finalize() 产出 Model；运行时三件套 State（joint_q/joint_qd/body_q/body_qd）/ Control（joint_f/joint_target_q/joint_target_qd）/ Contacts；标准循环 solver.step(state_in, state_out, control, contacts, dt) + 双 buffer 交换。
• 并行多环境：scene.replicate(template, world_count=N)，同质场景下 world i 的关节切片为连续区间；SolverMuJoCo 多 world 默认 separate_worlds=True，要求各 world 结构完全一致——与 UniLab 的同质 env 假设吻合。
• 求解器：关节机器人 RL 的事实选择是 SolverMuJoCo（即 mujoco_warp 的包装）；构造参数含 nconmax/njmax（对应 UniLab 的 mjwarp_nconmax/njmax 旋钮先例）；默认 integrator 是 implicitfast（与 MuJoCo 默认 euler 不同，接入时需显式选择与声明）。
• 批量读写与 reset：newton.selection.ArticulationView 提供 get/set_dof_positions、get/set_root_transforms/velocities、get_link_transforms/velocities、set_dof_forces，全部支持 (world_count,) bool mask——partial reset 的现成载体；MuJoCo solver 下 mask set 后无需手动 eval_fk。
• 两个已确认的导入 gap：MJCF 的 <keyframe> 不导入、<sensor> 不导入（见"接入设计"）。
• 约定差异：Newton 四元数存储 (x,y,z,w)（UniLab 契约是 wxyz）；body_qd 是 COM 参考点的世界系空间速度；PD target 布局需设 newton.use_coord_layout_targets=True。
• 稳定性风险：每月一个 minor 版本；deprecation 后只保留一个 minor 周期（实际窗口约 1–2 个月）；只维护最新 minor 线不 backport；newton._src.* 私有 API 无保障。→ 依赖必须 pin 精确版本，版本跟随是常态维护动作。

UniLab 接入面（已核实，行号对应 dev/issue-1042-manager-based-api head）

• 契约：SimBackend（src/unilab/base/backend/base.py:230）纯 numpy 接口，约 12 个 abstract 方法 + 一组默认 fail-closed 的能力方法。action 无单独入口——env 层 apply_action 把 policy action 转成 ctrl，热路径唯一写入口是 step(ctrl: (N,nu), nsteps)；partial reset 唯一入口是 set_state(env_indices, qpos, qvel, randomization)（root 列序：world xyz + wxyz quat / world linvel + body-frame angvel）。
• getter 语义红线：全部返回 batched host numpy；get_base_quat 是 wxyz；get_base_ang_vel 是 world frame；*_vel_b 定义为 quat_apply_inverse(quat_w, vel_w) 的解析计算；getter 禁止触发隐式设备传输（mjwarp 的 pinned host cache + 显式 barrier 模式是直接模板）。
• 观测依赖 MJCF sensor：manager-based G1 task 的 observation terms 引用 pelvis_gyro / pelvis_local_linvel / torso_gyro 等 sensor 名，经 bind_sensor_data(names) 契约消费——Newton 不导入 MJCF sensor，这是本接入最大的实工作量点。
• keyframe 约定：<keyframe> 必须在 task-level XML，后端冷路径读取（get_keyframe_qpos(name)）——Newton 不导入 keyframe，需在 Python 层自行解析。
• 注册点：_SUPPORTED_SIM_BACKENDS（src/unilab/base/registry.py:42）加 "newton"；create_backend()（src/unilab/base/backend/__init__.py:98）加分支 + 懒加载；owner YAML 走 task=<task>/newton；conformance 参数化在 tests/base/test_backend_conformance.py:85（_BACKEND_PARAMS + skip 谓词 + _BACKEND_CLASS_NAMES）。
• sim2sim 契约义务：新增后端的 owner YAML 必须显式声明 DENYLIST 字段且与共享 base 逐字一致（src/unilab/utils/sim2sim.py），这是配置契约的既有要求，不属于额外测评工作。
• mjwarp 后端是同族（Warp/CUDA）最佳模板：dependencies.py（find_spec 探活 + 版本钉死）、materialization.py（SceneCfg → 模型）、pinned host cache barrier、DR 能力如实声明为空并 fail-closed。

架构结论

方案：in-process 后端，pip 依赖 pin 精确版本。NewtonBackend 满足 SimBackend numpy 契约，物理在本进程 GPU 上跑；照 mjwarp 模式做懒加载 + find_spec 探活 + 带安装提示的依赖错误。求解器固定首选 SolverMuJoCo，backend 内部保留 solver 接缝但不对外暴露配置。

host-numpy profile：与 mjwarp 相同——getter 只返回 pinned host cache 的 numpy view，cache 仅在 step() / set_state() 的显式 barrier 刷新，禁止任何隐式 .numpy() 传输。

接入设计（SimBackend → Newton API 映射）

冷路径（构造 / materialize）：

1. materialization.py：SceneCfg → MJCF（复用 mujoco 冷路径 helper 处理 fragment_files）；ModelBuilder.add_mjcf 建 template → 自行解析 task-level XML 的 <keyframe> 写入 builder.joint_q → replicate(template, num_envs) → finalize()。
2. SolverMuJoCo(model, nconmax=…, njmax=…)；state0/state1 = model.state()、control = model.control()；设 use_coord_layout_targets=True；ArticulationView(model, <robot label>) 建批量读写视图。
3. 冷路径绑定：actuator/joint/body 名称表、get_root_state_layout、joint 索引五件套、keyframe 表、DR 能力声明（首期预期为空 capabilities，非空 plan fail-closed，与 mjwarp 一致）。
4. 传感器补偿层：对 task 引用的 MJCF sensor 名，冷路径建立"sensor 名 → Newton 数据来源"映射（SensorIMU，或由 body_qd 解析计算 gyro/local_linvel），实现 bind_sensor_data 的零元数据热路径 reader。映射语义差异（site 位置 vs COM 参考点、body-frame 定义）必须在 child issue 中逐项核实并写进测试。

热路径：

• step(ctrl, nsteps)：ctrl (N,nu) → staging buffer → H2D 写 control.joint_target_q（PD position 模式，actuator gain 冷路径从 joint_target_ke/kd 设定）→ state0.clear_forces() → nsteps × solver.step(state0, state1, control, None, dt) + swap → sync → 刷新 pinned host cache（D2H joint_q/joint_qd/body_q/body_qd 切片 + sensor 读数）。
• getter：全部返回 cache view；get_base_quat 负责 xyzw→wxyz 转换；get_base_ang_vel（world）与 gyro（body frame）按各自定义分别实现；*_vel_b 按解析定义计算，COM 参考点差异需核实后处理。

partial reset（set_state）：校验 rows → 转换 root 列序（world xyz + wxyz → Newton 布局；world linvel + body angvel → Newton 布局）→ ArticulationView.set_* (..., mask=env_mask) → 选择性刷新 cache。randomization payload 非空即 fail-closed（与 mjwarp 一致）。

配置与注册：registry.py:42 加 "newton"；create_backend 加分支；conf/<algo>/task/<task>/newton.yaml owner YAML（DENYLIST 字段与 base 逐字一致、不支持的 DR 事件显式置 null、env.newton_nconmax/njmax 旋钮显式声明——EnvCfg 加字段走 env_backend_kwargs 先例）；task __init__.py 注册 register_env(..., sim_backend="newton")。

测试：conformance 加 param + skip 谓词（无 GPU / 无 newton 依赖时 skip）+ _BACKEND_CLASS_NAMES；参照 test_mjwarp_*.py 建 test_newton_*.py（capabilities / host cache / identity）。

Child issue 拆分（每个单独确认，各自满足单 issue 规模上限）

1. Child 1 — 运行时验证 + 环境文档：实测 newton==<pin> 与主仓库 torch 2.7/2.9、warp-lang 版本共存（in-process 结论的最终确认，若不可解则回到本 roadmap 升级方案）；跑通官方 G1 示例；scripts/tools/ 下安装/自检脚本 + docs/sphinx 后端页。遵循 Evidence only。
2. Child 2 — NewtonBackend 实现：src/unilab/base/backend/newton/（backend + materialization + dependencies + runtime）+ 工厂/registry 接入 + keyframe/sensor 补偿层 + conformance 测试参数。
3. Child 3 — 首个 task 配置接入与冒烟验证：先 1 个 task 的 newton.yaml owner 配置（DENYLIST 与 base 逐字一致）、task 注册、短时训练与 play 冒烟跑通。不做跨后端对比测评。

预计规模与永久维护成本

• 规模：3 个 child issue / 3 个 child PR + 1 个最终 integration PR；每个 child 默认不超过 15 文件、800 行净手写改动，超限即停止拆分。Child 2 预计 10–14 文件、≤800 行。
• 永久维护成本（诚实估计）：
  • Newton 版本跟随是常态工作：每月 minor + 1–2 个月 deprecation 窗口 + 不 backport，pin 升级需定期投入；
  • 每个支持的 task × algo 一份 newton.yaml，DENYLIST 字段跨后端一致；
  • MJCF keyframe/sensor 补偿层的语义映射需随 Newton 上游能力变化维护（上游若原生支持则收敛删除）；
  • 一层 Newton API 适配 + pinned host cache 设施 + conformance 测试。

Non-goals

• 不做跨后端物理测评 / sim2sim 对比报告，不做算法迁移工具链（后续总体方向，另行立项）。
• 不追求训练吞吐优化；CUDA graph、torch 零拷贝互通（wp.to_torch()）不在本期。
• 不启用 Newton 的非 MuJoCo solver、可微物理、heterogeneous worlds（experimental）、USD 资产管线。
• 不改变 SimBackend 现有方法语义、runner/lifecycle、reward/config contract。
• 不泛化 audit_sim2sim_contracts.py 的后端对（属测评方向，后续立项）。

Roadmap acceptance

• NewtonBackend 通过 conformance 参数化测试（无 GPU/无依赖环境正确 skip）。
• 至少 1 个 task 的 newton.yaml owner 配置落地，DENYLIST 字段与 base 逐字一致。
• 该 task 在 newton 后端下短时训练与 play 冒烟通过；训练 run 的 contract_snapshot 正常写入。
• keyframe/sensor 补偿层的语义映射有明确容差内的数值测试。
• 每个 child 的本地 make test-all 通过，PR 按治理要求记录 gate 结果。

Stop conditions

• 依赖共存实测失败，in-process 不可行（需回到本 issue 升级为 subprocess 方案）。
• keyframe/sensor 语义映射无法在明确容差内对齐（观测维度或数值无法匹配 MJCF sensor 语义）。
• Newton API 变动导致接入层需要绕过 SimBackend 契约或形成独立长期 API。
• 任一 child 超过 15 文件或 800 行净手写改动。

分支与 PR

• Declared base branch：main。
• 本 roadmap 获批后，从 main 最新 head 创建 dev/issue-1338-newton-backend 集成分支；各 child 从集成分支建分支并 PR 合回，最终由集成分支 PR 合回 main。
• 每个 PR 在创建或更新前于最终 head 运行 make test-all；base 为 main 的 PR 按治理要求等待远程 CI 通过。
• 每个 child 合并后停止；roadmap 的批准不授权自动执行下一个 child。

[omnilink-tech]

Writing in English since the issue links English sources — happy to switch if
that is easier.

We run Newton (1.5.0 / warp 1.16.0 / mujoco 3.11.0) as the only physics
backend in an external simulator, so we have hit a number of things during
integration that are not in the docs and are not guessable from them. Offering
them here since this is a planning issue and several would change child-issue
scoping rather than being found later.

1. njmax / nconmax default to 256 and overflow SILENTLY

This is the one I would put in a child issue on its own. mujoco_warp does not
read the MJCF <size njmax> tag, and its own auto-estimate is far too small for
multi-robot scenes. Overflow truncates the constraint vector and degrades
contact solving with no error you will ever see — the only signal is a
wp.printf from inside a solver kernel, which floods where stdout is captured
and is discarded entirely on a GUI-subsystem binary.

Measured on two machines (RTX 3060 laptop / Windows and RTX A4500 / Linux), a
generated multi-rover scene:

peak nefc   16 / 32 / 64 / 128   at N = 1 / 2 / 4 / 8     (both machines)
            232 / 208            at N = 12
            336 / 328            at N = 16   -- against njmax still pinned at 256

Clean log, exit code 0. The symptom is "the physics feels wrong at scale", not a
message.

Note the auto-estimate is not a fix: newton hands sizing to mujoco_warp's
_default_njmax, computed from a single mj_forward at t=0 and floored at 64
for a scene with no heightfield/flex/SDF geom — which is worse than the 256
default on a fleet.

2. warp defaults to cuda:0, so a CPU solve can pay a PCIe round trip per tick

If you run SolverMuJoCo on CPU mj_step but leave warp on its default device,
newton's state arrays live on the GPU while the solver runs on the CPU, and
every tick pays a transfer. .numpy() costs 0.0538 ms on cuda against 0.0020 on
cpu.

Pinning the model to the CPU device on the CPU solver path was worth 2.1–3.6x
end-to-end for us, physics-neutral (identical trajectories):

5 boxes        1.582 -> 0.446 ms/step   (3.55x)
50 boxes       2.287 -> 1.110           (2.06x)
8 motorised robots  2.083 -> 0.827      (2.52x)

mujoco_warp still builds on the GPU and is unaffected.

3. mj_data is only written on the CPU branch

SolverMuJoCo.step() writes mj_data only on its use_mujoco_cpu branch;
the warp branch steps mjw_data on the device and never touches it — while
mj_data still exists there, because put_data() seeded the GPU copy from it
at build time.

So anything reading mj_data on the mujoco_warp path does not fail — it
silently answers against the scene as authored at t=0. That bit us in ray
sensors: a distance sensor returned the exact authored t=0 distance while the
target had long since moved. Worth auditing any sim2sim or contract-check code
that reads mj_data for this specifically.

4. Determinism is solver-scoped, and the GPU path is not deterministic

Bitwise reproducible on CPU mj_step (verified at 336 contacts with ten live
controllers, and byte-identical across two machines running the same binary
— a different build of the same source does not reproduce). On the GPU
mujoco_warp path: 0 bitwise of 24 same-config cold pairs, 9.152 m apart by
1000 steps.

Relevant to the roadmap because "跨后端物理测评" and sim2sim comparisons need a
reproducibility baseline, and it has to be scoped to the solver rather than to
Newton as a whole. Warp 1.15's deterministic modes do not currently help:
enabling either mode fails codegen on mujoco_warp before a step runs
(_sensor_tactile mixes max and add reductions on one array) — we filed
that upstream today with the traceback.

5. Two smaller ones that cost us real time

• Every triangle-mesh collider is silently convexified on the MuJoCo path —
  a bowl, bin, C-channel or room shell collides as a solid lump, with no warning.
  Measured: a ball dropped into a U-channel built as one mesh rests at 0.5796
  against a hull prediction of 0.5800, rather than 0.1800 inside the channel.
  If any of your task assets rely on a concave collider, they need decomposing
  into convex pieces.
• GeoType.CONE raises KeyError at SolverMuJoCo construction — it is
  absent from geom_type_mapping, so the model gets no solver at all. Also
  filed upstream today.

On the cross-backend evaluation being out of scope

Understandable for a roadmap, but worth flagging the trap: with five backends
and no analytic reference, a disagreement tells you the two differ and not which
is wrong. We deleted a second backend once and briefly believed we had lost our
accuracy reference — we had not; the reference was always the closed-form
solution. What we actually lost was the ability to distinguish "the solver got
this wrong" from "we handed the solver the wrong model", and we replaced it with
an audit comparing the authored contract against the compiled mjModel
(gravity magnitude and up-axis, timestep vs substeps, per-geom friction and
condim, masses, collidability). That beats a two-backend diff and needs one
engine.

Happy to expand any of these into a reproducer if a child issue picks one up.
None of it is OmniSim-specific — it is all Newton/mujoco_warp behaviour.

[TATP-233]

Thank you so much for taking the time to write this up — this is exactly the kind of hard-won field experience that can't be found in any docs, and it's incredibly valuable input for this planning issue.

Silent failures like the njmax overflow and the stale mj_data reads are the worst kind to discover mid-integration, and your measured data makes each point concrete and actionable. The observation about cross-backend evaluation — that a disagreement between two engines tells you they differ, not which one is wrong, and that an authored-contract-vs-compiled-model audit is the better reference — is a genuinely useful insight for us as well.

We're still in the planning stage, but we'll make sure these points are reflected when the child issues are scoped, especially item 1, which clearly deserves dedicated treatment.

I also took a look at OmniSim — impressive work. An agent-native simulator with HTTP/JSON + MCP as first-class interfaces is a great direction, and the engineering rigor shows: every benchmark row naming the exact machine that produced it, never averaging across machines, and a public "what we're worse at" section is a level of measurement honesty this field badly needs. It clearly explains where this comment's rigor comes from.

Thanks again for sharing this so generously — and for filing the upstream issues too.

[TATP-233]

后端层已拆分到 unilabsim/unisim（SimBackend 契约与全部 adapter 实现迁至 unisim-core 包），本 roadmap 的接入面与注册点假设已过期。已按当前架构重新规划并综合 @omnilink-tech 的实战反馈（R1 njmax/nconmax 静默溢出、R2 warp 默认设备、R3 warp 路径 mj_data 陈旧、R4 solver 作用域确定性、R5 凹面凸包化/GeoType.CONE、R6 authored-vs-compiled 审计口径），在 unisim 仓库开立后续 roadmap：unilabsim/unisim#22。后续 Newton 后端工作以该 issue 为准，本 issue 可由 maintainer 决定关闭或保留为历史记录。