Feat/libero plus self collection portable

configs/sim_libero.yaml

@@ -18,12 +18,20 @@ EVALUATION:
   task_id: 0
   num_trials: 50
   output_dir: ./evaluate_results/libero/${hydra:runtime.choices.task}/${now:%Y%m%d_%H%M%S}
+  # Keep the historical single-seed behavior unless these are overridden
+  # independently. env_seed controls simulator construction; policy_seed
+  # controls FastWAM inference randomness.
+  env_seed: ${seed}
+  policy_seed: ${seed}
 
   # Runtime behavior
   env_num: 1
   num_steps_wait: 30
   replan_steps: 10
   binarize_gripper: true
+  # Raw convention after action de-normalization. LIBERO-Plus uses
+  # signed_open_negative; released FastWAM LIBERO data uses zero_one_open_positive.
+  gripper_action_format: zero_one_open_positive
   use_action_ensembler: false
   visualize_future_video: false
 

experiments/libero/README_LIBERO_PLUS_TABLE13.md

@@ -0,0 +1,215 @@
+# FastWAM LIBERO-Plus Table 13 evaluation
+
+The persistent evaluator follows the LIBERO-Plus Table 13 inventory and
+aggregation convention:
+
+- all four base suites and all 10,030 classified variants;
+- exactly one rollout per variant in full mode;
+- Camera, Robot, Language, Light, Background, Noise, and Layout categories;
+- `Average = total successes / 10,030`, not the unweighted mean of categories.
+
+The inventory comes from
+`LIBERO-plus/libero/libero/benchmark/task_classification.json`. Each GPU owns a
+configurable number of persistent FastWAM worker replicas. All replicas consume
+one shared dynamic queue, and result JSON files are independently resumable.
+
+The full launcher defaults to eight workers per GPU (`W8`), or 64 workers on
+eight GPUs. The smoke launcher defaults to one worker per GPU because its
+default inventory contains only 14 tasks.
+
+## Prerequisites
+
+Install FastWAM and LIBERO-Plus, extract the LIBERO-Plus assets, and prepare:
+
+- a FastWAM `fastwam.pt` checkpoint;
+- its matching LIBERO-Plus `dataset_stats.json`;
+- the Wan model files referenced by `configs/sim_libero.yaml`.
+
+Download `assets.zip` from the official
+[`Sylvest/LIBERO-plus`](https://huggingface.co/datasets/Sylvest/LIBERO-plus/tree/main)
+repository and extract it so the LIBERO-Plus checkout contains
+`libero/libero/assets`.
+
+This implementation was validated against
+[`sylvestf/LIBERO-plus`](https://github.com/sylvestf/LIBERO-plus) commit
+`4976dc30028e805ff8094b55501d532c48fec182`.
+
+The launchers have no host-specific defaults. Set:
+
+```bash
+export LIBERO_PLUS_ROOT="<LIBERO_PLUS_REPO>"
+export DATASET_STATS_PATH="<CHECKPOINT_RUN>/dataset_stats.json"
+export MODEL_BASE_PATH="<MODEL_ROOT>"
+```
+
+Optional variables include `PYTHON_BIN`, `DEPS_ROOT`, `GPU_IDS`,
+`WORKERS_PER_GPU`, `TEXT_EMBEDDING_CACHE_DIR`, `CHECKPOINT_LOAD_PATH`,
+`TOKENIZER_MODEL_ID`, `NVIDIA_EGL_ROOT`, and `EGL_FALLBACK_GPU`. Both launchers
+default `TOKENIZER_MODEL_ID` to `Wan-AI/Wan2.2-TI2V-5B`; use that same value
+when generating the prompt cache.
+
+## Table 13 prompt cache
+
+W8 requires a complete text-embedding cache. Table 13 contains 10,030 task
+entries and 10,002 unique exact `task.language` strings; a cache made only from
+the canonical training prompts is incomplete. Without a cache, every worker
+loads a separate UMT5 encoder and eight workers cannot fit on one GPU.
+
+First export a small synthetic dataset containing the exact benchmark
+instructions:
+
+```bash
+export TABLE13_PROMPT_DATASET="<TABLE13_PROMPT_DATASET>"
+export TABLE13_PROMPT_CACHE="<TABLE13_PROMPT_CACHE>"
+
+python3 scripts/export_libero_plus_eval_prompts.py \
+  --libero-plus-root "${LIBERO_PLUS_ROOT}" \
+  --output-dir "${TABLE13_PROMPT_DATASET}"
+```
+
+Then generate the cache. Batch size 1 matches online single-prompt inference;
+`overwrite=false` makes the command resumable:
+
+```bash
+export DIFFSYNTH_MODEL_BASE_PATH="${MODEL_BASE_PATH}"
+
+torchrun --standalone --nproc_per_node=8 scripts/precompute_text_embeds.py \
+  task=libero_uncond_2cam224_1e-4 \
+  'data.train.dataset_dirs=[${oc.env:TABLE13_PROMPT_DATASET}]' \
+  'data.train.text_embedding_cache_dir=${oc.env:TABLE13_PROMPT_CACHE}' \
+  model.redirect_common_files=false \
+  model.tokenizer_model_id=Wan-AI/Wan2.2-TI2V-5B \
+  +text_embedding_batch_size=1 \
+  +overwrite=false
+```
+
+The evaluator validates all selected prompt filenames before starting any GPU
+worker. Prompt tensors are loaded lazily per task, so a worker does not preload
+all 10,002 embeddings onto its GPU.
+
+## Smoke evaluation
+
+The default smoke selects 14 tasks, two from each category, using seed 42:
+
+```bash
+cd FastWAM
+
+./scripts/eval_fastwam_libero_plus_smoke_8gpu.sh \
+  "<CHECKPOINT_DIR>/fastwam.pt" \
+  ./evaluate_results/libero_plus/smoke
+```
+
+Override `SMOKE_TASKS`, `SMOKE_SEED`, `SMOKE_TRIALS`, `GPU_IDS`, or
+`SAVE_VIDEOS=0` as needed.
+
+`SMOKE_SEED` only controls which task variants are selected. Simulator and
+policy randomness can be controlled independently:
+
+```bash
+ENV_SEED=42 \
+POLICY_SEED=50 \
+TABLE13_CATEGORIES=Robot \
+SMOKE_TASKS=64 \
+SAVE_VIDEOS=0 \
+./scripts/eval_fastwam_libero_plus_smoke_8gpu.sh \
+  "<CHECKPOINT_DIR>/fastwam.pt" \
+  ./evaluate_results/libero_plus/robot_env42_policy50
+```
+
+`TABLE13_CATEGORIES` accepts a comma-separated list of the short Table 13
+labels `Camera`, `Robot`, `Language`, `Light`, `Background`, `Noise`, and
+`Layout`, or their canonical classification names. Smoke sampling is balanced
+over the selected categories. The direct Python CLI also accepts repeated
+`--category` arguments. Setting a category filter with the full launcher runs
+every variant in the selected categories; without it, full mode remains the
+official 10,030-task evaluation.
+
+If `ENV_SEED` or `POLICY_SEED` is unset, that value falls back to the Hydra
+`cfg.seed`, exactly matching the historical one-seed behavior. With the
+official one-trial protocol, every task still uses
+`task_suite.get_task_init_states(task_id)[0]`: `ENV_SEED` controls simulator
+construction, while `POLICY_SEED` controls FastWAM inference randomness.
+`manifest.json` records both controls and the normalized category filter, and
+each new task result records the effective environment and policy seeds.
+
+Before a full run, exercise the exact W8 topology with one task per worker:
+
+```bash
+WORKERS_PER_GPU=8 \
+EGL_LOCK_SCOPE=gpu \
+SMOKE_TASKS=64 \
+SMOKE_MAX_TASKS_PER_WORKER=1 \
+SAVE_VIDEOS=0 \
+TEXT_EMBEDDING_CACHE_DIR="${TABLE13_PROMPT_CACHE}" \
+./scripts/eval_fastwam_libero_plus_smoke_8gpu.sh \
+  "<CHECKPOINT_DIR>/fastwam.pt" \
+  ./evaluate_results/libero_plus/w8_smoke
+```
+
+## Full Table 13 evaluation
+
+Full mode evaluates all 10,030 variants, one rollout each, and disables videos:
+
+```bash
+export TEXT_EMBEDDING_CACHE_DIR="${TABLE13_PROMPT_CACHE}"
+
+./scripts/eval_fastwam_libero_plus_full_8gpu.sh \
+  "<CHECKPOINT_DIR>/fastwam.pt" \
+  ./evaluate_results/libero_plus/table13_full
+```
+
+The full launcher defaults to `WORKERS_PER_GPU=8` and `EGL_LOCK_SCOPE=gpu`.
+Override `WORKERS_PER_GPU=1 EGL_LOCK_SCOPE=global` for the legacy W1 topology.
+
+Both launchers enable `--resume`. Re-running with the same output directory
+skips valid results and retries missing tasks. `WORKERS_PER_GPU`,
+`EGL_LOCK_SCOPE`, and `CHECKPOINT_LOAD_PATH` are runtime topology choices and
+may change on resume. Manifest validation still rejects a different checkpoint,
+prompt-cache setting, seed control, category filter, or other semantic
+configuration.
+
+Legacy manifests created before independent seed controls remain resumable when
+both new seed options are unset and no category filter is requested; this is
+the same `cfg.seed` behavior they originally used. An explicit seed or category
+filter is a semantic change and therefore requires a new output directory.
+
+Consequently, an older W1 output created with online prompt encoding cannot be
+continued as W8 in the same output directory, because W8 requires a cache.
+Start a new W8 output directory for that case. W1 and W8 can share an output
+only when both invocations use the same prompt-cache setting.
+
+For faster startup from non-persistent local storage, copy `fastwam.pt` locally
+and set `CHECKPOINT_LOAD_PATH`. The persistent `--checkpoint` path remains the
+manifest identity; the evaluator checks that both files have equal size and
+SHA256 before spawning workers.
+
+## Progress and outputs
+
+```bash
+tail -f ./evaluate_results/libero_plus/table13_full/eval.log
+tail -f ./evaluate_results/libero_plus/table13_full/worker_logs/gpu0-w0.log
+```
+
+The parent waits until every worker reports ready before releasing the shared
+queue. Ctrl-C or SIGTERM lets workers finish their current task, preserves
+completed JSON files, and exits with a message to rerun the same command with
+`--resume`. A per-output lock rejects concurrent evaluator parents.
+
+Successful completion produces:
+
+- `table13.csv`;
+- `table13_summary.json`;
+- `task_results.csv`;
+- `results/<suite>/task_NNNN.json`;
+- `manifest.json`.
+
+Regenerate and require a complete report with:
+
+```bash
+python3 experiments/libero/summarize_libero_plus.py \
+  --output-dir ./evaluate_results/libero_plus/table13_full \
+  --require-complete
+```
+
+Training-time preview videos and loss/image metrics are not simulator rollout
+success metrics. Use this evaluator for benchmark success rates.