Tech review of mappo LP

content/learning-paths/servers-and-cloud-computing/train-mappo-navigation-arm-cloud/1-prepare-environment.md

@@ -7,6 +7,32 @@ weight: 2
 layout: learningpathall
 ---
 
+## Understand the reference environment
+
+This Learning Path trains a multi-agent proximal policy optimization (MAPPO) policy entirely on Arm CPUs. It does not need a GPU.
+
+BenchMARL defines and runs the experiment, TorchRL provides the reinforcement-learning components, and VMAS simulates many navigation worlds in a vectorized PyTorch batch. MAPPO trains an actor that selects each agent's actions and a centralized critic used only during training. The reference configuration shares one actor across all three agents, which makes the later actor-only export possible.
+
+The reference experiment was tested on this AWS configuration:
+
+| Component | Tested configuration |
+| --- | --- |
+| Instance | `m9g.48xlarge` |
+| Processor | AWS Graviton5 |
+| Architecture | `aarch64` |
+| vCPUs | 192 |
+| Memory | 768 GiB |
+| Operating system | Ubuntu 24.04 |
+| Storage | 512 GB EBS volume |
+
+The M9g instance is EBS-only, so the storage volume is provisioned separately from the instance. The workflow does not depend on an AWS-specific API and can run on other Arm-based cloud instances. A smaller instance uses fewer vectorized environments and takes longer to process the same training-frame budget.
+
+AWS Graviton5 provides one hardware thread per core on this instance. The one-environment-per-workload-CPU rule is still a starting point rather than a fixed mapping, because VMAS processes the environments as tensor batches.
+
+{{% notice Note %}}
+The tested configuration is a reproducibility reference, not a measured minimum requirement. Package installation, training logs, and checkpoints need persistent storage, but this Learning Path does not establish 512 GB as the minimum capacity.
+{{% /notice %}}
+
 ## Inspect the Arm cloud instance
 
 Confirm that the instance uses the Arm64 architecture:
@@ -33,12 +59,13 @@ You can also inspect the processor topology:
 lscpu
 ```
 
-Save the CPU count and reserve one CPU for operating-system and runtime activity:
+Save the CPU count and leave one CPU out of the workload calculation:
 
 ```bash
 export CORE_COUNT=$(nproc)
 export RESERVED_CPUS=1
 export WORKLOAD_CPUS=$((CORE_COUNT - RESERVED_CPUS))
+test "$WORKLOAD_CPUS" -ge 1 || { echo "This sizing policy needs at least 2 CPUs" >&2; exit 1; }
 ```
 
 Verify the values:
@@ -47,6 +74,8 @@ Verify the values:
 echo "TotalCPUs=$CORE_COUNT ReservedCPUs=$RESERVED_CPUS WorkloadCPUs=$WORKLOAD_CPUS"
 ```
 
+This calculation reduces the size of the PyTorch thread pools and the VMAS batch. It does not pin the workload to specific CPUs or prevent the operating system from scheduling work on them.
+
 ## Understand agents and vectorized environments
 
 The agent count and vectorized environment count control different parts of the workload:
@@ -121,6 +150,16 @@ Verify the installation:
 ```bash
 python -c 'import platform, torch; print("Architecture:", platform.machine()); print("PyTorch:", torch.__version__); print("CUDA available:", torch.cuda.is_available()); print("CUDA device:", torch.cuda.get_device_name(0) if torch.cuda.is_available() else "N/A")'
 ```
+On the tested system, the output was:
+
+```output
+Architecture: aarch64
+PyTorch: 2.13.0+cu130
+CUDA available: False
+CUDA device: N/A
+```
+
+The PyTorch version can differ when you run an unpinned `pip` installation. `CUDA available: False` is expected on the CPU-only M9g instance, even if the wheel version contains a CUDA suffix.
 
 ## Install BenchMARL and VMAS
 
@@ -145,10 +184,23 @@ Verify the software stack:
 python -c 'import torch, torchrl, benchmarl, vmas; print("PyTorch:", torch.__version__); print("TorchRL: OK"); print("BenchMARL: OK"); print("VMAS: OK")'
 ```
 
+On the tested system, the output was:
+
+```output
+PyTorch: 2.13.0+cu130
+TorchRL: OK
+BenchMARL: OK
+VMAS: OK
+```
+
 Record the BenchMARL revision used for the experiment:
 
 ```bash
 git rev-parse HEAD
 ```
 
 Keep this revision with your experiment notes so you can reproduce the software environment later.
+
+## What you've accomplished
+
+You have verified the `aarch64` environment, sized the initial VMAS workload, and installed the training stack. Next, you will keep the training-frame budget consistent while adapting the vectorized environment count to the available CPUs.

content/learning-paths/servers-and-cloud-computing/train-mappo-navigation-arm-cloud/2-configure-training.md

@@ -9,6 +9,8 @@ layout: learningpathall
 
 ## Choose the agent count and devices
 
+MAPPO uses the centralized critic to learn from joint training information while each agent acts from its own observation. With `share_policy_params=true`, the three agents use the same actor parameters but receive different observations.
+
 Move to the BenchMARL repository:
 
 ```bash
@@ -39,6 +41,7 @@ Recreate the CPU sizing variables so the configuration works after a new SSH log
 export CORE_COUNT=$(nproc)
 export RESERVED_CPUS=1
 export WORKLOAD_CPUS=$((CORE_COUNT - RESERVED_CPUS))
+test "$WORKLOAD_CPUS" -ge 1 || { echo "This sizing policy needs at least 2 CPUs" >&2; exit 1; }
 export N_ENVS=$WORKLOAD_CPUS
 ```
 
@@ -49,13 +52,16 @@ export FRAMES_PER_ENV_PER_BATCH=100
 export FRAMES_PER_BATCH=$((N_ENVS * FRAMES_PER_ENV_PER_BATCH))
 ```
 
-Run 100 training batches:
+Use 1,910,000 frames as the target training budget. Calculate enough complete batches to meet or slightly exceed that target:
 
 ```bash
-export TRAINING_BATCHES=100
+export TARGET_MAX_FRAMES=1910000
+export TRAINING_BATCHES=$(((TARGET_MAX_FRAMES + FRAMES_PER_BATCH - 1) / FRAMES_PER_BATCH))
 export MAX_FRAMES=$((FRAMES_PER_BATCH * TRAINING_BATCHES))
 ```
 
+BenchMARL collects complete batches. `MAX_FRAMES` therefore equals the target on the 192-vCPU reference system and can be up to one batch larger on another instance. Keeping the target fixed prevents CPU count from reducing the amount of training.
+
 Evaluate every 20 batches using ten evaluation episodes:
 
 ```bash
@@ -66,10 +72,10 @@ export EVAL_EPISODES=10
 
 The reference sizing gives:
 
-| Total CPUs | Workload CPUs / environments | Frames per batch | Total frames | Evaluation interval |
-| ---: | ---: | ---: | ---: | ---: |
-| 64 | 63 | 6,300 | 630,000 | 126,000 |
-| 192 | 191 | 19,100 | 1,910,000 | 382,000 |
+| Total CPUs | Environments | Frames per batch | Training batches | Target frames | Actual frames | Evaluation interval |
+| ---: | ---: | ---: | ---: | ---: | ---: | ---: |
+| 64 | 63 | 6,300 | 304 | 1,910,000 | 1,915,200 | 126,000 |
+| 192 | 191 | 19,100 | 100 | 1,910,000 | 1,910,000 | 382,000 |
 
 {{% notice Note %}}
 When VMAS sampling runs on CUDA, tune `N_ENVS` for the GPU instead of deriving it from the CPU count.
@@ -90,8 +96,11 @@ Name the run so that the agent count, environment count, sampling device, and tr
 export RUN_NAME="agents_${AGENTS}__envs_${N_ENVS}__sampling_${SAMPLING_DEVICE}__train_${TRAIN_DEVICE}"
 export RUN_DIR="$OUTPUT_ROOT/$RUN_NAME"
 mkdir -p "$RUN_DIR"
+ln -sfn "$RUN_DIR" "$OUTPUT_ROOT/latest"
 ```
 
+The `latest` symbolic link gives later sections a stable way to recover the run directory after a new SSH login.
+
 ## Limit CPU thread parallelism
 
 Cap the main CPU thread pools at the number of workload CPUs:
@@ -105,18 +114,35 @@ export NUMEXPR_MAX_THREADS=$WORKLOAD_CPUS
 
 This avoids library thread pools using more CPU threads than the workload allocation.
 
+Record the software revision, installed packages, and shell configuration with the run:
+
+```bash
+export BENCHMARL_REVISION=$(git rev-parse HEAD)
+python -m pip freeze > "$RUN_DIR/software-versions.txt"
+declare -px \
+  AGENTS SAMPLING_DEVICE TRAIN_DEVICE \
+  CORE_COUNT RESERVED_CPUS WORKLOAD_CPUS N_ENVS \
+  FRAMES_PER_ENV_PER_BATCH FRAMES_PER_BATCH TARGET_MAX_FRAMES \
+  TRAINING_BATCHES MAX_FRAMES EVAL_EVERY_BATCHES EVAL_INTERVAL EVAL_EPISODES \
+  OUTPUT_ROOT RUN_NAME RUN_DIR \
+  OMP_NUM_THREADS MKL_NUM_THREADS OPENBLAS_NUM_THREADS NUMEXPR_MAX_THREADS \
+  BENCHMARL_REVISION > "$RUN_DIR/run.env"
+```
+
+`run.env` contains only the variables listed in the command. It does not copy credentials or the rest of your shell environment.
+
 ## Validate the configuration
 
 Print the complete configuration before training:
 
 ```bash
-echo "Agents=$AGENTS TotalCPUs=$CORE_COUNT ReservedCPUs=$RESERVED_CPUS WorkloadCPUs=$WORKLOAD_CPUS Environments=$N_ENVS FramesPerBatch=$FRAMES_PER_BATCH MaxFrames=$MAX_FRAMES EvalInterval=$EVAL_INTERVAL EvalEpisodes=$EVAL_EPISODES Sampling=$SAMPLING_DEVICE Training=$TRAIN_DEVICE"
+echo "Agents=$AGENTS TotalCPUs=$CORE_COUNT ReservedCPUs=$RESERVED_CPUS WorkloadCPUs=$WORKLOAD_CPUS Environments=$N_ENVS FramesPerBatch=$FRAMES_PER_BATCH TrainingBatches=$TRAINING_BATCHES MaxFrames=$MAX_FRAMES EvalInterval=$EVAL_INTERVAL EvalEpisodes=$EVAL_EPISODES Sampling=$SAMPLING_DEVICE Training=$TRAIN_DEVICE"
 ```
 
 For a 192-CPU system, the reference values are:
 
 ```output
-Agents=3 TotalCPUs=192 ReservedCPUs=1 WorkloadCPUs=191 Environments=191 FramesPerBatch=19100 MaxFrames=1910000 EvalInterval=382000 EvalEpisodes=10 Sampling=cpu Training=cpu
+Agents=3 TotalCPUs=192 ReservedCPUs=1 WorkloadCPUs=191 Environments=191 FramesPerBatch=19100 TrainingBatches=100 MaxFrames=1910000 EvalInterval=382000 EvalEpisodes=10 Sampling=cpu Training=cpu
 ```
 
 Do not start training if a required field is blank.
@@ -126,7 +152,31 @@ Do not start training if a required field is blank.
 Start MAPPO training:
 
 ```bash
-python benchmarl/run.py algorithm=mappo task=vmas/navigation task.n_agents="$AGENTS" experiment.sampling_device="$SAMPLING_DEVICE" experiment.train_device="$TRAIN_DEVICE" experiment.on_policy_n_envs_per_worker="$N_ENVS" experiment.on_policy_collected_frames_per_batch="$FRAMES_PER_BATCH" 'experiment.loggers=[csv]' experiment.render=false experiment.evaluation=true experiment.max_n_frames="$MAX_FRAMES" experiment.checkpoint_at_end=true experiment.prefer_continuous_actions=true experiment.evaluation_interval="$EVAL_INTERVAL" experiment.evaluation_episodes="$EVAL_EPISODES" experiment.save_folder="$RUN_DIR"
+python benchmarl/run.py \
+  algorithm=mappo \
+  task=vmas/navigation \
+  task.n_agents="$AGENTS" \
+  experiment.sampling_device="$SAMPLING_DEVICE" \
+  experiment.train_device="$TRAIN_DEVICE" \
+  experiment.on_policy_n_envs_per_worker="$N_ENVS" \
+  experiment.on_policy_collected_frames_per_batch="$FRAMES_PER_BATCH" \
+  'experiment.loggers=[csv]' \
+  experiment.render=false \
+  experiment.evaluation=true \
+  experiment.max_n_frames="$MAX_FRAMES" \
+  experiment.checkpoint_at_end=true \
+  experiment.prefer_continuous_actions=true \
+  experiment.evaluation_interval="$EVAL_INTERVAL" \
+  experiment.evaluation_episodes="$EVAL_EPISODES" \
+  experiment.save_folder="$RUN_DIR"
 ```
 
-BenchMARL performs training and periodic evaluation and saves a checkpoint when the run completes.
+This is the argument set used for the reference experiment. The tested BenchMARL and VMAS defaults define the 18-value observation and `256, 256` actor architecture checked by the exporter. BenchMARL saves a checkpoint when training completes.
+
+{{% notice Important %}}
+Training takes several hours on the reference system. Run the command in a persistent terminal session, such as `tmux`, so an SSH disconnection does not terminate the process. This tested command saves its checkpoint at the end of training.
+{{% /notice %}}
+
+## What you've accomplished
+
+You have configured a CPU-sized VMAS batch without changing the target training budget. BenchMARL now evaluates the policy periodically and saves the final checkpoint with the package versions, BenchMARL revision, and workload variables recorded alongside the run.

content/learning-paths/servers-and-cloud-computing/train-mappo-navigation-arm-cloud/3-validate-checkpoint.md

@@ -7,6 +7,19 @@ weight: 4
 layout: learningpathall
 ---
 
+## Restore the run configuration
+
+Activate the training environment and recover the saved variables. The `latest` link was created when you configured the run:
+
+```bash
+source $HOME/venvs/mappo/bin/activate
+export RUN_DIR=$HOME/mappo_navigation_runs/latest
+source "$RUN_DIR/run.env"
+cd $HOME/BenchMARL
+```
+
+If you want to validate an older run, set `RUN_DIR` to that run directory before you source `run.env`.
+
 ## Find the checkpoint
 
 BenchMARL creates an experiment directory below `RUN_DIR` with a structure similar to:
@@ -36,9 +49,12 @@ Display and verify the path:
 
 ```bash
 echo "$CHECKPOINT"
-test -f "$CHECKPOINT" && echo "Checkpoint found: $CHECKPOINT"
+test -n "$CHECKPOINT" && test -f "$CHECKPOINT" || { echo "No checkpoint found under $RUN_DIR" >&2; exit 1; }
+echo "Checkpoint found: $CHECKPOINT"
 ```
 
+The command stops the shell if no checkpoint exists, which prevents later commands from deriving paths from an empty value.
+
 ## Keep `config.pkl` with the checkpoint
 
 Determine the BenchMARL experiment directory:
@@ -86,6 +102,34 @@ python -c "import pickle; f=open('$SOURCE_EXPERIMENT_DIR/config.pkl','rb'); task
 
 This value comes from the trained experiment and prevents a checkpoint from being registered with a stale shell value for the agent count.
 
+Save the checkpoint variables so later sections can restore them after a new SSH login:
+
+```bash
+declare -px CHECKPOINT SOURCE_EXPERIMENT_DIR CHECKPOINT_AGENTS >> "$RUN_DIR/run.env"
+```
+
+## Evaluate the reloaded policy
+
+Run BenchMARL's evaluation entry point against the selected checkpoint:
+
+```bash
+python benchmarl/evaluate.py "$CHECKPOINT"
+```
+
+This command reconstructs the experiment from `config.pkl`, loads the trained state, and runs the saved evaluation configuration. A successful run proves that BenchMARL can execute the policy, while the earlier file checks prove only that the artifacts exist.
+
+List the CSV logs produced by the experiment:
+
+```bash
+find "$SOURCE_EXPERIMENT_DIR" -type f -name '*.csv' -print
+```
+
+Compare the first and final evaluation returns in these logs. Returns vary with the random seed and package versions, so this Learning Path does not use an unverified numeric threshold. A flat or falling return means that checkpoint creation succeeded but the policy did not demonstrate learning.
+
 {{% notice Security %}}
 Only load `config.pkl` from a training run you trust. Python pickle files can execute code when loaded.
 {{% /notice %}}
+
+## What you've accomplished
+
+You have selected a valid checkpoint, confirmed that its matching configuration is present, recovered deployment metadata, and reloaded the policy for evaluation. The checkpoint and `config.pkl` now form the portable BenchMARL experiment used by the optional GUI stage.

content/learning-paths/servers-and-cloud-computing/train-mappo-navigation-arm-cloud/4-deploy-gui.md

@@ -11,6 +11,8 @@ layout: learningpathall
 
 The MARL GUI used in this stage is a separate application from BenchMARL and VMAS. It is not created automatically by the training workflow.
 
+The GUI source and helper scripts are not included in the Arm Learning Paths repository. Treat this stage as optional unless you have the companion checkout described in the prerequisites. Checkpoint validation and actor export do not depend on the GUI.
+
 If you already have the companion GUI code, use that checkout and continue with the steps below.
 
 If you do not have the GUI code, you can either skip this stage and continue to the actor-export section, or create an equivalent visualization application. A compatible GUI should be able to:
@@ -48,8 +50,12 @@ The GUI reloads the full BenchMARL experiment, so use the same Python environmen
 
 ```bash
 source $HOME/venvs/mappo/bin/activate
+export RUN_DIR=$HOME/mappo_navigation_runs/latest
+source "$RUN_DIR/run.env"
 ```
 
+Sourcing `run.env` restores `CHECKPOINT`, `SOURCE_EXPERIMENT_DIR`, the agent count, and the device labels saved in the previous section.
+
 Set the GUI root to your local checkout:
 
 ```bash
@@ -197,16 +203,28 @@ Confirm that the output again contains:
 
 ## Start the GUI
 
-Start the application:
+Start the application on the cloud instance. Bind it to the loopback interface so it is not exposed directly through the instance network:
 
 ```bash
 cd "$GUI_ROOT"
 ```
 
 ```bash
-./run_demo.sh --host 0.0.0.0 --port 8045
+./run_demo.sh --host 127.0.0.1 --port 8045
+```
+
+From your local workstation, open an SSH tunnel to the instance:
+
+```bash
+ssh -L 8045:127.0.0.1:8045 ubuntu@INSTANCE_PUBLIC_IP
 ```
 
+Keep the SSH connection open and visit `http://127.0.0.1:8045` in your local browser. Replace `ubuntu` and `INSTANCE_PUBLIC_IP` with the SSH user and address for your instance.
+
+{{% notice Security %}}
+The SSH tunnel avoids opening TCP port 8045 to the internet. If you intentionally bind the GUI to `0.0.0.0`, restrict the cloud firewall or security-group rule to a trusted source address and confirm that the GUI's authentication is suitable for your environment.
+{{% /notice %}}
+
 Refresh the browser after registration. Select the model in this order:
 
 ```text
@@ -237,3 +255,7 @@ The GUI uses a single VMAS environment for interactive playback. This is indepen
 | GUI registry | `$GUI_ROOT/configs/models.yaml` | Drives model selection |
 
 The full BenchMARL checkpoint is the correct artifact for the GUI. For a downstream inference runtime, you can export only the actor parameters, as shown in the next section.
+
+## What you've accomplished
+
+You have preserved the BenchMARL directory layout, registered the checkpoint with the companion GUI, reloaded the deployed model, and accessed interactive playback through a secure tunnel. Next, you will extract the shared actor for a runtime that does not include BenchMARL.