Skip to content

Refine MAPPO training and actor export Learning Path #3696

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
armwaheed wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Refine MAPPO training and actor export Learning Path #3696

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
59 changes: 40 additions & 19 deletions ...s-and-cloud-computing/train-mappo-navigation-arm-cloud/1-prepare-environment.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ layout: learningpathall

## Inspect the Arm cloud instance

Confirm that the instance uses the Arm64 architecture:
Confirm that the instance reports the `aarch64` architecture:

```bash
uname -m
Expand All @@ -36,9 +36,10 @@ lscpu
Save the CPU count and reserve one CPU for operating-system and runtime activity:

```bash
export CORE_COUNT=$(nproc)
CORE_COUNT="$(nproc)"
export CORE_COUNT
export RESERVED_CPUS=1
export WORKLOAD_CPUS=$((CORE_COUNT - RESERVED_CPUS))
export WORKLOAD_CPUS=$((CORE_COUNT > RESERVED_CPUS ? CORE_COUNT - RESERVED_CPUS : 1))
```

Verify the values:
Expand All @@ -56,14 +57,14 @@ The agent count and vectorized environment count control different parts of the

For example, three agents and 191 environments means that VMAS simulates 191 navigation worlds concurrently, with three agents in each world.

For CPU sampling in this Learning Path, use one VMAS environment for each workload CPU:
For the reference run, start with one VMAS environment for each workload CPU:

```bash
export N_ENVS=$WORKLOAD_CPUS
export N_ENVS="$WORKLOAD_CPUS"
```

{{% notice Note %}}
This is a workload-sizing policy. VMAS uses vectorized PyTorch operations, so an environment is not permanently mapped to one operating-system thread.
This is a starting point, not a claim that one environment maps to one operating-system thread. VMAS applies vectorized PyTorch operations across the environment batch. Reduce `N_ENVS` if memory pressure causes swapping, and compare throughput before increasing it.
{{% /notice %}}

## Install system packages
Expand All @@ -89,13 +90,13 @@ sudo apt-get install -y git build-essential python3-pip python3-dev pkg-config c
Create a Python environment:

```bash
python3.12 -m venv $HOME/venvs/mappo
python3.12 -m venv "$HOME/venvs/mappo"
```

Activate it:

```bash
source $HOME/venvs/mappo/bin/activate
source "$HOME/venvs/mappo/bin/activate"
```

Confirm the active Python interpreter:
Expand All @@ -110,10 +111,16 @@ Upgrade the Python packaging tools:
python -m pip install --upgrade pip setuptools wheel packaging
```

Install PyTorch:
Install pinned PyTorch, TorchRL, TensorDict, and VMAS versions:

```bash
python -m pip install torch torchvision torchaudio
python -m pip install \
"torch==2.8.0" \
"torchvision==0.23.0" \
"torchaudio==2.8.0" \
"torchrl==0.10.1" \
"tensordict==0.10.0" \
"vmas==1.5.2"
```

Verify the installation:
Expand All @@ -124,25 +131,35 @@ python -c 'import platform, torch; print("Architecture:", platform.machine()); p

## Install BenchMARL and VMAS

Clone BenchMARL:
Clone the pinned BenchMARL revision:

```bash
cd $HOME
git clone https://github.com/facebookresearch/BenchMARL.git
cd $HOME/BenchMARL
export BENCHMARL_ROOT="$HOME/BenchMARL"
git clone --filter=blob:none --no-checkout --depth 1 \
https://github.com/facebookresearch/BenchMARL.git \
"$BENCHMARL_ROOT"
git -C "$BENCHMARL_ROOT" fetch --depth 1 origin \
65d649d80e0bdcbdbe2c5d6a3f02dbfed8f0bec1
git -C "$BENCHMARL_ROOT" checkout --detach \
65d649d80e0bdcbdbe2c5d6a3f02dbfed8f0bec1
cd "$BENCHMARL_ROOT"
```

Install BenchMARL and VMAS:
Install BenchMARL and its remaining dependencies:

```bash
python -m pip install -e .
python -m pip install vmas
python -m pip install -e ".[vmas]"
```

Verify the software stack:

```bash
python -c 'import torch, torchrl, benchmarl, vmas; print("PyTorch:", torch.__version__); print("TorchRL: OK"); print("BenchMARL: OK"); print("VMAS: OK")'
python - <<'PY'
from importlib.metadata import version

for package in ("torch", "torchrl", "tensordict", "vmas", "benchmarl"):
print(f"{package}: {version(package)}")
PY
```

Record the BenchMARL revision used for the experiment:
Expand All @@ -151,4 +168,8 @@ Record the BenchMARL revision used for the experiment:
git rev-parse HEAD
```

Keep this revision with your experiment notes so you can reproduce the software environment later.
The revision must be `65d649d80e0bdcbdbe2c5d6a3f02dbfed8f0bec1`. The version pins and revision keep the checkpoint layout and exporter assumptions reproducible.

## What you've accomplished

You have validated the Arm cloud instance, selected an explicit workload size, and installed a pinned training stack. Next, you will configure and run the MAPPO experiment.
49 changes: 36 additions & 13 deletions ...rs-and-cloud-computing/train-mappo-navigation-arm-cloud/2-configure-training.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,8 +12,9 @@ layout: learningpathall
Move to the BenchMARL repository:

```bash
source $HOME/venvs/mappo/bin/activate
cd $HOME/BenchMARL
source "$HOME/venvs/mappo/bin/activate"
export BENCHMARL_ROOT="$HOME/BenchMARL"
cd "$BENCHMARL_ROOT"
```

Use three agents for the reference experiment:
Expand All @@ -29,17 +30,18 @@ export SAMPLING_DEVICE=cpu
export TRAIN_DEVICE=cpu
```

BenchMARL configures sampling and training devices independently. You can also test `sampling=cpu, train=cuda` or `sampling=cuda, train=cuda` on a compatible system.
BenchMARL configures sampling and training devices independently. This Learning Path keeps both workloads on the Arm CPU so the reference configuration is reproducible.

## Size the CPU sampling workload

Recreate the CPU sizing variables so the configuration works after a new SSH login:

```bash
export CORE_COUNT=$(nproc)
CORE_COUNT="$(nproc)"
export CORE_COUNT
export RESERVED_CPUS=1
export WORKLOAD_CPUS=$((CORE_COUNT - RESERVED_CPUS))
export N_ENVS=$WORKLOAD_CPUS
export WORKLOAD_CPUS=$((CORE_COUNT > RESERVED_CPUS ? CORE_COUNT - RESERVED_CPUS : 1))
export N_ENVS="$WORKLOAD_CPUS"
```

Collect 100 frames from each environment before every MAPPO update:
Expand Down Expand Up @@ -94,16 +96,16 @@ mkdir -p "$RUN_DIR"

## Limit CPU thread parallelism

Cap the main CPU thread pools at the number of workload CPUs:
Give PyTorch access to the workload CPUs and keep secondary numerical libraries single-threaded:

```bash
export OMP_NUM_THREADS=$WORKLOAD_CPUS
export MKL_NUM_THREADS=$WORKLOAD_CPUS
export OPENBLAS_NUM_THREADS=$WORKLOAD_CPUS
export NUMEXPR_MAX_THREADS=$WORKLOAD_CPUS
export OMP_NUM_THREADS="$WORKLOAD_CPUS"
export MKL_NUM_THREADS="$WORKLOAD_CPUS"
export OPENBLAS_NUM_THREADS=1
export NUMEXPR_MAX_THREADS=1
```

This avoids library thread pools using more CPU threads than the workload allocation.
This avoids nested OpenBLAS or NumExpr pools competing with PyTorch for every CPU. Monitor memory use and frames per second during the first batches. If the instance swaps or throughput drops, stop the run and retry with a smaller `N_ENVS`.

## Validate the configuration

Expand All @@ -126,7 +128,28 @@ Do not start training if a required field is blank.
Start MAPPO training:

```bash
python benchmarl/run.py algorithm=mappo task=vmas/navigation task.n_agents="$AGENTS" experiment.sampling_device="$SAMPLING_DEVICE" experiment.train_device="$TRAIN_DEVICE" experiment.on_policy_n_envs_per_worker="$N_ENVS" experiment.on_policy_collected_frames_per_batch="$FRAMES_PER_BATCH" 'experiment.loggers=[csv]' experiment.render=false experiment.evaluation=true experiment.max_n_frames="$MAX_FRAMES" experiment.checkpoint_at_end=true experiment.prefer_continuous_actions=true experiment.evaluation_interval="$EVAL_INTERVAL" experiment.evaluation_episodes="$EVAL_EPISODES" experiment.save_folder="$RUN_DIR"
python benchmarl/run.py \
algorithm=mappo \
task=vmas/navigation \
task.n_agents="$AGENTS" \
experiment.sampling_device="$SAMPLING_DEVICE" \
experiment.train_device="$TRAIN_DEVICE" \
experiment.on_policy_n_envs_per_worker="$N_ENVS" \
experiment.on_policy_collected_frames_per_batch="$FRAMES_PER_BATCH" \
'experiment.loggers=[csv]' \
experiment.create_json=true \
experiment.render=false \
experiment.evaluation=true \
experiment.max_n_frames="$MAX_FRAMES" \
experiment.checkpoint_at_end=true \
experiment.prefer_continuous_actions=true \
experiment.evaluation_interval="$EVAL_INTERVAL" \
experiment.evaluation_episodes="$EVAL_EPISODES" \
experiment.save_folder="$RUN_DIR"
```

BenchMARL performs training and periodic evaluation and saves a checkpoint when the run completes.

## What you've accomplished

You have configured a CPU-only MAPPO run with bounded thread pools, periodic evaluation, and machine-readable evaluation output. Next, you will inspect the returns and validate the saved checkpoint.
Loading
Loading