Implement Red/Blue Self-Play/Co-training for CEC Experiments

[Dmujt]

Summary

• Core co-training and learned Red support
  • a35964b — Adds simultaneous Blue/Red training, frozen-opponent support, joint environments, multi-policy checkpoint bundles, and learned-policy matchups.
  • 1e36af6 — Restricts learned Red to observed information.
  • fcec351, c8ca0b5 — Fix agent initialization and align Red observation tests with discovered-host behavior.
  • 6bac6a8 — Fixes credit assignment across dormant Red periods and adds training diagnostics and historical-opponent evaluation.
• Additional Agent Architectures, e.g., MAPPO
  • e38ccad — Adds RNN/LSTM policy support.
  • 93fc881 — Preserves recurrent sequences during PPO minibatching.
  • 6fd56ea — Adds per-team architecture overrides and propagates recurrent state through evaluation.
  • 563b6cc — Adds MAPPO training, centralized critics, evaluation integration, and tests.
  • 6e19db9 — Extends MAPPO support and experiment configurations to both teams.
• Reproducible topology generation and sampling
  • 459c8f6 — Adds recipe-driven topology generation and separate training/evaluation seed ranges.
  • 776f71f — Connects pre-generated topology banks to evaluation.
  • c8a74df — Improves runtime topology generation performance.
  • 269ace7, 1269823 — Introduce sampling without replacement for training topology batches and update the documentation.
• Add multi-step evaluation and more metrics
  • f16c573 — Adds evaluation against scripted FSM/rule-based Red agents.
  • 381ea36 — Supports multiple ordered post-training evaluation jobs and standardizes evaluation argument naming.
  • 2239196 — Adds confidentiality, integrity, and availability resilience metrics alongside reward.
  • 9bdddb3 — Adds cross-seed evaluation and organizes post-training evaluation scripts.
  • e1451f8 — Adds support for rerunning an individual evaluation.
  • 072514d — Adds native CybORG benchmarks to post-training workflows.
  • 5c03ddb — Adds fixed-FSM evaluation of checkpoints used in cross-play.
• MLflow tracking and analysis
  • 7491534, d17daf8, 3159504 — Improves logging from remote runs.
  • abceb44, ef33422 — Fix MLflow port conflicts and duplicate-key errors.
  • b0df254, d20b62d — Add experiment-comparison plotting and combine CIA mean/std metrics in plots.
  • 6ddec51 — Adds co-training training/evaluation figures and analysis utilities.
• Training and evaluation performance/correctness
  • be19833, 0c48a7c — Batch learned-policy and scripted-Red evaluation; reduce experiment evaluation workloads.
  • f163b21 — Enables GPU execution for post-training evaluation.
  • 7f7e3b5 — Reduces joint-environment reset and evaluation overhead, with corresponding recipe budget adjustments.
  • 318b97a — Masks unavailable Blue actions during scripted evaluation.
  • 68282ba — Adds a parity-check launcher
  • c7f823e, 6f6d7b9, 9bf99ae— Maintenance
• Experiment recipes and comparisons
  • 2bdd4ff — Adds Blue/Red best-response experiment recipes.
  • 42d660a — Adds an experiment for asymmetric CC4 rule changes.
  • aa45515 — Updates IPPO/MAPPO environment-diversity comparison recipes.
  • a0ebe09 — Adds MAPPO experiments using pooled team observations instead of global-state critic inputs.
  • 9119576 — Consolidates co-training recipes under recipes/cotraining/ and updates references.
  • 4880c34 — Aligns recipe evaluation settings for consistent final runs.
  • 2b876e3 — Adds comparisons of standard and diversity-trained policies against the same learned Red opponents.
  • 4ab8019 — Matches recurrent and feedforward optimizer budgets.
• SOTA Comparison
  • b5afc45 — Adds configurable Red reward and Blue traffic-blocking rules for investigating co-training collapse.
  • a0190b1 — Adds enhanced Blue evidence observations across training and evaluation for HMARL comparisons.