A continual-learning architecture that restructures itself — heads split and merge, pathways self-modulate, and slow-pathway weights are selectively consolidated across tasks.
Catastrophic forgetting is what got me into continual learning in the first place. Train a network on one task, then another, then another, and it tends to quietly overwrite what it already knew. Most fixes I'd come across, EWC, DER++, protect the old weights after training is done, without changing how the model learns the new task in the first place. I wanted to try the opposite: let the architecture itself adapt as it goes, instead of bolting protection on afterward. That's what ARIA is.
ARIA combines four differentiable mechanisms and one post-training consolidation step:
Attention heads split when specialised and merge when redundant, dynamically adjusting the head count during training. Heads are pre-allocated up to n_heads_max; a boolean mask activates them. Split/merge decisions respect a per-head cooldown to prevent oscillation. Viabilities are softmax-normalised so total output magnitude is constant across head counts.
Each MLP layer runs two pathways — fast (volatile, high learning rate) and slow (stable, consolidated) — gated by a per-token scalar π ∈ (0,1). The gate is learned, not fixed. A specialisation loss pushes π toward 0 or 1 (bimodal), activated only after warmup_steps steps. Slow-pathway gradients are multiplied by (1 − π̄), protecting consolidated representations during high-plasticity phases.
A global latent vector z ∈ ℝ^G is co-optimised with model weights. It is decoded into:
- Per-layer skip probabilities (stochastic depth)
- Attention temperature (sharpness control)
- FiLM scale/shift (affine conditioning of all block outputs)
FiLM conditioning (γ·h + β) provides stronger architectural signal than additive injection because it can rescale entire feature dimensions, not just add a bias.
Predicts a per-layer compute budget b_l ∈ [0,1] from raw input statistics (standard deviation, entropy proxy, range). High-complexity inputs get full compute; simple inputs are partially short-circuited.
After each task, Fisher information is estimated over the slow-pathway weights only (not all weights as in EWC). The fast pathway remains unconstrained — it adapts freely to new tasks. SPC uses 50% fewer Fisher parameters than standard EWC while targeting exactly the weights most responsible for retaining old task knowledge.
Task stream → Input projection → [AGV conditioning]
↓
Block_1: MA + PG-MLP + CBA budget gating
Block_2: ...
Block_L: ...
↓
Task-specific linear head
Real results from 5-seed GPU runs on parameter-matched baselines (~3.77M params for ARIA).
| Model | Avg Acc | Forgetting |
|---|---|---|
| ARIA+SPC | 98.50 ± 0.32% | 0.86 ± 0.34% |
| ARIA-noSPC | 98.54 ± 0.33% | 1.06 ± 0.35% |
| EWC | 97.35 ± 2.04% | 2.29 ± 2.78% |
| DER++ | 75.63 ± 6.04% | 30.18 ± 7.55% |
| StaticMLP | 76.91 ± 3.02% | 28.45 ± 3.71% |
Key results:
- ARIA+SPC beats EWC by +1.15 points accuracy and 62% less forgetting ✓
- ARIA-noSPC achieves highest accuracy (98.54%) — the architecture alone outperforms EWC
- SPC reduces forgetting by 19% (1.06% → 0.86%) with minimal accuracy cost
- DER++ and StaticMLP show severe catastrophic forgetting (~30%) without continual learning safeguards
| Model | Avg Acc | Forgetting |
|---|---|---|
| EWC | 78.92 ± 0.51% | 0.06 ± 0.03% |
| StaticMLP | 73.93 ± 0.30% | 13.54 ± 0.46% |
| ARIA-noSPC | 70.44 ± 0.69% | 4.74 ± 0.77% |
| DER++ | 69.62 ± 0.68% | 19.75 ± 1.00% |
| ARIA+SPC | 68.24 ± 1.91% | 5.40 ± 1.39% |
Key results:
- EWC dominates on CIFAR-10 accuracy (harder benchmark); ARIA-v2 reduced forgetting by 56% vs ARIA-v1 (12.28% → 5.40%) ✓
- ARIA-noSPC achieves competitive forgetting (4.74%), suggesting slow-pathway consolidation is effective
- ARIA trades accuracy for low forgetting on this benchmark — a design choice reflecting the plasticity-stability tradeoff
- Results are honest and reproducible; see
paper/aria.pdffor full analysis
git clone https://github.com/rsd-darshan/ARIA.git
cd ARIA
pip install -e .For development (tests included):
pip install -e ".[dev]"import aria
aria.set_seed(42)
device = aria.get_device()
cfg = aria.ARIAConfig(input_dim=784, n_classes=2, d_model=256, n_layers=4)
tasks = aria.get_split_mnist_tasks(data_dir="./data", batch_size=64)
matrix = aria.train_aria(
cfg = cfg,
tasks = tasks,
device = device,
epochs_per_task = 5,
use_spc = True,
verbose = True,
)
from aria.metrics import compute_metrics
m = compute_metrics(matrix)
print(f"Avg accuracy : {m['avg_acc']:.3f}")
print(f"Forgetting : {m['forgetting']:.3f}")
print(f"BWT : {m['bwt']:.3f}")# Full multi-seed evaluation on Split-MNIST
python scripts/main.py --benchmark split_mnist --seeds 42 123 999 7 2024 --epochs 5
# Full multi-seed evaluation on Split-CIFAR-10
python scripts/main.py --benchmark split_cifar10 --seeds 42 123 999 7 2024 --epochs 10
# Ablation study (component contributions)
python scripts/ablation.py --seeds 42 123 999 --epochs 5Output:
results/results_table.json— machine-readable metric tableresults/figures/— accuracy curves, summary bars, forgetting heatmap, ablation waterfall
pytest tests/ -v -m "not integration" # unit tests (no data download)
pytest tests/ -v # all testsARIA/
├── aria/ # importable package
│ ├── __init__.py # public API
│ ├── model.py # ARIA, StaticMLP, EWCWrapper, DERPlusPlus
│ ├── train.py # per-model training loops
│ ├── data.py # Split-MNIST, Split-CIFAR-10 loaders
│ ├── metrics.py # avg_acc, BWT, FWT, forgetting
│ ├── evaluate.py # multi-seed harness + summary_table
│ └── plot.py # publication-quality figures
├── scripts/
│ ├── main.py # main evaluation entry point
│ └── ablation.py # component ablation study
├── examples/
│ ├── split_mnist_quickstart.py
│ └── split_cifar10_quickstart.py
├── tests/
│ ├── test_model.py
│ ├── test_metrics.py
│ └── test_train.py
├── paper/
│ └── aria.tex # full research paper
├── results/ # curated results and figures (gitignored: raw artifacts)
├── setup.py
├── pyproject.toml
├── requirements.txt
└── .github/workflows/ci.yml
- Python 3.9+ and PyTorch 2.0+.
- Fix seeds with
aria.set_seed(seed). - Recommended multi-seed command (matches paper):
python scripts/main.py --benchmark split_mnist --seeds 42 123 999 7 2024I didn't want to stop at computer vision, so I ported three of ARIA's core ideas to Hugging Face's TRL library for continual learning in LLM fine-tuning. It's called aria-trl:
- PlasticityGatedMLP: Fast/slow pathways in transformer FFN layers
- Slow-Pathway Consolidation (SPC): Fisher-weighted regularization during sequential task training
- Task-Specific Adapters: Lightweight, frozen residual modules per task
If you're fine-tuning LLMs on multiple tasks and want to prevent catastrophic forgetting, try aria-trl:
pip install aria-trlSee aria-trl README for quickstart and working example on DistilGPT2.
Before ARIA, I built NCG (Novelty-triggered Capacity Growth), a different attempt at the same forgetting problem. Where ARIA restructures its architecture proactively, NCG grows capacity reactively, adding neurons when novelty is high and accuracy plateaus. Roughly how they compare:
| Dimension | NCG | ARIA |
|---|---|---|
| Core mechanism | Reactive capacity growth: add 64 neurons when novelty is low and accuracy plateaus | Proactive architecture: restructure heads dynamically, route through learned fast/slow pathways |
| Model architecture | Simple 2-layer MLP with expandable hidden layer | 4-layer transformer-style with 4 cores (MA, PG-MLP, AGV, CBA) |
| Memory strategy | New tasks get new neurons (capacity growth) + gated knowledge buffer K | Dual pathways: fast pathway plastic, slow pathway consolidated via Fisher (SPC) |
| Meta-parameters | Learned online: α (exploration), β (complexity), λ (regularization) via bi-level optimization | Fixed and differentiable: genome vector z, skip probabilities, FiLM scale/shift |
| Split-MNIST accuracy vs EWC | 55.1% (EWC wins) | 98.5% (ARIA wins) |
| Split-MNIST forgetting | ~33% | 0.86% |
| Split-CIFAR-10 forgetting | ~8.4% | 5.4% (ARIA-v2) |
Philosophy:
- NCG: "When should we grow?" — answers with a learned growth trigger based on novelty and convergence plateaus. Simple, interpretable, but limited by 2-layer MLP expressiveness.
- ARIA: "How should we structure learning?" — answers with joint adaptation of attention heads, gating, genome, and budget allocation. The architecture itself balances plasticity (fast pathway) and stability (slow pathway consolidation).
When to use which:
- NCG: If you need interpretability and a minimal model, or if your domain favors capacity growth (e.g., sparse, tree-structured tasks).
- ARIA: If you need strong continual learning performance, can tolerate architectural complexity, and want the model to learn its own capacity allocation strategy.
- Evaluated on image classification continual-learning benchmarks; larger-scale and language settings are future work.
- Morphogenesis trigger currently uses viability scores; grad-norm-based triggers are available in
archive/files/aria_train_v4.pyand may be re-integrated. - SPC Fisher estimation is diagonal (standard approximation); full-matrix or Kronecker-factored Fisher is future work.
- aria.pdf — compiled from
paper/aria.tex
@article{poudel2026aria,
title = {Adaptive Recurrent Intelligence Architecture: Morphogenic Attention and Slow-Pathway Consolidation for Continual Learning},
author = {Poudel, Darshan},
year = {2026},
note = {Preprint. Under review.},
url = {https://github.com/rsd-darshan/ARIA}
}MIT License. See LICENSE.