contilab is a research framework for continual learning (also called lifelong learning or incremental learning): training a model on a sequence of tasks without catastrophic forgetting. It is built for empirical studies — fair, reproducible comparisons of methods on vision and language (LLM) benchmarks — and is designed around three goals:
- Speed. Several random seeds are trained simultaneously as one stacked-parameter ensemble
(
torch.func.vmap+functional_call), with optionaltorch.compileand bf16 autocast and a DataLoader-free, device-resident data pipeline. - Evaluation depth. 30+ metrics from the continual-learning literature — accuracy matrix, forgetting, backward/forward transfer, the stability gap, worst-case accuracy, linear probing and CKA, plasticity loss (dormant neurons, effective rank), task-recency bias, calibration, few-shot recovery, loss barriers, gradient interference and LLM general-ability retention — computed post hoc from stored raw data, with bootstrap confidence intervals, paired tests and Friedman/Nemenyi critical differences.
- Distillation and consolidation. Besides sequential training, contilab compares train-then-consolidate pipelines that merge per-task experts into one model with Task Arithmetic, TIES-Merging, multi-teacher knowledge distillation or teacher → student continual distillation (optionally into a smaller student).
Replay (DER++, Experience Replay) and tuned multi-teacher distillation lead; regularisation alone (EWC, LwF) stays at the Naive level in class-incremental learning. All 6 benchmarks: Results.
Documentation: https://jonpark0.github.io/contilab/ (FAQ · Getting started · 한국어)
| Area | What you get |
|---|---|
| Scenarios | Task-incremental (Task-IL), class-incremental (Class-IL), domain-incremental (Domain-IL) |
| Methods | Naive fine-tuning, Joint (upper bound), EWC (online), LwF, Experience Replay (ER), ER-ACE, DER++, A-GEM, GEM, LoRA-per-task, O-LoRA |
| Consolidation | Task Arithmetic, TIES-Merging, MagMax (full weights or LoRA ΔW), multi-teacher distillation with full / buffer / proxy transfer data, teacher → student continual distillation, smaller student architectures |
| Benchmarks | Split MNIST, Permuted MNIST, Split CIFAR-10, Split CIFAR-100, text5 (AG News → Yelp → Amazon → DBpedia → Yahoo Answers) with Hugging Face LMs (e.g. Pythia, Qwen2.5) |
| Metrics | 30+ quantitative metrics + figures (accuracy heatmaps, stability-gap curves, confusion, CKA, drift, t-SNE/PCA, loss barriers, critical-difference diagrams) |
| Statistics | Mean ± std over seeds, bootstrap 95 % CI, Wilcoxon / exact permutation tests, Friedman + Nemenyi CD, validation-based hyperparameter selection for sweeps |
| Engineering | Seed-parallel vmap ensembles, torch.compile, bf16, idempotent experiment matrix, per-task checkpoints and resume, failure isolation, divergence detection, Markdown/HTML reports |
git clone https://github.com/JonPark0/contilab && cd contilab
python3 -m venv .venv && .venv/bin/pip install -e '.[dev]'
# every method on tiny synthetic streams (under a minute on 4 CPU cores)
.venv/bin/contilab run configs/experiments/smoke_all.yaml
.venv/bin/contilab report runs/smoke_all # runs/smoke_all/report.md + report.html
# the CPU study: 16 methods × Split/Permuted MNIST, Split CIFAR-10/100, text5 (resumable; about 30 h)
.venv/bin/contilab run configs/experiments/cpu_main.yaml
.venv/bin/contilab run configs/experiments/cpu_main.yaml "matrix.preset=[split_mnist_class]" train.epochs=2
# add a metric later without re-training (raw data are stored)
.venv/bin/contilab metrics runs/cpu_main --add cka_final recovery
# throughput: eager vs compile × ensemble width
.venv/bin/contilab bench --model mlp --E 1,2,4,8 --compile off,onDatasets download from their original mirrors into $CONTILAB_CACHE (default ~/.cache/contilab);
pretrained language models come from the Hugging Face Hub.
| Method | Family | Reference |
|---|---|---|
| EWC (online) | regularisation | Kirkpatrick et al., PNAS 2017; Schwarz et al., ICML 2018 |
| LwF | distillation-based regularisation | Li & Hoiem, TPAMI 2017 |
| ER (reservoir) | replay | Chaudhry et al., 2019 (On Tiny Episodic Memories); Vitter 1985 |
| ER-ACE | replay with asymmetric cross-entropy | Caccia et al., ICLR 2022 |
| DER++ | replay + logit distillation | Buzzega et al., NeurIPS 2020 |
| A-GEM | gradient projection | Chaudhry et al., ICLR 2019 |
| GEM | gradient projection (one constraint per task) | Lopez-Paz & Ranzato, NeurIPS 2017 |
| LoRA-per-task | parameter isolation | Hu et al., ICLR 2022 |
| O-LoRA | orthogonal LoRA subspaces | Wang et al., Findings of EMNLP 2023 |
| Task Arithmetic | model merging | Ilharco et al., ICLR 2023 |
| TIES-Merging | model merging | Yadav et al., NeurIPS 2023 |
| MagMax | model merging (sequential fine-tuning, max-magnitude selection) | Marczak et al., ECCV 2024 |
| Multi-teacher distillation | consolidation | Hinton et al., 2015 |
| Teacher → student continual distillation | consolidation (KD + self-distillation on a replay buffer) | Hinton et al., 2015; Li & Hoiem, TPAMI 2017 |
| Axis | Metrics | References |
|---|---|---|
| Overall | ACC, Average Incremental Accuracy | Lopez-Paz & Ranzato 2017; Rebuffi et al. 2017 |
| Stability | BWT, forgetting, relative forgetting | Lopez-Paz & Ranzato 2017; Chaudhry et al. 2018 |
| Plasticity | FWT, intransigence, learning accuracy, LCA | Chaudhry et al. 2018/2019; Riemer et al. 2019 |
| Continual evaluation | min-ACC, WC-ACC, stability gap, windowed forgetting, anytime accuracy | De Lange et al. 2023; Caccia et al. 2022 |
| Representation | linear-probe forgetting (representation vs head), CKA, feature drift | Davari et al. 2022; Kornblith et al. 2019 |
| Plasticity loss | dormant-neuron ratio, effective rank, srank | Sokar et al. 2023; Kumar et al. 2021; Dohare et al. 2024 |
| Bias & calibration | task-recency bias, task confusion, ECE | Wu et al. 2019; Masana et al. 2022 |
| Recovery | few-shot relearning recovery (true vs spurious forgetting) | Zheng et al., ICLR 2025 |
| Landscape | linear-interpolation loss barrier, gradient interference | Mirzadeh et al. 2020 |
| LLMs | general-ability retention (perplexity drift) | Wang et al. 2023 (TRACE) |
| Cost | wall time, throughput, peak memory, FLOPs, trainable params, buffer/model bytes | Díaz-Rodríguez et al. 2018; Prabhu et al. 2023 |
Full definitions and formulas: Metrics.
Training throughput on 4 CPU cores (Intel Xeon @ 2.1 GHz with AMX), samples/s summed over ensemble members:
| Model | Baseline (fp32, eager, one seed) | Best configuration | Speed-up |
|---|---|---|---|
| MLP (2×400) | 10.8 k | fp32 eager, 8 seeds in one vmap ensemble: 40.7 k |
3.8× |
| Reduced ResNet-18 | 318 | bf16 autocast + torch.compile, one seed: 730 |
2.3× |
Small models gain most from training seeds together; convolutional models gain mostly from bf16. Each benchmark preset selects its fastest configuration. Details: Performance.
A CPU study of 16 methods on 6 benchmarks (432 runs: 6 seeds on the MNIST benchmarks, 3 on the others). Final average accuracy (mean over seeds):
| Method | Split MNIST Class-IL | Split MNIST Task-IL | Permuted MNIST | Split CIFAR-10 Class-IL | Split CIFAR-100 Class-IL | text5 + Pythia-70M |
|---|---|---|---|---|---|---|
| Naive | 0.196 | 0.980 | 0.805 | 0.182 | 0.046 | 0.097 |
| Joint (upper bound) | 0.950 | 0.993 | 0.909 | 0.462 | 0.120 | 0.529 |
| EWC | 0.196 | 0.981 | 0.832 | 0.159 | 0.041 | 0.105 |
| LwF | 0.228 | 0.990 | 0.817 | 0.171 | 0.047 | 0.105 |
| ER | 0.858 | 0.983 | 0.869 | 0.290 | 0.057 | 0.329 |
| ER-ACE | 0.875 | 0.986 | 0.883 | 0.393 | 0.103 | 0.377 |
| DER++ | 0.871 | 0.991 | 0.902 | 0.285 | 0.063 | 0.397 |
| A-GEM | 0.335 | 0.984 | 0.864 | 0.180 | 0.044 | 0.100 |
| GEM | 0.882 | 0.986 | 0.874 | 0.262 | 0.071 | 0.384 |
| LoRA-per-task | 0.184 | 0.986 | 0.619 | 0.129 | 0.015 | 0.024 |
| O-LoRA | 0.195 | 0.970 | 0.607 | 0.152 | 0.040 | 0.029 |
| Task Arithmetic | 0.531 | 0.979 | 0.570 | 0.151 | 0.011 | 0.034 |
| TIES-Merging | 0.561 | 0.983 | 0.677 | 0.160 | 0.013 | 0.033 |
| MagMax | 0.572 | 0.982 | 0.836 | 0.207 | 0.092 | 0.126 |
| Multi-teacher distillation | 0.863 | 0.986 | 0.773 | 0.267 | 0.040 | 0.452 |
| Teacher → student continual distillation | 0.586 | 0.980 | 0.818 | 0.229 | 0.030 | 0.331 |
- Ranking. Replay methods rank best after Joint: DER++ 3.3, ER-ACE 3.5 and GEM 4.2 (average ranks; Friedman p = 1.5·10⁻⁸). On Split MNIST Class-IL, GEM, ER-ACE, DER++ and multi-teacher distillation do not differ significantly over 6 seeds.
- ER-ACE is the best continual method on both CIFAR benchmarks (0.393 and 0.103 vs 0.290 and 0.057 for ER): its asymmetric loss cuts the bias towards the newest classes (0.70 → 0.24 on Split CIFAR-10).
- GEM vs A-GEM. One constraint per past task instead of A-GEM's averaged one lifts Split MNIST Class-IL from 0.335 to 0.616; the margin of the reference code adds the rest, up to 0.882.
- Distillation needed tuning. It needed validation-tuned defaults: a TIES-merged student and targets routed to each sample's teacher. With the v0.1 defaults it ranked last.
- Continual distillation stores no experts and still beats Naive on every Class- and Domain-IL benchmark except Split CIFAR-100.
- Regularisation stays at the Naive level in Class-IL even after tuning. EWC (λ = 30 on vision, chosen on validation accuracy, with its stiffness bounded under SGD) no longer diverges, but stays below Naive on CIFAR.
- Merging. MagMax, which merges sequentially fine-tuned models, is the only method without stored samples above Naive on every Class- and Domain-IL benchmark. Merging independently trained experts (Task Arithmetic, TIES) is not.
- O-LoRA does not lift LoRA above Naive in Class-IL: the shared classifier head still predicts only the newest classes.
- Split CIFAR-100 at 2 CPU epochs is a floor test: Joint reaches only 0.120, and ER-ACE (0.103) and MagMax (0.092) come closest.
- Language models. LoRA-per-task keeps the language model's general ability (perplexity +186 vs +7,481 for Naive) but learns little. At the design scale of text5 (2,000 / 500 texts per task instead of 1,000 / 200) the method order holds (rank correlation 0.97), and multi-teacher distillation reaches 0.477.
Setup, a validation-tuned comparison, a smaller-student experiment, text5 at the design scale, per-benchmark metrics (BWT, stability gap, recency bias, memory, wall time), figures and limitations: Results.
- Site: https://jonpark0.github.io/contilab/, with search and a Korean home page
- Wiki: https://github.com/JonPark0/contilab/wiki (the same pages; source in
docs/wiki/) - FAQ: which method to use, scenarios, the stability gap, language models, GPUs
- Per-run results of the CPU study:
docs/assets/results/data/and the Hugging Face dataset jonpark0/contilab-cpu-study - Changelog · design spec:
docs/design/2026-10-22-contilab-design.md
If contilab helps your research, please cite it (GitHub's Cite this repository uses CITATION.cff):
@software{contilab2026,
author = {Park, Yeochoon},
title = {contilab: a performance-first framework for continual learning benchmarking, distillation and model merging},
year = {2026},
version = {0.4.0},
doi = {10.5281/zenodo.23906803},
url = {https://github.com/JonPark0/contilab}
}Every release is archived on Zenodo. The DOI 10.5281/zenodo.23906803 stands for all versions and
resolves to the latest one; each release also has its own DOI, listed on the Zenodo page.
contilab은 지속 학습(Continual Learning, 연속 학습·평생 학습) 연구와 실증을 위한 PyTorch 프레임워크입니다. 여러 태스크를 순서대로 학습할 때 생기는 파국적 망각(catastrophic forgetting) 을 비전·언어(LLM) 모델에서 공정하게 비교합니다.
-
빠른 실험: 여러 시드를 하나의 앙상블로 동시에 학습합니다(
vmap). 여기에torch.compile, bf16, DataLoader 없는 디바이스 상주 데이터를 씁니다. -
풍부한 평가 지표 30여 종: 다음 지표들을 저장된 원자료에서 사후 계산하므로, 재학습 없이 지표를 추가할 수 있습니다.
- 정확도 행렬, BWT/FWT, 망각
- 안정성 격차(stability gap), 최악 정확도
- 선형 probe·CKA
- 가소성 손실
- 최근 태스크 편향, 보정(ECE)
- 복구성, 손실 장벽
- LLM 일반 능력 보존
-
통계: 부트스트랩 신뢰구간, 대응 검정, Friedman/Nemenyi 임계 차이를 제공합니다.
-
증류·병합: 태스크별 전문가 모델을 하나로 통합해 순차 학습과 비교합니다.
- 통합 방법: Task Arithmetic, TIES-Merging, MagMax, 다중 교사 지식 증류, 교사→학생 연속 증류
- 학생 모델은 교사보다 작은 아키텍처로도 지정할 수 있습니다.
-
하이퍼파라미터 선택: sweep 변형 중 하나를 검증 정확도로 자동 선택합니다(메서드별 그리드 지원).
-
지원 범위
- 메서드: EWC, LwF, ER, ER-ACE, DER++, A-GEM, GEM, 태스크별 LoRA, O-LoRA
- 벤치마크: Split/Permuted MNIST, Split CIFAR, text5(AG News·Yelp·Amazon·DBpedia·Yahoo)
-
CPU 실증 결과 (16개 메서드 × 6개 벤치마크, 432회: MNIST 벤치마크 6시드, 나머지 3시드): 자세한 내용은 Results를 참고하세요.
- Joint 다음으로 리플레이 계열(DER++, ER-ACE, GEM)의 평균 순위가 가장 높았습니다(3.3, 3.5, 4.2).
- ER-ACE는 두 CIFAR 벤치마크에서 연속 학습 방법 중 가장 높았습니다. 최신 클래스로 쏠리는 예측이 크게 줄었습니다.
- GEM은 A-GEM의 평균 제약 하나 대신 이전 태스크마다 제약을 둡니다. 이것만으로 Split MNIST Class-IL이 0.335에서 0.616으로 올랐고, 참조 코드의 margin을 더하면 0.882입니다.
- MagMax(순차 미세조정 모델 병합)는 샘플을 저장하지 않는 방법 중 유일하게 모든 Class-IL·Domain-IL 벤치마크에서 Naive보다 높았습니다. O-LoRA는 Class-IL에서 LoRA를 Naive 위로 올리지 못했습니다.
- 교사→학생 연속 증류는 전문가 모델을 저장하지 않고도 Split CIFAR-100을 제외한 Class-IL·Domain-IL 벤치마크에서 Naive보다 높았습니다.
- 정규화 계열(EWC, LwF)은 튜닝 후에도 Class-IL에서 Naive 수준에 머물렀습니다.
- Split CIFAR-100(CPU 2 epoch)은 Joint도 0.120에 그쳤고, ER-ACE(0.103)와 MagMax(0.092)가 가장 가까웠습니다.
- text5를 설계 규모(태스크당 학습 2,000·테스트 500개)로 다시 돌려도 메서드 순위는 거의 같았습니다(순위 상관 0.97).