A research-oriented PyTorch reproduction of Rasul et al. (ICML 2021)
This project was jointly developed by:
Both authors collaboratively contributed to the design, implementation, and development of this project.
School of Electrical and Computer Engineering, University of Tehran — 2026
Paper · PDF · Original authors' implementation · Quick start · Results
Scope. This repository implements the TimeGrad methodology in a modular PyTorch codebase rather than importing the original PyTorchTS estimator. It covers five of the six datasets reported by Rasul, Seward, Schuster, and Vollgraf: Exchange, Solar, Electricity, Traffic, and Taxi. The archived metrics displayed here come from one saved run per dataset, whereas the ICML paper reports aggregates over ten independent runs. They must not be interpreted as a statistically matched reproduction of Table 2.
Figure 1. Ratio of archived single-run normalized CRPSsum to the ten-run TimeGrad mean reported in Table 2 of the paper. Values below 1 indicate lower recorded error; this is a descriptive comparison, not a significance test. Data: benchmark_snapshot.csv.
- Research motivation
- Mathematical formulation
- Architecture and implementation
- Benchmark datasets
- Evaluation protocol and metrics
- Experimental results
- Getting started
- Reproducibility, fidelity, and limitations
- Repository structure
- References and citation
Multivariate forecasting requires a model of future joint distributions, not merely point estimates for individually predicted series. In applications such as energy demand, traffic systems, and financial analysis, marginal uncertainty can be insufficient: dependencies between dimensions affect the distribution of aggregated demand, simultaneous extremes, and downstream decisions.
TimeGrad [1] separates this problem into two coupled components:
- An autoregressive recurrent state summarizes historical observations and known temporal covariates.
- A conditional denoising diffusion process models the full multivariate observation distribution at each time step, given that recurrent state.
This architecture replaces a restrictive parametric emission head (e.g., a diagonal Gaussian or low-rank Gaussian) with a learned, non-Gaussian generative process. Joint future scenarios are generated sequentially: each newly sampled vector is fed back into the temporal model before the next forecast step.
| Component | Implementation | Location |
|---|---|---|
| Temporal dynamics | Two-layer LSTM, hidden size 40 | models/model.py |
| Conditional emission | DDPM with 100 noise levels | models/diffusion.py |
| Noise predictor | Eight-block gated, dilated 1-D WaveNet | models/components.py |
| Historical context | Dataset-specific lags, Fourier calendar features, dimension embeddings | data/data_loader.py, models/model.py |
| Scale handling | Per-dimension context mean absolute scaling; disabled for Traffic | models/model.py |
| Optimization | Adam, validation-based model selection, gradient clipping, optional CUDA AMP | scripts/train.py |
| Probabilistic assessment | Rolling windows; 100 trajectories; normalized 19-quantile CRPSsum | scripts/evaluate.py, utils/metrics.py |
| Experimental records | Saved evaluation JSON, training traces, and forecast figures in the supplied archive | outputs/ (normally ignored by Git) |
Consider a multivariate stochastic process with D jointly observed dimensions. At forecast origin T, we observe the history up to T and seek an H-step predictive joint distribution. Denote an unnoised observation by the diffusion superscript 0:
The chain rule gives the temporal autoregressive decomposition:
Here c_t comprises known-in-advance time covariates. The distribution of the entire D-dimensional vector at each step is generated jointly, rather than assuming independent dimensions.
Let L be the lag set, C the recurrent context length, and s a per-dimension scale estimated exclusively from the available context. For target dimension j:
The LSTM input concatenates normalized lagged observations, learned dimension-identity embeddings, and calendar Fourier features. With a compact notation for the input construction:
The implementation uses a two-layer LSTM with 40 hidden units and maps its state to a 100-dimensional conditioner. The time features are sinusoidal encodings of frequency-dependent calendar attributes. The lag configuration is dataset-specific; it is not universally [1, 24, 168].
Importantly, the conditioning state for time t uses past observations through lagged inputs; the current observation x_t is the target of the conditional denoising objective, not a leaked input.
Within each forecast time step, let n index the diffusion stage, from 1 to N. The forward process gradually perturbs a scaled clean vector with isotropic Gaussian noise:
With alpha_n = 1 - beta_n and the cumulative product bar_alpha_n, its marginal is available in closed form:
The fixed configuration uses 100 diffusion steps with a linearly increasing variance schedule from 0.0001 to 0.1. In code, steps are indexed from 0 to 99; the equations use the conventional one-based indexing.
Figure 2. Analytical visualization of the configured forward process; this is a deterministic schedule, not a learned experimental result.
The denoiser is a neural map conditioned on the noisy multivariate sample, diffusion stage, and autoregressive state:
Instead of directly estimating the reverse Gaussian mean, the model predicts injected noise. The reverse mean follows the DDPM reparameterization [2]:
This implementation samples using the fixed forward-posterior variance:
At n = 1, the last update is deterministic (z = 0). Each future vector begins from fresh standard Gaussian noise at n = N and is iteratively denoised to n = 0. The learned distribution is thereby represented implicitly by samples rather than a closed-form observation likelihood.
Training uses the standard simplified epsilon-prediction DDPM loss, conditional on the LSTM state. It uniformly samples the diffusion level and Gaussian perturbation:
The equation is understood in normalized observation space, after applying the context-derived scale. The implementation averages mean-squared error over the batch, training target times, and dimensions. It trains on the context-plus-prediction target positions with teacher forcing; inference replaces future ground truth with recursively sampled observations.
This simplified unweighted objective is related to, but is not numerically identical to, the fully weighted variational negative-log-likelihood bound derived in [1, 2].
The WaveNet-style denoiser operates on the target dimension axis: a noisy vector is represented as (batch, 1, D) before conditional, circularly padded 1-D convolutions. Dimensions have a fixed ordering; this is a practical architectural inductive bias, not a claim that all neighboring dataset dimensions are physically adjacent.
| Property | Value | Interpretation |
|---|---|---|
| LSTM | 2 layers × 40 hidden units | Autoregressive historical representation |
| RNN dropout | 0.1 | Between recurrent layers |
| Dimension embeddings | 1 learned scalar per dimension | Fixed identity features, repeated across time |
| Conditioner projection | 40 → 100 | Diffusion-network conditional representation |
| Diffusion Fourier embedding | 16 sine + 16 cosine features | Discrete noise-level representation; 500-entry lookup table |
| Diffusion embedding projection | 32 → 64 → 64 | Learned SiLU transformations |
| WaveNet residual stack | 8 blocks, 8 channels | Gated tanh/sigmoid activation and skip accumulation |
| Convolution dilation | 1, 2, 1, 2, 1, 2, 1, 2 |
Two-stage dilation cycle, circular padding |
| Diffusion levels | 100 | Linear beta schedule; reverse-posterior variance |
| Sampling | 100 forecast trajectories | Monte Carlo quantiles and interval estimation |
The conditioner is projected from the LSTM state and upsampled across dimensions. The gated residual blocks combine the noised sample, diffusion-level embedding, and recurrent conditioning signal; residual/skip paths are aggregated to estimate injected Gaussian noise. The output projection is initialized to zero.
| Tensor | Shape | Description |
|---|---|---|
history |
(B, context_length + max(lags), D) |
Sufficient past for all lagged features |
future |
(B, H, D) |
Teacher-forced targets during training |
time_features |
(B, C + H, F) |
Known calendar features for objective positions |
conditioning |
(B × (C + H), 100) |
Flattened per-step recurrent conditioning |
noisy_target |
(B × (C + H), 1, D) |
DDPM training inputs |
forecast |
(B, S, H, D) |
Sampled future trajectories |
Indexing distinction: the history tensor contains
context_length + max(lags)observations, but only its most recentcontext_lengthtarget positions are used for context scaling and warm-up conditioning. Earlier points support lag retrieval.
The paper's Table 1 defines six benchmarks. This code registers five of them using the GluonTS benchmark names below; Wikipedia is not implemented.
| Dataset | GluonTS dataset name | Dimensions D |
Frequency | Training steps | Horizon H |
Test windows | Scaling |
|---|---|---|---|---|---|---|---|
| Exchange | exchange_rate_nips |
8 | Business day | 6,071 | 30 | 5 | Yes |
| Solar | solar_nips |
137 | Hourly | 7,009 | 24 | 7 | Yes |
| Electricity | electricity_nips |
370 | Hourly | 5,833 | 24 | 7 | Yes |
| Traffic | traffic_nips |
963 | Hourly | 4,001 | 24 | 7 | No |
| Taxi | taxi_30min |
1,214 | 30-minute | 1,488 | 24 | 56 | Yes |
Pipeline. data/data_loader.py retrieves the preprocessed GluonTS datasets, aligns individual component series, constructs multivariate observations and rolling evaluation windows, and caches the resulting arrays. It also produces frequency-aware Fourier covariates (calendar hour, weekday, minute, or day-of-year as applicable).
Dataset-specific lag sets are specified in config/: Exchange uses [1, 2]; Solar, Electricity, and Traffic use [1, 24, 168]; Taxi uses [1, 4, 12, 24, 48]. The Taxi profile additionally specifies a one-step validation stride, leading to overlapping validation windows. This is a repository-level setting and should be reported explicitly in any experimental comparison.
- Train on sliding context/future windows drawn from the available training prefix.
- Reserve trailing training observations for chronological validation, select a checkpoint by validation diffusion loss, and retain the best model state.
- At each official test forecast origin, generate
S = 100joint trajectories of lengthH. - Evaluate probabilistic scores and marginal prediction-interval coverage across the rolling windows.
The current active training script completes step 2 and saves its selected checkpoint. Although the configuration contains validation.retrain_full: true, the active code does not subsequently refit on the complete training split. Some archived history files and checkpoints have a retrain_full phase from a different/earlier execution path. See Reproducibility, fidelity, and limitations before comparing new runs with the supplied archived results.
For a univariate predictive cumulative distribution function F and observation y, the continuous ranked probability score is:
For multivariate trajectories, the paper evaluates the distribution of the sum across all dimensions, retaining the contribution of cross-dimensional dependence. At test window w and horizon step t, define:
The reported CRPS_sum field in this repository uses the normalized 19-quantile approximation employed by the authors' GluonTS-style evaluator. For empirical quantiles q = 0.05, 0.10, ..., 0.95, with pinball loss rho, it is:
The repository separately reports the exact empirical-sample CRPS identity, normalized analogously, under empirical_CRPS_sum:
These are two different estimators/approximations and should not be silently interchanged when reporting results. Moreover, aggregate CRPS tests the distribution of the cross-sectional sum, not all aspects of the full multivariate dependence structure; full-joint assessment would require additional multivariate diagnostics (for example, an energy score or variogram score).
Other metrics in evaluation.json are normalized deviation (ND), normalized RMSE (NRMSE), and marginal central 50%/90% interval coverage (across dimensions, horizons and test windows). Coverage is not a joint multivariate guarantee.
| Benchmark | Archived CRPSsum ↓ | Paper TimeGrad CRPSsum ↓ | Archived / paper | Empirical 90% coverage |
|---|---|---|---|---|
| Exchange | 0.00512 | 0.006 ± 0.001 | 0.85× | 79.0% |
| Solar | 0.35074 | 0.287 ± 0.020 | 1.22× | 26.7% |
| Electricity | 0.02451 | 0.0206 ± 0.0010 | 1.19× | 80.9% |
| Traffic | 0.03334 | 0.044 ± 0.006 | 0.76× | 88.8% |
| Taxi | 0.13522 | 0.114 ± 0.020 | 1.19× | 77.8% |
Interpretation and provenance. Archived values come from outputs/reports/<dataset>/evaluation.json in the supplied project snapshot; tracked plotting inputs are in assets/readme/benchmark_snapshot.csv. Paper figures are from Table 2 of [1] and represent ten-run means with the paper's reported uncertainty, whereas each archived point is a single run with no reproducibility distribution. The bold entry in each row indicates the smaller reported point value only. This table cannot establish superiority, parity, or statistical significance, and does not certify that the active code will regenerate the archived checkpoint.
Figure 3. Archived central-interval empirical coverage, with dashed lines marking nominal 50% and 90% levels. Solar's 90% coverage is only 26.7% in the saved experiment; this indicates severe undercoverage and warrants calibration and pipeline investigation, rather than being treated as a successful probabilistic result.
Figure 4. Traffic, first archived rolling evaluation window, first six target dimensions. The observed future and prediction median are overlaid with 50% and 90% intervals computed from simulated trajectories; the vertical boundary separates history from forecast.
Figure 5. Electricity, first archived rolling evaluation window, first six dimensions. These are actual saved forecast plots from the supplied project, not illustrative synthetic forecasts.
Figure 6. Log-scale train and validation noise-prediction MSE from the archived tune phase. The saved training histories also contain retrain_full segments, but those are intentionally excluded here because the currently active training script does not perform that phase.
The package targets Python 3.10+ with PyTorch 2.4+, NumPy, pandas, PyYAML, Matplotlib, and GluonTS 0.16.2. A CUDA GPU is strongly recommended for full-dimensional benchmarks because ancestral inference is sequential over diffusion levels and forecast time.
# From the repository root
python -m venv .venv
source .venv/bin/activate # Windows PowerShell: .venv\Scripts\Activate.ps1
python -m pip install --upgrade pip
python -m pip install -r requirements.txtThe raw benchmark datasets are not embedded in this README package. The loader acquires them through GluonTS on first use (network access may be required) and caches transformed data under data/dataset/.
python -m scripts.main --config config/exchange.yaml --mode downloadpython -m scripts.main --config config/exchange.yaml --mode allFor another supported dataset, replace exchange.yaml with solar.yaml, electricity.yaml, traffic.yaml, or taxi.yaml. The configured outputs are written beneath models/saved_models/, outputs/reports/, and outputs/plots/.
To evaluate a previously saved, configuration-compatible checkpoint, run:
python -m scripts.main --config config/exchange.yaml --mode evaluateCheckpoint safety:
--mode allwrites to the configured checkpoint path. To preserve an existing model, overrideoutputs.checkpointand corresponding report/plot paths when launching exploratory runs.
The following uses the lowest-dimensional benchmark, fewer optimizer steps, fewer trajectories, a shorter diffusion chain, and a separate output prefix. It is a pipeline smoke run, not a paper reproduction:
python -m scripts.main --config config/exchange.yaml --mode all \
--set training.epochs=2 \
--set training.steps_per_epoch=4 \
--set training.batch_size=8 \
--set model.diffusion_steps=10 \
--set evaluation.num_samples=5 \
--set evaluation.max_windows=1 \
--set outputs.checkpoint=models/saved_models/exchange_smoke.pt \
--set outputs.history_csv=outputs/reports/smoke/training_history.csv \
--set outputs.evaluation_report=outputs/reports/smoke/evaluation.json \
--set outputs.run_summary=outputs/reports/smoke/run_summary.json \
--set outputs.loss_plot=outputs/plots/smoke/training_loss.png \
--set outputs.forecast_plot=outputs/plots/smoke/forecast.pngCLI options are defined in scripts/main.py. Nested YAML fields can be overridden through repeated --set key=value arguments.
The scripts.reproduce CLI trains, evaluates, and aggregates separate runs. Its current default is three seeds (0, 10, 20), despite text in its docstring referring to ten. To request ten runs explicitly:
python -m scripts.reproduce --config config/electricity.yaml \
--seeds 0 10 20 30 40 50 60 70 80 90These jobs are computationally substantial. Aggregate any already-created result files using:
python -m scripts.aggregate outputs/reports/paper_runs/electricity/seed_*/evaluation.jsonThis reports the arithmetic mean, sample-based standard error, and constituent CRPS_sum values; it does not automatically correct any discrepancies between the current active training pipeline and the archived runs.
The plots are checked into assets/readme/ because the repository's .gitignore excludes normal experiment output directories. The figure inputs are also tracked for provenance:
python assets/readme/render_plots.pyThis script reads benchmark_snapshot.csv and the archived tuning curves in assets/readme/data/. It does not download datasets, retrain a model, rerun evaluation, or synthesize forecast samples. The two qualitative forecast PNGs are copied from the archived project output.
This repository is best understood as a substantial implementation of the TimeGrad architecture with a documented experimental snapshot, not a verified exact regeneration of every published result.
| Issue | Verified status | Implication |
|---|---|---|
| Original paper scope | Six datasets; this registry contains five | No Wikipedia reproduction is provided. |
| Saved experimental statistics | One archived evaluation report per dataset | Paper's ten-run uncertainty cannot be reconstructed from these points. |
| Current refitting implementation | validation.retrain_full: true is present, but active train_experiment() stops after validation-selected training |
Re-running the current code is not identical to the archived checkpoints tagged retrain_full. |
| Independent seeds | scripts.reproduce defaults to three explicit seeds |
Pass --seeds with ten values to request ten independent runs. |
| Protocol checks | paper_protocol_complete checks configured settings at evaluation time |
This flag does not audit refitting, provenance, random-seed coverage, or numerical equivalence. |
| Forecast calibration | Significant Solar undercoverage in the saved report | Aggregate CRPSsum alone does not certify reliable uncertainty quantification. |
| Convolution geometry | Circular 1-D convolutions across a fixed dimension order | Sensitivity to arbitrary dimension permutations is a worthwhile ablation. |
| Inference cost | Sequential DDPM × autoregressive horizon | Runtime scales strongly with sampling steps, horizon, and generated trajectories. |
| Tests | pyproject.toml references tests/, but the supplied archive does not contain that directory |
No shipped regression suite can be claimed; add tests before asserting continuous integration coverage. |
| Dataset and output files | Excluded by .gitignore |
Commit assets/readme/ to preserve README visuals and summary statistics. |
Recommended reproducibility practice. Record the exact Git commit, configuration, seed, package versions, evaluation-window counts, dataset preprocessing versions, checkpoint phase, and hardware for every reported run. Publish per-seed results before presenting cross-run standard errors. Report the model-selection protocol and any modifications to the scale transform, lag construction, and variance schedule. Do not infer full benchmark reproduction solely from close CRPS values.
Several natural, falsifiable next experiments follow from this implementation: (i) dimension-permutation sensitivity under circular convolutions; (ii) the diffusion-length/forecast-quality/runtime Pareto frontier; (iii) post-hoc calibration and conditional coverage on low-coverage benchmarks; (iv) an explicit full-data refit ablation; (v) comparison of quantile-grid and exact sample CRPS estimators; and (vi) variance reduction across independently seeded model and forecast-sampling runs. None of these experiments is claimed to have been performed here.
.
├── assets/
│ └── readme/ # tracked figures, archived metrics, plotting script
├── config/
│ ├── base.yaml # shared model, training, and evaluation defaults
│ ├── exchange.yaml
│ ├── solar.yaml
│ ├── electricity.yaml
│ ├── traffic.yaml
│ └── taxi.yaml
├── data/
│ ├── benchmarks.py # dataset metadata and paper reference values
│ └── data_loader.py # download, grouping, rolling windows, covariates
├── models/
│ ├── components.py # diffusion embedding, WaveNet, gated residuals
│ ├── diffusion.py # forward noising, loss, ancestral sampling
│ └── model.py # LSTM conditioner and TimeGrad model
├── scripts/
│ ├── main.py # download / train / evaluate / all CLI
│ ├── train.py # optimizer, validation, checkpointing
│ ├── evaluate.py # rolling-window probabilistic evaluation
│ ├── reproduce.py # multiple independent-seed experiments
│ └── aggregate.py # metrics aggregation
├── utils/
│ ├── metrics.py # CRPS, ND, NRMSE, coverage
│ ├── visualization.py # forecast and training plots
│ └── config.py # configuration, paths, random seeds
├── outputs/ # runtime artifacts; gitignored by default
├── requirements.txt
├── pyproject.toml
└── README.md
[1] K. Rasul, C. Seward, I. Schuster, and R. Vollgraf. Autoregressive Denoising Diffusion Models for Multivariate Probabilistic Time Series Forecasting. Proceedings of the 38th International Conference on Machine Learning, PMLR 139:8857–8868, 2021. Paper · PDF · arXiv.
[2] J. Ho, A. Jain, and P. Abbeel. Denoising Diffusion Probabilistic Models. Advances in Neural Information Processing Systems, 2020. arXiv.
[3] A. van den Oord et al. WaveNet: A Generative Model for Raw Audio. 2016. arXiv.
[4] Z. Kong, W. Ping, J. Huang, K. Zhao, and B. Catanzaro. DiffWave: A Versatile Diffusion Model for Audio Synthesis. ICLR, 2021. arXiv.
[5] A. Alexandrov et al. GluonTS: Probabilistic and Neural Time Series Modeling in Python. JMLR, 2020. JMLR.
Original TimeGrad code: Zalando Research / pytorch-ts. The present project is a separate reproduction and is not represented as the original authors' software.





