PatchTST-RUL-PHM2012

Remaining useful life (RUL) prediction for rotating machinery with a PatchTST backbone, evaluated on the IEEE PHM 2012 bearing benchmark with a leakage-safe protocol. The study covers 30+ independent experimental configurations with multi-seed replication and 6-fold LOBO validation.

Keywords: PatchTST · RUL · Prognostics · PHM2012 · Time Series · Bearing

中文版 README

---

1. Project Overview

This repository studies how PatchTST-style long-sequence models perform on remaining useful life prediction for rolling bearings, using the IEEE PHM 2012 Prognostic Challenge dataset (FEMTO/PRONOSTIA accelerated bearing life tests).

Important: PHM2012 is a public proxy benchmark. It is not field data from metro wheelsets or any specific in-service rail component. The study's scope is method-level feasibility on a public, reproducible benchmark — conclusions about real-world components would require domain-specific run-to-failure data (see Limitations).

Highlights

• 28.0% lower MAE vs the two-stage PatchTST baseline (3467.7 → 2497.9 s, official 11-bearing protocol, 2 seeds)
• 22.5% lower MAE vs the best external baseline (GRU) under the same two-stage pipeline (3222.7 → 2497.9 s)
• 4-seed replication: MAE 2678.4 s (seeds {42, 2025, 7, 3407}), reported alongside the 2-seed result
• Leakage-safe evaluation: bearing-level train/test isolation, train-only feature scaling, causal FPT detection, and 6-fold LOBO validation for all tuning decisions

2. Motivation

Vibration-based RUL prediction is hard for several well-known reasons:

• High-frequency, non-stationary, noisy signals — point-level samples carry little semantics; regressors must extract degradation trends from noise.
• Severe train-label imbalance — most of a run-to-failure trajectory is healthy, and the RUL label saturates at the early-life cap, which biases full-trajectory regression toward mean RUL.
• Weak local degradation cues — this project's baseline uses a non-overlapping patch configuration (stride = patch_len); such hard splits fragment the local waveform context that carries early fault signatures (overlapping strides are supported by PatchTST and are explored as a configuration strategy in this work).
• Multi-sensor coupling — vibration (time/frequency domain) and operating channels are physically coupled, yet naive full-channel attention overfits on six training bearings.

3. Method

Raw vibration bursts (0.1 s @ 25.6 kHz, every 10 s)
        │  scripts/build_features.py
        ▼
41-channel segment features ──► windowing (128 segments ≈ 21 min)
        │
        ▼
HI & FPT (PCA + Mahalanobis distance, causal detection)
        │
        ├──► Stage 1: full-life pretraining (learning bearings)
        │
        ├──► Stage 2: degradation-segment fine-tuning (post-FPT windows)
        │         (+ optional log-space RUL labels)
        ▼
PatchTST / SoftPatchTST ──► per-window RUL ──► causal EOL smoothing
        ▼
Endpoint RUL ──► MAE / error-std / PHM2012 score (11 official test bearings)

The improved model (SoftPatchTST) differs from the baseline in three categories (see Model): configuration strategy (overlapping patch stride), structural additions (rFFT frequency branch, grouped channel mixer), and training strategy (FPT-aligned two-stage training, log-space labels).

4. Data Protocol

Full details in docs/experiment_protocol.md.

Split	Bearings	Use
Learning set	6 bearings (run to failure)	training, early-stopping validation, all hyper-parameter selection
Test set	11 truncated bearings, official ground truth	one-shot final evaluation

Anti-leakage guarantees (audited in code): feature standardization is fit on learning bearings only; early stopping uses the last 20% of one learning bearing; FPT detection is causal (baseline = the bearing's own early segments); no random window shuffling across splits; test ground truth is read only to compute the final error; every tuned hyper-parameter was chosen on a truncated-RUL LOBO validation protocol, never on the test set.

5. Model

Baseline: PatchTST-RUL

Channel-independent patch embedding (non-overlapping patches) + Transformer encoder + attentive pooling regression (rulpatch/models/patchtst.py). Implemented from the PatchTST paper; not a fork of the official codebase.

SoftPatchTST (this work)

Category	Change	Rationale
Configuration strategy	Overlapping patches (stride = patch_len × (1 − overlap), overlap = 0.5)	PatchTST's patching natively supports stride < patch_len; overlapping configurations preserve local waveform context across patch boundaries for noisy vibration signals. This is a design-space configuration choice, not a new mechanism.
Structural addition	rFFT frequency branch + fusion	A per-patch log-magnitude rFFT embedding is fused (concat → MLP) with the time-domain patch embedding, adding frequency-domain fault signatures.
Structural addition	GroupChannelMixer	Per-physical-group (vibration-time / vibration-freq / operating) multi-head attention + FFN; groups stay independent, keeping channel robustness while modeling strong intra-group coupling.
Training strategy	Full-life pretrain → FPT-aligned fine-tune	Training only on post-FPT (degradation) windows removes label-saturated healthy windows that drive mean regression.
Training strategy	Log-space RUL labels	log1p(RUL)/log1p(horizon) emphasizes relative error and stabilizes long-RUL bearings.

6. Experiments

All numbers below are read from results/*.csv (see results/README.md). Aggregation: seeds {42, 2025}, predictions averaged over seeds; std = population std (ddof=0) over the 11 test bearings.

6.1 Single-stage, official protocol (endpoint RUL, 11 test bearings)

Model	MAE ↓ (s)	error std ↓ (s)	PHM score ↑
SoftPatchTST (ours)	3474	2432	0.171
CNN-1D	3698	2876	0.268
LSTM	3769	1899	0.089
− freq branch (ablation)	3896	3211	0.160
− group mixer (ablation)	4035	3327	0.206
GRU	4116	2699	0.140
− overlap patches (ablation)	4175	3076	0.172
PatchTST baseline	4350	3310	0.158
Vanilla Transformer (step-token)	5091	2550	0.100

Notes:

• Under the identical single-stage protocol, the step-token vanilla Transformer's MAE (5091 s) is 46% higher than ours (3474 s) — tokenization granularity matters on this task. We report this as a controlled comparison, not as a claim that gains come from any single mechanism.
• A score-oriented variant (pinball loss + log labels + conservative γ = 0.9) lowers MAE to 2669 s (−23%) but its PHM score (0.105) stays below the original model's 0.171 (test_benchmark_scores_phmopt_pinball02.csv).
• By the official asymmetric PHM score, CNN-1D (0.268) is the best model in this set; our losses concentrate on over-estimating short-lived bearings (see Limitations).

6.2 Two-stage FPT-aligned (degradation-segment training, official protocol)

Model / variant	MAE ↓ (s)	error std ↓ (s)	PHM score ↑
SoftPatchTST + log labels — 2 seeds	2498	1731.6	0.068
SoftPatchTST + log labels — 4 seeds	2678.4	1673.4	0.068
GRU (same two-stage pipeline)	3223	1944	—
PatchTST baseline (same pipeline)	3468	2214.9	—
LSTM	6623	4619.7	—
CNN-1D	9396	17742.1	—
Vanilla Transformer	11120	15112.4	—

Two-stage FPT alignment reduces our MAE from 3474 s (single-stage) to 2498 s (2-seed) / 2678 s (4-seed) at the cost of the official PHM score (0.171 → 0.068). The seed gap (~180 s) is reported alongside; with 11 test points, differences of a few hundred seconds are within seed noise.

7. Ablation

Single-component ablations of SoftPatchTST (single-stage, official protocol):

Variant	MAE (s)	Δ vs full
Full SoftPatchTST	3474	—
− overlap patches (stride = patch_len)	4175	+701
− rFFT frequency branch	3896	+422
− GroupChannelMixer	4035	+561

Each component contributes independently; the combination is best. (The overlap-patch row is a configuration ablation within PatchTST's design space; the two structural additions are implemented in rulpatch/models/improved.py.)

8. Visualization

Selected figures (full set in results/figures):

Figure	Content
mae_progression.png	MAE across training strategies (4350 → 3474 → 2498 s)
mae_two_stage_by_model.png	single- vs two-stage MAE per model
phm_score_comparison.png	PHM score across models and score-optimization variants
test_benchmark_per_bearing.png	per-bearing absolute errors, single-stage
lobo_Bearing1_1.png	LOBO prediction trajectory (example fold)

9. Quick Start

pip install -r requirements.txt

# 0) smoke test (synthetic data, CPU, <1 min) — no dataset needed
python tests/smoke_test.py

# 1) obtain the dataset (see data/README.md) into data/raw/

# 2) build feature tables
python scripts/build_features.py --root data/raw/Learning_set --out data/processed/femto_features.parquet
python scripts/build_features.py --root data/raw/Test_set    --out data/processed/femto_test_features.parquet

# 3) LOBO cross-validation (6 folds x 2 seeds)
python scripts/run_lobo.py --data data/processed/femto_features.parquet

# 4) official-protocol benchmark (9 models; ~18 min on a laptop GPU)
python scripts/run_test.py

# 5) two-stage FPT-aligned headline numbers
python scripts/run_test_fpt.py --stage2 nn --log-label --seeds 42 2025

10. Repository Structure

├── rulpatch/            # core package: data / features / HI / models / training / metrics
├── scripts/             # build_features, run_lobo, run_test, run_test_fpt, tuning, plots
├── configs/femto.yaml   # window / horizon / architecture / seeds / device: auto
├── data/README.md       # dataset download & preprocessing (data itself is not shipped)
├── docs/                # experiment_protocol.md
├── results/             # CSVs + figures + COMPARISON.md (see results/README.md)
└── tests/smoke_test.py  # minimal end-to-end pipeline check on synthetic data

11. Reproducibility

• Python 3.10, PyTorch ≥ 2.0 (device: auto → CUDA when available)
• Seeds: {42, 2025} (4-seed replications add {7, 3407}); set_seed fixes Python/NumPy/PyTorch RNGs
• Key hyper-parameters: configs/femto.yaml (window 128, horizon 12000 s, patch 16, d_model 96, 2 layers, lr 7e-4, batch 128)
• Hardware used for the reported numbers: RTX 4070 Laptop (8 GB), Windows 11
• Protocol: docs/experiment_protocol.md

12. Limitations

1. Proxy benchmark: results are on PHM2012 bearings (accelerated life tests), not on metro wheelsets or any specific field component.
2. No field run-to-failure data for the target components was available; conclusions are method-level feasibility, and transfer requires resolving domain shift (operating conditions, failure modes, sensor setup).
3. Small test sample: 11 test bearings; MAE differences within a few hundred seconds are seed-level noise — hence multi-seed reporting.
4. PHM score vs MAE trade-off: score-optimization variants (log-space regression, pinball, weighted L1, conservative γ, label horizon, smoothed quantiles — all selected on leakage-safe LOBO validation) did not beat the original model's score on the test set; details and analysis in results/COMPARISON.md.

Acknowledgements

• PatchTST: Nie, Yuqi, et al. A Time Series is Worth 64 Words: Long-term Forecasting with Transformers. ICLR 2023. The baseline here is an independent re-implementation of the paper's idea for RUL regression, not a fork of the official repository.
• Dataset: IEEE PHM 2012 Prognostic Challenge / PRONOSTIA platform (FEMTO-ST). Citation: Nectoux, P. et al., IEEE PHM 2012. Distributed via the NASA Prognostics Data Repository; not redistributed here.
• Two-stage paradigm: Wang, Peng, Zi et al. (IEEE PHM 2012 classic solution family); reference implementation: akshatshah21/RUL-Prediction.
• Literature comparisons cited in results/COMPARISON.md belong to their respective authors.

License

Apache License 2.0