diff --git a/akida1/model_zoo/uored_vafcls/README.md b/akida1/model_zoo/uored_vafcls/README.md new file mode 100644 index 0000000..421bd50 --- /dev/null +++ b/akida1/model_zoo/uored_vafcls/README.md @@ -0,0 +1,327 @@ +BrainChip Dev Hub + +# Bearing Fault Diagnosis from Vibration data (UORED-VAFCLS) + +Multi-label detection of rolling-element bearing faults — inner race, outer race, ball and cage — from one second of raw 42 kHz accelerometer data. The four faults are independent labels rather than classes, and a healthy bearing is the all-zero vector, not a fifth class. The reported metric is therefore macro AUROC, which needs no decision threshold. + +This example tries to convey two different points: +- how to prepare and evaluate a model for Akida 1, for single-channel time-series data +- the particularities of this and related datasets that mean that special care needs to be taken in preparing the training and evaluation splits, and that comparison across published results is extremely difficult + +## Model Card + +The table reports results for a **single model architecture**. Only the method used to **split the training and evaluation data** is changed. See below for full details. + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
SplitFloat AUROCQAT AUROCAkida AUROCParamsActivation sparsity
Segment-level0.99830.99630.9973335,44439.06%
Bearing-level0.89500.89390.8939335,44437.81%
+ +**AKD1500 hardware benchmark** + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
MappingNPsPassesCyclesLatency (ms)Total Power (mW)Total Energy (mJ/inf)Dyn. Power (mW)Dyn. Energy (mJ/inf)
Minimal17139308799.827179.21.78267.50.671
AllNPs27129329337.332201.91.50689.00.664
HwPr64330631527.658216.21.791103.10.854
+ +Measured on the model provided in the `pretrained_models/` folder, trained on the bearing-level split at fold 42. + +Full model benchmark + +The model is small enough to map entirely to hardware in a single sequence, single pass. The actual mapping depends on the +requested mapping mode, as follows. `Minimal` mapping uses the fewest neural processors (NPs) that will hold the model; +`AllNPs` spreads it over the available NPs without increasing the number of passes: in this case, because the model almost +fills the device anyway, there is only a slight difference between these modes. `HwPr` mode tries to accelerate the model +even further, by splitting the model over more passes (effectively increasing the number of NPs available per layer). That +strategy is effective for a majority of models, but not in this case: that's because the model is small, and fits in a single +pass by default; any gain per-layer is outweighed by the need to reload model weights on every inference. + +## Requirements + +See the [requirements section](../../../README.md#requirements) of the top-level README for the environment used +throughout this repository. + +## Dataset + +**UORED-VAFCLS** — the University of Ottawa Rolling-element Dataset, Vibration and Acoustic Faults under Constant Load and Speed. + +This dataset comprises 60 accelerometer recordings, each 10 s at 42 kHz, from 20 bearings. Every bearing contributes three recordings: +one healthy, plus two severities of a single fault mode. + +| | | +|---|---| +| Recordings | 60 (20 bearings × 3) | +| Length | 10 s at 42 kHz (420,000 samples) | +| Model input | 1 s window | +| Labels | 4 independent: inner, outer, ball, cage (healthy = all zero) | +| Windows per fold | 720 training (random crops), 240 held-out (tiled) | + +Sample recordings from the UORED-VAFCLS dataset + +Downloaded from [Mendeley Data, doi:10.17632/y2px5tg92h.5](https://data.mendeley.com/datasets/y2px5tg92h/5), licensed **CC BY 4.0** (https://creativecommons.org/licenses/by/4.0/): + +> Sehri, Maryam; Dumond, Patrick (2023), *"University of Ottawa Rolling-element Dataset – Vibration and Acoustic Faults under Constant Load and Speed conditions (UORED-VAFCLS)"*, Mendeley Data, V5, doi: 10.17632/y2px5tg92h.5 + +*Changes made, as the licence requires us to state:* for convenience, we make the data required in this example available +as a single file for download from the BrainChip servers. No sample values are altered: the `Accelerometer` column of each +raw CSV is repacked verbatim as float32 into a single `.npz`, along with the first row of the `Speed` column as a +per-recording nominal RPM, and per-recording metadata (bearing id, fault type, severity, waveform id and the multi-label +targets) derived from the original filenames and folder structure. + +## The Problem of Realistic Evaluation of Bearing Fault Diagnosis Models + +This example draws very directly from the following article, which itself builds on a number of related publications that establish a +widespread methodological problem in the evaluation of machine learning approaches to the key datasets in the diagnosis of bearing faults +from vibration data (UORED-VAFCLS used here, but even more strikingly in the key datasets in the domain, CWRU and Paderborn): + +> J. P. Vieira, V. A. Bauler, R. K. Rosa, D. Silva, *"Towards a more realistic evaluation of machine learning models for bearing fault diagnosis"*, Mechanical Systems and Signal Processing 258:114640, 2026. doi:10.1016/j.ymssp.2026.114640 ([arXiv:2509.22267](https://arxiv.org/abs/2509.22267), [code](https://github.com/gama-ufsc/bearing-data-leakage)) + +The abstract from that article sets the context clearly: +> *"While recent advances in machine learning (ML), particularly + deep learning, have shown strong performance in controlled settings, many studies fail to + generalize to real-world applications due to methodological flaws, most notably data leakage. + This paper investigates the issue of data leakage in vibration-based bearing fault diagnosis and + its impact on model evaluation. We demonstrate that common dataset partitioning strategies, + such as segment-wise and condition-wise splits, introduce spurious correlations that inflate + performance metrics."* + +The risk is specific and easy to walk into. These datasets are built from a small number of physical bearings, each recorded +over a short, uninterrupted run under a single load and speed, and the fault mode is a property of the bearing rather than of +the signal: here, bearings 1–5 carry inner-race faults, 6–10 outer, 11–15 ball, 16–20 cage. Any split that lets windows from +one recording — or from one bearing — fall on both sides of the train/test boundary therefore hands the model a shortcut. It +can re-identify the recording from its noise floor, its mounting resonances or its running speed and read the label straight +off that identity, without learning anything transferable about bearing faults at all. Segment-wise splits (cutting each +recording in time) and condition-wise splits (holding out load or speed settings while keeping the same bearings) both leak in +exactly this way, and both are common in the published literature. The consequence is that near-perfect headline accuracies — +99%+ is routine on CWRU — measure the ease of recognising a recording, not the ability to diagnose an unseen bearing, and they +collapse as soon as the model meets hardware it has not been recorded on. It also makes cross-paper comparison largely +meaningless: two results on the same dataset are usually not measuring the same thing. + +The reference citation above sets out a rigorous protocol for each of the datasets explored. The UORED-VAFCLS dataset used here +has 60 recordings from **20 physical bearings**. A fold trains on 12 bearings and is tested on the 8 held out, the +key point being that an individual physical bearing never appears in both train and test data (even data from separate recordings +is disallowed, e.g. if the 'healthy' recording for bearing 0 is put in the train data, then the 'fault' recordings for that bearing +must not be in the test split). You can consult the details of the data split preparation in the relevant script, `uored_vafcls_data.py`. + +That rigorous data split has knock-on consequences: the data splits are small enough that the choice of which bearings are held out +matters more than almost anything about the model, as can be seen from the plot of the distribution of performance across folds +below: + +- The **across-fold** standard deviation is 0.0666, and folds range from 0.6989 to 0.9892 + with the architecture, the recipe and the seed all held fixed. +- The **within-fold** standard deviation across random seeds is 0.0186 (measured over 25 folds at 3 + seeds each). + +Distribution of per-fold AUROC + +*(Note: Fold 42 is flagged because the model trained on this fold is the one set aside in the `pretrained_models` folder.)* + +So a single-fold, single-seed AUROC cannot distinguish two architectures on this dataset. It is absolutely necessary to run multi-fold +cross-validation to have anything approaching an accurate evaluation. For the bearing-wise split, this example follows the protocol +set out by the reference paper above: +1. **The reported number is the mean over all 100 evaluation folds**, produced by `uored_vafcls_cross_validate.py`. The per-fold results are committed in [`docs/cv_results.csv`](docs/cv_results.csv) so the mean is auditable without re-running anything. +2. **Folds 0–4** are used for tuning hyperparameters, such as the number of epochs, learning rate, the QAT schedule and the input encoding constants. The remaining **folds (5–104)** are reserved as the test split, and should never be used for tuning. + +### Results for the Segment-level Split + +Read on its own, results for the bearing-disjoint score invite the wrong conclusion: that +the model is weak. It is not - the evaluation is honest, and honest numbers on +this dataset are lower than the ones usually published. To make that visible, +we also show results for a segment-level data split (each recording contributes different +time segments to both the train and test splits). Published results very commonly use +this kind of data splitting. + +Both modes yield identical budgets - 720 training windows (6 steps per epoch) +and 240 test windows - so a score difference between them is attributable to the +split and nothing else. Sure enough, the model achieves **>99.5% AUROC**, and without +any further tuning for this version of the task. + +Unlike the bearing-level split, the segment-level split needs no cross-validation. It has no folds +to average over (there is one time cut, the same for every run) and its score +is stable to about 0.001 where the bearing-disjoint folds span 0.04-0.05. One run +is sufficient. You can re-run uored_vafcls_naive_split.sh with a different seed to +confirm that for yourself. + +## Dataset setup + +The prepared cache downloads automatically on first use, to `--data` (default `./data/uored_vafcls`). To fetch it ahead of time: + +```bash +python uored_vafcls_data.py -d ./data/uored_vafcls +``` + +If you prefer a different data location to the default (e.g. because your system has a dedicated data drive), you may find it +easy to set up a symbolic link from the default location. That way you can avoid passing the data path argument to all of the +scripts: + +```bash +ln -s /path/to/shared/uored_vafcls ./data/uored_vafcls +``` + +If you already hold the raw dataset, rebuild the cache from it instead. Download `1_CSV_Raw_Data_Files (.csv)` from the +Mendeley record and point at the directory containing the five `1_Healthy` … `5_Cage_Faults` subfolders: + +```bash +python uored_vafcls_data.py --prepare-raw /path/to/1_CSV_Raw_Data_Files +``` + + +## Pipeline + +| Stage | Description | +|---|---| +| Full-precision training | 30 epochs, Adam, batch 120 (6 steps/epoch), cosine decay with 5% warmup to a peak LR of 2e-4 | +| Post-training quantization | `cnn2snn quantize` reduces to 4-bit weights and activations (8-bit input) | +| Quantization-aware tuning | 10 epochs at a peak LR of 5e-5, recovering almost all of the quantization loss | +| Conversion to Akida | `cnn2snn convert` produces the `.fbz` model that runs on hardware | + +## Reference Models + +The trained models in `pretrained_models/` are stored with Git LFS. See the [trained models section](../../../README.md#trained-models) of the top-level README if they arrive as text pointer files rather than real weights. + +These models are necessarily trained on a specific fold. For that, we've selected fold 42, because performance is close to the 100-fold cross-validation value +(obviously, that in itself doesn't make the single-fold accuracy value any more meaningful; rather we hope to avoid any misunderstandings about performance +if a reader is not following the detail of this example). In any case, the pretrained model is only used for benchmarking on hardware which, since the architecture +is constant, should be more or less constant across folds (will vary only to the extent that learned sparsity within the model varies across training runs). + +## Usage + +### Notebook + +> ⚠️ **Work in progress — the notebooks are not included in this release yet.** The section below describes what they will +> cover; until they land, use the scripts described under [Script](#script). The links will not resolve. + +[`uored_vafcls_notebook_training.ipynb`](uored_vafcls_notebook_training.ipynb) walks through the whole pipeline on a single fold: the dataset and its two leakage traps, how a 1 s waveform becomes a framed uint8 tensor, training, quantization, tuning and conversion. It also loads `docs/cv_results.csv` to show the fold distribution without needing a two-hour run. + +[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Brainchip-Inc/brainchip_devhub/blob/main/akida1/model_zoo/uored_vafcls/uored_vafcls_notebook_training.ipynb) + +[`uored_vafcls_notebook_benchmark.ipynb`](uored_vafcls_notebook_benchmark.ipynb) evaluates the converted model, measures activation sparsity, and runs the hardware latency and power benchmarks. + +> **Note:** the benchmark notebook needs a physical AKD1500 device, so it does not run in Colab. The power measurements additionally need the FTDI current sensor. + +### Script + +Run the full pipeline on one fold — build, train, quantize, tune, convert, evaluate at each stage, and benchmark: + +```bash +bash uored_vafcls_train.sh [DATADIR] [FOLD] [SEED] +``` + +`FOLD` defaults to 42 and `SEED` to 0. It takes about a minute on a single GPU, plus benchmarking. The nine steps are the standard Akida 1 sequence; see the script for the exact commands. + +### Cross-validation + +**Use this command to reproduce the reported cross-validation performance reported above.** + +```bash +python uored_vafcls_cross_validate.py --save-metrics +``` + +It runs the whole chain on each of the 100 evaluation folds, appending to `docs/cv_results.csv` after every fold, and takes roughly 30–60 minutes. +It resumes by default, so an interrupted run picks up where it stopped. Add `--skip-akida` for a roughly 4× faster float-only sweep when comparing +recipes, and `--first-fold`/`--last-fold` to split the work across machines, or to limit to the first 5 folds (0-4) if working on tuning the model +or pipeline (remember, folds 5-104 are for the final evaluation only). + +## Contributing and Maintenance + +`README.md` in this folder is **generated** — edit [`docs/README.md.template`](docs/README.md.template), never `README.md` directly. + +The performance tables are filled from `docs/metrics.json`, which is written by the `--save-metrics` flags (if updating results +following changes to the model or pipeline, remember to delete the `.csv` files first, or use the `--no-resume` argument on the +cross-validation runs): + +```bash +# Bearing-level split (Model Card's bottom row). +# Its AUROCs come from the sweep, never from a single model. +python uored_vafcls_cross_validate.py --save-metrics # 30-60 min +python uored_vafcls_cross_validate.py --first-fold 5 --last-fold 29 --seeds 3 \ + -o docs/cv_seed_std.csv --save-seed-std # the seed-noise figure +python uored_vafcls_eval.py -l pretrained_models/akdcnn_uored_vafcls.h5 --save-metrics # params +python uored_vafcls_eval.py -l pretrained_models/akdcnn_uored_vafcls_qat.fbz --save-metrics # sparsity +python uored_vafcls_benchmark.py -l pretrained_models/akdcnn_uored_vafcls_qat.fbz --save-metrics + +# Naive segment-level split (Model Card's top row). +# One model, one run: the split is singular and its score is stable to ~0.001. +bash uored_vafcls_naive_split.sh +python uored_vafcls_eval.py -l models/akdcnn_uored_vafcls_segment.h5 --split-mode segment --save-metrics +python uored_vafcls_eval.py -l models/akdcnn_uored_vafcls_segment_qat.h5 --split-mode segment --save-metrics +python uored_vafcls_eval.py -l models/akdcnn_uored_vafcls_segment_qat.fbz --split-mode segment --save-metrics + +python update_readme.py +``` + +Each key in `metrics.json` has exactly one writer, so the scripts can be run in any order without racing. There is deliberately no bearing-level single-model AUROC: nothing in the README reports one, because reporting one is the mistake this example is about. + +**The standing rule:** any change to the architecture, the training recipe or the `ENCODE_*` constants invalidates the cross-validated table, and requires a fresh 100-fold sweep. Re-running the fixed fold is *not* a substitute — that is the whole point of this example. diff --git a/akida1/model_zoo/uored_vafcls/colab_setup.py b/akida1/model_zoo/uored_vafcls/colab_setup.py new file mode 100644 index 0000000..9289aec --- /dev/null +++ b/akida1/model_zoo/uored_vafcls/colab_setup.py @@ -0,0 +1,34 @@ +#This handles the set up necessary to run this notebook in Google Colab. + +import sys, os + +def setup(): + if 'google.colab' not in sys.modules: + print('Not running in Google Colab. No setup needed.') + return + + import subprocess + + def _run(cmd): + print(f'$ {cmd}') + subprocess.run(cmd, shell=True, check=True) + + REPO_URL = "https://github.com/Brainchip-Inc/brainchip_devhub.git" + REPO_DIR = "brainchip_devhub" + + if not os.path.exists(REPO_DIR): + _run(f"git clone --depth 1 {REPO_URL} {REPO_DIR}") + + EXAMPLE_SUBDIR = "akida1/model_zoo/uored_vafcls" + os.chdir(os.path.join(REPO_DIR, EXAMPLE_SUBDIR)) + + repo_root = os.path.abspath(os.path.join(os.getcwd(), "..", "..", "..")) + sys.path.insert(0, repo_root) + sys.path.insert(0, os.getcwd()) + + #install packages + _run("pip install -q akida_models==1.14.0 tf_keras pooch scikit-learn") + + print('Colab setup complete. Working directory:', os.getcwd()) + print('If TensorFlow was just installed, restart the runtime ' + '(Runtime > Restart runtime) and re-run this cell before continuing.') \ No newline at end of file diff --git a/akida1/model_zoo/uored_vafcls/data/.gitignore b/akida1/model_zoo/uored_vafcls/data/.gitignore new file mode 100644 index 0000000..f353a40 --- /dev/null +++ b/akida1/model_zoo/uored_vafcls/data/.gitignore @@ -0,0 +1,4 @@ +# Git to Ignore everything in this directory +* +# Except this .gitignore file +!.gitignore diff --git a/akida1/model_zoo/uored_vafcls/docs/README.md.template b/akida1/model_zoo/uored_vafcls/docs/README.md.template new file mode 100644 index 0000000..be226d4 --- /dev/null +++ b/akida1/model_zoo/uored_vafcls/docs/README.md.template @@ -0,0 +1,327 @@ +BrainChip Dev Hub + +# Bearing Fault Diagnosis from Vibration data (UORED-VAFCLS) + +Multi-label detection of rolling-element bearing faults — inner race, outer race, ball and cage — from one second of raw 42 kHz accelerometer data. The four faults are independent labels rather than classes, and a healthy bearing is the all-zero vector, not a fifth class. The reported metric is therefore macro AUROC, which needs no decision threshold. + +This example tries to convey two different points: +- how to prepare and evaluate a model for Akida 1, for single-channel time-series data +- the particularities of this and related datasets that mean that special care needs to be taken in preparing the training and evaluation splits, and that comparison across published results is extremely difficult + +## Model Card + +The table reports results for a **single model architecture**. Only the method used to **split the training and evaluation data** is changed. See below for full details. + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
SplitFloat AUROCQAT AUROCAkida AUROCParamsActivation sparsity
Segment-level{segment_float_auroc}{segment_qat_auroc}{segment_akida_auroc}{segment_params}{segment_sparsity}
Bearing-level{cv_float_auroc_mean}{cv_qat_auroc_mean}{cv_akida_auroc_mean}{params}{sparsity}
+ +**AKD1500 hardware benchmark** + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
MappingNPsPassesCyclesLatency (ms)Total Power (mW)Total Energy (mJ/inf)Dyn. Power (mW)Dyn. Energy (mJ/inf)
Minimal{minimal_nps}{minimal_passes}{minimal_cycles}{minimal_latency_ms}{minimal_total_P}{minimal_total_E}{minimal_dyn_P}{minimal_dyn_E}
AllNPs{allnps_nps}{allnps_passes}{allnps_cycles}{allnps_latency_ms}{allnps_total_P}{allnps_total_E}{allnps_dyn_P}{allnps_dyn_E}
HwPr{hwpr_nps}{hwpr_passes}{hwpr_cycles}{hwpr_latency_ms}{hwpr_total_P}{hwpr_total_E}{hwpr_dyn_P}{hwpr_dyn_E}
+ +Measured on the model provided in the `pretrained_models/` folder, trained on the bearing-level split at fold {fixed_fold}. + +Full model benchmark + +The model is small enough to map entirely to hardware in a single sequence, single pass. The actual mapping depends on the +requested mapping mode, as follows. `Minimal` mapping uses the fewest neural processors (NPs) that will hold the model; +`AllNPs` spreads it over the available NPs without increasing the number of passes: in this case, because the model almost +fills the device anyway, there is only a slight difference between these modes. `HwPr` mode tries to accelerate the model +even further, by splitting the model over more passes (effectively increasing the number of NPs available per layer). That +strategy is effective for a majority of models, but not in this case: that's because the model is small, and fits in a single +pass by default; any gain per-layer is outweighed by the need to reload model weights on every inference. + +## Requirements + +See the [requirements section](../../../README.md#requirements) of the top-level README for the environment used +throughout this repository. + +## Dataset + +**UORED-VAFCLS** — the University of Ottawa Rolling-element Dataset, Vibration and Acoustic Faults under Constant Load and Speed. + +This dataset comprises 60 accelerometer recordings, each 10 s at 42 kHz, from 20 bearings. Every bearing contributes three recordings: +one healthy, plus two severities of a single fault mode. + +| | | +|---|---| +| Recordings | 60 (20 bearings × 3) | +| Length | 10 s at 42 kHz (420,000 samples) | +| Model input | 1 s window | +| Labels | 4 independent: inner, outer, ball, cage (healthy = all zero) | +| Windows per fold | 720 training (random crops), 240 held-out (tiled) | + +Sample recordings from the UORED-VAFCLS dataset + +Downloaded from [Mendeley Data, doi:10.17632/y2px5tg92h.5](https://data.mendeley.com/datasets/y2px5tg92h/5), licensed **CC BY 4.0** (https://creativecommons.org/licenses/by/4.0/): + +> Sehri, Maryam; Dumond, Patrick (2023), *"University of Ottawa Rolling-element Dataset – Vibration and Acoustic Faults under Constant Load and Speed conditions (UORED-VAFCLS)"*, Mendeley Data, V5, doi: 10.17632/y2px5tg92h.5 + +*Changes made, as the licence requires us to state:* for convenience, we make the data required in this example available +as a single file for download from the BrainChip servers. No sample values are altered: the `Accelerometer` column of each +raw CSV is repacked verbatim as float32 into a single `.npz`, along with the first row of the `Speed` column as a +per-recording nominal RPM, and per-recording metadata (bearing id, fault type, severity, waveform id and the multi-label +targets) derived from the original filenames and folder structure. + +## The Problem of Realistic Evaluation of Bearing Fault Diagnosis Models + +This example draws very directly from the following article, which itself builds on a number of related publications that establish a +widespread methodological problem in the evaluation of machine learning approaches to the key datasets in the diagnosis of bearing faults +from vibration data (UORED-VAFCLS used here, but even more strikingly in the key datasets in the domain, CWRU and Paderborn): + +> J. P. Vieira, V. A. Bauler, R. K. Rosa, D. Silva, *"Towards a more realistic evaluation of machine learning models for bearing fault diagnosis"*, Mechanical Systems and Signal Processing 258:114640, 2026. doi:10.1016/j.ymssp.2026.114640 ([arXiv:2509.22267](https://arxiv.org/abs/2509.22267), [code](https://github.com/gama-ufsc/bearing-data-leakage)) + +The abstract from that article sets the context clearly: +> *"While recent advances in machine learning (ML), particularly + deep learning, have shown strong performance in controlled settings, many studies fail to + generalize to real-world applications due to methodological flaws, most notably data leakage. + This paper investigates the issue of data leakage in vibration-based bearing fault diagnosis and + its impact on model evaluation. We demonstrate that common dataset partitioning strategies, + such as segment-wise and condition-wise splits, introduce spurious correlations that inflate + performance metrics."* + +The risk is specific and easy to walk into. These datasets are built from a small number of physical bearings, each recorded +over a short, uninterrupted run under a single load and speed, and the fault mode is a property of the bearing rather than of +the signal: here, bearings 1–5 carry inner-race faults, 6–10 outer, 11–15 ball, 16–20 cage. Any split that lets windows from +one recording — or from one bearing — fall on both sides of the train/test boundary therefore hands the model a shortcut. It +can re-identify the recording from its noise floor, its mounting resonances or its running speed and read the label straight +off that identity, without learning anything transferable about bearing faults at all. Segment-wise splits (cutting each +recording in time) and condition-wise splits (holding out load or speed settings while keeping the same bearings) both leak in +exactly this way, and both are common in the published literature. The consequence is that near-perfect headline accuracies — +99%+ is routine on CWRU — measure the ease of recognising a recording, not the ability to diagnose an unseen bearing, and they +collapse as soon as the model meets hardware it has not been recorded on. It also makes cross-paper comparison largely +meaningless: two results on the same dataset are usually not measuring the same thing. + +The reference citation above sets out a rigorous protocol for each of the datasets explored. The UORED-VAFCLS dataset used here +has 60 recordings from **20 physical bearings**. A fold trains on 12 bearings and is tested on the 8 held out, the +key point being that an individual physical bearing never appears in both train and test data (even data from separate recordings +is disallowed, e.g. if the 'healthy' recording for bearing 0 is put in the train data, then the 'fault' recordings for that bearing +must not be in the test split). You can consult the details of the data split preparation in the relevant script, `uored_vafcls_data.py`. + +That rigorous data split has knock-on consequences: the data splits are small enough that the choice of which bearings are held out +matters more than almost anything about the model, as can be seen from the plot of the distribution of performance across folds +below: + +- The **across-fold** standard deviation is {cv_akida_auroc_std}, and folds range from {cv_akida_auroc_min} to {cv_akida_auroc_max} + with the architecture, the recipe and the seed all held fixed. +- The **within-fold** standard deviation across random seeds is {cv_seed_std} (measured over {cv_seed_std_folds} folds at 3 + seeds each). + +Distribution of per-fold AUROC + +*(Note: Fold 42 is flagged because the model trained on this fold is the one set aside in the `pretrained_models` folder.)* + +So a single-fold, single-seed AUROC cannot distinguish two architectures on this dataset. It is absolutely necessary to run multi-fold +cross-validation to have anything approaching an accurate evaluation. For the bearing-wise split, this example follows the protocol +set out by the reference paper above: +1. **The reported number is the mean over all {cv_folds} evaluation folds**, produced by `uored_vafcls_cross_validate.py`. The per-fold results are committed in [`docs/cv_results.csv`](docs/cv_results.csv) so the mean is auditable without re-running anything. +2. **Folds 0–4** are used for tuning hyperparameters, such as the number of epochs, learning rate, the QAT schedule and the input encoding constants. The remaining **folds (5–104)** are reserved as the test split, and should never be used for tuning. + +### Results for the Segment-level Split + +Read on its own, results for the bearing-disjoint score invite the wrong conclusion: that +the model is weak. It is not - the evaluation is honest, and honest numbers on +this dataset are lower than the ones usually published. To make that visible, +we also show results for a segment-level data split (each recording contributes different +time segments to both the train and test splits). Published results very commonly use +this kind of data splitting. + +Both modes yield identical budgets - 720 training windows (6 steps per epoch) +and 240 test windows - so a score difference between them is attributable to the +split and nothing else. Sure enough, the model achieves **>99.5% AUROC**, and without +any further tuning for this version of the task. + +Unlike the bearing-level split, the segment-level split needs no cross-validation. It has no folds +to average over (there is one time cut, the same for every run) and its score +is stable to about 0.001 where the bearing-disjoint folds span 0.04-0.05. One run +is sufficient. You can re-run uored_vafcls_naive_split.sh with a different seed to +confirm that for yourself. + +## Dataset setup + +The prepared cache downloads automatically on first use, to `--data` (default `./data/uored_vafcls`). To fetch it ahead of time: + +```bash +python uored_vafcls_data.py -d ./data/uored_vafcls +``` + +If you prefer a different data location to the default (e.g. because your system has a dedicated data drive), you may find it +easy to set up a symbolic link from the default location. That way you can avoid passing the data path argument to all of the +scripts: + +```bash +ln -s /path/to/shared/uored_vafcls ./data/uored_vafcls +``` + +If you already hold the raw dataset, rebuild the cache from it instead. Download `1_CSV_Raw_Data_Files (.csv)` from the +Mendeley record and point at the directory containing the five `1_Healthy` … `5_Cage_Faults` subfolders: + +```bash +python uored_vafcls_data.py --prepare-raw /path/to/1_CSV_Raw_Data_Files +``` + + +## Pipeline + +| Stage | Description | +|---|---| +| Full-precision training | 30 epochs, Adam, batch 120 (6 steps/epoch), cosine decay with 5% warmup to a peak LR of 2e-4 | +| Post-training quantization | `cnn2snn quantize` reduces to 4-bit weights and activations (8-bit input) | +| Quantization-aware tuning | 10 epochs at a peak LR of 5e-5, recovering almost all of the quantization loss | +| Conversion to Akida | `cnn2snn convert` produces the `.fbz` model that runs on hardware | + +## Reference Models + +The trained models in `pretrained_models/` are stored with Git LFS. See the [trained models section](../../../README.md#trained-models) of the top-level README if they arrive as text pointer files rather than real weights. + +These models are necessarily trained on a specific fold. For that, we've selected fold 42, because performance is close to the 100-fold cross-validation value +(obviously, that in itself doesn't make the single-fold accuracy value any more meaningful; rather we hope to avoid any misunderstandings about performance +if a reader is not following the detail of this example). In any case, the pretrained model is only used for benchmarking on hardware which, since the architecture +is constant, should be more or less constant across folds (will vary only to the extent that learned sparsity within the model varies across training runs). + +## Usage + +### Notebook + +> ⚠️ **Work in progress — the notebooks are not included in this release yet.** The section below describes what they will +> cover; until they land, use the scripts described under [Script](#script). The links will not resolve. + +[`uored_vafcls_notebook_training.ipynb`](uored_vafcls_notebook_training.ipynb) walks through the whole pipeline on a single fold: the dataset and its two leakage traps, how a 1 s waveform becomes a framed uint8 tensor, training, quantization, tuning and conversion. It also loads `docs/cv_results.csv` to show the fold distribution without needing a two-hour run. + +[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Brainchip-Inc/brainchip_devhub/blob/main/akida1/model_zoo/uored_vafcls/uored_vafcls_notebook_training.ipynb) + +[`uored_vafcls_notebook_benchmark.ipynb`](uored_vafcls_notebook_benchmark.ipynb) evaluates the converted model, measures activation sparsity, and runs the hardware latency and power benchmarks. + +> **Note:** the benchmark notebook needs a physical AKD1500 device, so it does not run in Colab. The power measurements additionally need the FTDI current sensor. + +### Script + +Run the full pipeline on one fold — build, train, quantize, tune, convert, evaluate at each stage, and benchmark: + +```bash +bash uored_vafcls_train.sh [DATADIR] [FOLD] [SEED] +``` + +`FOLD` defaults to {fixed_fold} and `SEED` to 0. It takes about a minute on a single GPU, plus benchmarking. The nine steps are the standard Akida 1 sequence; see the script for the exact commands. + +### Cross-validation + +**Use this command to reproduce the reported cross-validation performance reported above.** + +```bash +python uored_vafcls_cross_validate.py --save-metrics +``` + +It runs the whole chain on each of the 100 evaluation folds, appending to `docs/cv_results.csv` after every fold, and takes roughly 30–60 minutes. +It resumes by default, so an interrupted run picks up where it stopped. Add `--skip-akida` for a roughly 4× faster float-only sweep when comparing +recipes, and `--first-fold`/`--last-fold` to split the work across machines, or to limit to the first 5 folds (0-4) if working on tuning the model +or pipeline (remember, folds 5-104 are for the final evaluation only). + +## Contributing and Maintenance + +`README.md` in this folder is **generated** — edit [`docs/README.md.template`](docs/README.md.template), never `README.md` directly. + +The performance tables are filled from `docs/metrics.json`, which is written by the `--save-metrics` flags (if updating results +following changes to the model or pipeline, remember to delete the `.csv` files first, or use the `--no-resume` argument on the +cross-validation runs): + +```bash +# Bearing-level split (Model Card's bottom row). +# Its AUROCs come from the sweep, never from a single model. +python uored_vafcls_cross_validate.py --save-metrics # 30-60 min +python uored_vafcls_cross_validate.py --first-fold 5 --last-fold 29 --seeds 3 \ + -o docs/cv_seed_std.csv --save-seed-std # the seed-noise figure +python uored_vafcls_eval.py -l pretrained_models/akdcnn_uored_vafcls.h5 --save-metrics # params +python uored_vafcls_eval.py -l pretrained_models/akdcnn_uored_vafcls_qat.fbz --save-metrics # sparsity +python uored_vafcls_benchmark.py -l pretrained_models/akdcnn_uored_vafcls_qat.fbz --save-metrics + +# Naive segment-level split (Model Card's top row). +# One model, one run: the split is singular and its score is stable to ~0.001. +bash uored_vafcls_naive_split.sh +python uored_vafcls_eval.py -l models/akdcnn_uored_vafcls_segment.h5 --split-mode segment --save-metrics +python uored_vafcls_eval.py -l models/akdcnn_uored_vafcls_segment_qat.h5 --split-mode segment --save-metrics +python uored_vafcls_eval.py -l models/akdcnn_uored_vafcls_segment_qat.fbz --split-mode segment --save-metrics + +python update_readme.py +``` + +Each key in `metrics.json` has exactly one writer, so the scripts can be run in any order without racing. There is deliberately no bearing-level single-model AUROC: nothing in the README reports one, because reporting one is the mistake this example is about. + +**The standing rule:** any change to the architecture, the training recipe or the `ENCODE_*` constants invalidates the cross-validated table, and requires a fresh 100-fold sweep. Re-running the fixed fold is *not* a substitute — that is the whole point of this example. diff --git a/akida1/model_zoo/uored_vafcls/docs/cv_results.csv b/akida1/model_zoo/uored_vafcls/docs/cv_results.csv new file mode 100644 index 0000000..fb095a6 --- /dev/null +++ b/akida1/model_zoo/uored_vafcls/docs/cv_results.csv @@ -0,0 +1,101 @@ +fold,seed,float_auroc,qat_auroc,akida_auroc,float_auroc_inner,float_auroc_outer,float_auroc_ball,float_auroc_cage,akida_auroc_inner,akida_auroc_outer,akida_auroc_ball,akida_auroc_cage,params,n_train_windows,n_test_windows,seconds +5,0,0.979125,0.967891,0.966000,0.999625,0.970875,0.957875,0.988125,0.996313,0.933688,0.935250,0.998750,335444,720,240,23.8 +6,0,0.932438,0.926438,0.926406,0.988750,1.000000,0.995750,0.745250,0.986250,0.999875,0.992750,0.726750,335444,720,240,20.9 +7,0,0.850781,0.859094,0.858578,0.988750,0.894500,0.793875,0.726000,0.989375,0.864688,0.831750,0.748500,335444,720,240,20.4 +8,0,0.882875,0.894594,0.897125,0.914250,0.989500,0.984625,0.643125,0.947187,0.895500,0.988750,0.757062,335444,720,240,20.7 +9,0,0.932313,0.959172,0.960812,0.972250,0.960750,0.997375,0.798875,0.984750,0.951875,0.996125,0.910500,335444,720,240,21.0 +10,0,0.862469,0.881625,0.882266,0.943125,0.897750,0.965750,0.643250,0.949250,0.911375,0.952813,0.715625,335444,720,240,20.1 +11,0,0.843313,0.794547,0.793922,0.920625,0.951500,0.986250,0.514875,0.908563,0.914438,0.981875,0.370812,335444,720,240,20.6 +12,0,0.880750,0.918297,0.919672,0.949000,1.000000,0.918875,0.655125,0.955375,0.998125,0.961688,0.763500,335444,720,240,20.2 +13,0,0.885594,0.867422,0.868688,0.945125,0.957250,0.972375,0.667625,0.935125,0.943375,0.960812,0.635437,335444,720,240,20.2 +14,0,0.934594,0.931766,0.935609,0.993250,0.934125,0.951000,0.860000,0.988125,0.925812,0.948875,0.879625,335444,720,240,20.6 +15,0,0.811937,0.774172,0.773297,0.910250,1.000000,0.925875,0.411625,0.904563,1.000000,0.914500,0.274125,335444,720,240,20.2 +16,0,0.841656,0.829406,0.828625,0.926375,0.994625,0.985000,0.460625,0.931812,0.988000,0.982625,0.412062,335444,720,240,20.2 +17,0,0.877969,0.866922,0.865719,0.927625,0.982750,0.969375,0.632125,0.937125,0.944750,0.991875,0.589125,335444,720,240,20.6 +18,0,0.849938,0.813516,0.819438,0.928250,1.000000,0.999250,0.472250,0.937937,0.998500,0.985750,0.355563,335444,720,240,20.2 +19,0,0.913594,0.888219,0.896406,0.990875,0.995375,0.933125,0.735000,0.990438,0.995500,0.889437,0.710250,335444,720,240,20.1 +20,0,0.955844,0.973375,0.976031,0.999375,0.933750,0.958125,0.932125,0.999250,0.947063,0.971625,0.986187,335444,720,240,20.7 +21,0,0.963094,0.951156,0.950563,0.995750,0.995125,0.985500,0.876000,0.994000,0.992563,0.991000,0.824688,335444,720,240,20.3 +22,0,0.887844,0.919125,0.919766,0.990500,0.901250,0.881000,0.778625,0.990250,0.952750,0.871125,0.864938,335444,720,240,20.8 +23,0,0.876187,0.904187,0.899844,0.939625,1.000000,0.978125,0.587000,0.940562,0.998750,0.985313,0.674750,335444,720,240,20.2 +24,0,0.938312,0.936922,0.938719,0.986875,1.000000,0.973625,0.792750,0.986000,0.999875,0.985750,0.783250,335444,720,240,20.7 +25,0,0.792750,0.800828,0.798906,0.954250,0.938500,0.902875,0.375375,0.975000,0.956688,0.932000,0.331937,335444,720,240,20.2 +26,0,0.970656,0.981297,0.984141,0.999875,1.000000,0.994375,0.888375,0.999125,0.989500,0.984500,0.963437,335444,720,240,20.2 +27,0,0.895281,0.910016,0.908906,0.988875,1.000000,0.964375,0.627875,0.989187,0.996875,0.963000,0.686562,335444,720,240,20.1 +28,0,0.954750,0.933781,0.926016,0.994000,0.997375,0.978875,0.848750,0.994000,0.998500,0.933062,0.778500,335444,720,240,20.9 +29,0,0.948406,0.935656,0.939234,1.000000,0.844875,0.951250,0.997500,0.998125,0.816688,0.945750,0.996375,335444,720,240,20.2 +30,0,0.867687,0.850797,0.850141,0.935500,0.989375,0.973500,0.572375,0.931000,0.979812,0.965438,0.524312,335444,720,240,20.2 +31,0,0.924531,0.922141,0.912797,0.988750,0.984875,0.931000,0.793500,0.987875,0.984625,0.902938,0.775750,335444,720,240,20.3 +32,0,0.949437,0.950031,0.949594,0.998750,0.999250,0.893125,0.906625,0.997250,0.965500,0.915937,0.919687,335444,720,240,20.1 +33,0,0.817406,0.797625,0.794562,0.907500,0.992000,0.946375,0.423750,0.929063,0.993125,0.939125,0.316937,335444,720,240,21.0 +34,0,0.826187,0.817078,0.816734,0.925750,0.969875,0.984250,0.424875,0.926500,0.961063,0.977750,0.401625,335444,720,240,20.3 +35,0,0.923812,0.922609,0.914281,0.990625,1.000000,0.905000,0.799625,0.990500,0.977187,0.905875,0.783563,335444,720,240,20.7 +36,0,0.856906,0.864906,0.870109,0.950750,0.986750,0.954500,0.535625,0.952500,0.986000,0.959062,0.582875,335444,720,240,20.2 +37,0,0.736344,0.742844,0.741734,0.898875,0.998750,0.889750,0.158000,0.892375,0.999375,0.898375,0.176812,335444,720,240,20.7 +38,0,0.879063,0.862219,0.863937,0.931750,0.996625,0.979000,0.608875,0.930125,0.995625,0.973375,0.556625,335444,720,240,21.1 +39,0,0.885344,0.909937,0.911594,0.929000,0.996000,0.973500,0.642875,0.945063,0.933437,0.982812,0.785063,335444,720,240,20.3 +40,0,0.882375,0.846641,0.843406,0.947125,0.912625,0.965750,0.704000,0.957125,0.783688,0.958125,0.674687,335444,720,240,20.2 +41,0,0.934656,0.941109,0.940469,0.988750,0.990000,0.911125,0.848750,0.987562,0.980687,0.899812,0.893813,335444,720,240,20.2 +42,0,0.902969,0.893516,0.892766,0.943750,0.994500,0.993375,0.680250,0.942625,0.996000,1.000000,0.632437,335444,720,240,20.2 +43,0,0.936063,0.888469,0.894266,0.996375,0.886750,0.928375,0.932750,0.973875,0.833625,0.800937,0.968625,335444,720,240,20.7 +44,0,0.838187,0.814359,0.808875,0.912750,0.858000,0.906125,0.675875,0.916188,0.835813,0.832000,0.651500,335444,720,240,21.1 +45,0,0.952437,0.951875,0.948641,0.999000,0.905000,0.907875,0.997875,0.996125,0.897625,0.920687,0.980125,335444,720,240,20.3 +46,0,0.929187,0.939875,0.941188,0.928750,1.000000,0.978875,0.809125,0.946937,0.998938,0.987250,0.831625,335444,720,240,20.2 +47,0,0.866625,0.879703,0.877047,0.920500,0.931375,0.969625,0.645000,0.922312,0.932500,0.950562,0.702812,335444,720,240,20.2 +48,0,0.931063,0.934734,0.936266,0.992875,0.987500,0.986500,0.757375,0.994125,0.979375,0.995687,0.775875,335444,720,240,20.1 +49,0,0.940344,0.927766,0.935359,0.992375,0.999125,0.880125,0.889750,0.991000,0.999375,0.846250,0.904812,335444,720,240,20.2 +50,0,0.974938,0.973281,0.974781,0.997750,0.999875,0.908750,0.993375,0.993875,0.992000,0.924750,0.988500,335444,720,240,21.2 +51,0,0.916438,0.907563,0.911000,0.984625,0.989000,0.981625,0.710500,0.983313,0.996437,0.975688,0.688562,335444,720,240,20.2 +52,0,0.860906,0.878031,0.879344,0.922875,1.000000,0.980250,0.540500,0.923500,0.990875,0.988688,0.614313,335444,720,240,20.2 +53,0,0.835906,0.822797,0.827000,0.940375,0.997750,0.975250,0.430250,0.936875,0.987563,0.957375,0.426187,335444,720,240,20.8 +54,0,0.991250,0.988734,0.989219,0.999125,0.990250,0.985500,0.990125,0.998500,0.974875,0.987625,0.995875,335444,720,240,20.8 +55,0,0.734531,0.706812,0.698906,0.974625,0.998625,0.747750,0.217125,0.971063,0.983625,0.726437,0.114500,335444,720,240,20.3 +56,0,0.971906,0.967672,0.969375,0.998625,1.000000,0.922500,0.966500,0.995750,0.998750,0.919188,0.963813,335444,720,240,20.2 +57,0,0.916750,0.912344,0.916203,0.990000,0.940625,0.936250,0.800125,0.987687,0.934312,0.967375,0.775438,335444,720,240,20.2 +58,0,0.939813,0.928781,0.934516,0.996875,0.891375,0.881750,0.989250,0.993250,0.917375,0.845375,0.982062,335444,720,240,21.6 +59,0,0.787063,0.754531,0.753016,0.952750,0.972500,0.906375,0.316625,0.957188,0.971125,0.912438,0.171312,335444,720,240,20.3 +60,0,0.887375,0.946578,0.946125,0.934000,0.994750,0.972500,0.648250,0.970625,0.996750,0.961125,0.856000,335444,720,240,20.8 +61,0,0.953625,0.976094,0.976234,0.997875,0.988875,0.964875,0.862875,0.998625,0.996125,0.958000,0.952187,335444,720,240,20.3 +62,0,0.884062,0.909625,0.903906,0.990125,0.935625,0.864875,0.745625,0.989000,0.942500,0.859062,0.825063,335444,720,240,20.2 +63,0,0.959406,0.948375,0.951734,0.988250,0.957250,0.904375,0.987750,0.971375,0.957250,0.915875,0.962437,335444,720,240,20.2 +64,0,0.863844,0.854703,0.850344,0.944750,0.987250,0.989750,0.533625,0.971000,0.901188,0.988375,0.540812,335444,720,240,20.7 +65,0,0.919437,0.921109,0.916313,0.989500,0.959625,0.997375,0.731250,0.991812,0.952188,0.996500,0.724750,335444,720,240,20.3 +66,0,0.715063,0.698906,0.700969,0.923375,0.951125,0.884625,0.101125,0.927250,0.943812,0.882938,0.049875,335444,720,240,21.6 +67,0,0.974844,0.966859,0.966062,0.998250,0.992125,0.925250,0.983750,0.994062,0.967313,0.922312,0.980563,335444,720,240,20.7 +68,0,0.917719,0.902469,0.900094,0.993000,0.954875,0.964500,0.758500,0.994250,0.933937,0.964875,0.707313,335444,720,240,20.7 +69,0,0.899969,0.894016,0.899422,0.991625,0.983375,0.900125,0.724750,0.987812,0.978313,0.883687,0.747875,335444,720,240,20.7 +70,0,0.921656,0.922984,0.928984,1.000000,0.965375,0.844500,0.876750,0.999500,0.956000,0.860937,0.899500,335444,720,240,20.2 +71,0,0.932594,0.910281,0.910547,0.998625,0.996125,0.899875,0.835750,0.997375,0.970125,0.903375,0.771313,335444,720,240,20.2 +72,0,0.893781,0.883547,0.881062,0.958625,1.000000,0.986625,0.629875,0.956250,0.984688,0.987437,0.595875,335444,720,240,20.2 +73,0,0.844094,0.833828,0.836656,0.902375,0.958250,0.981875,0.533875,0.902375,0.924875,0.979437,0.539938,335444,720,240,20.2 +74,0,0.851656,0.847391,0.851844,0.957000,0.941125,0.963250,0.545250,0.958500,0.918687,0.978000,0.552187,335444,720,240,20.2 +75,0,0.903875,0.882203,0.895359,0.991000,0.994500,0.905875,0.724125,0.989250,0.896875,0.915875,0.779438,335444,720,240,22.3 +76,0,0.914219,0.912469,0.911000,0.991125,0.991375,0.971125,0.703250,0.985313,0.993875,0.971187,0.693625,335444,720,240,20.8 +77,0,0.976406,0.983969,0.983797,0.999750,0.967375,0.997375,0.941125,0.999375,0.967187,0.998500,0.970125,335444,720,240,20.2 +78,0,0.877719,0.856625,0.849391,0.928000,0.903750,0.927125,0.752000,0.929500,0.879938,0.835250,0.752875,335444,720,240,20.3 +79,0,0.940844,0.911687,0.915250,0.977625,1.000000,0.970375,0.815375,0.973875,0.993375,0.968625,0.725125,335444,720,240,20.3 +80,0,0.760531,0.767922,0.765781,0.907375,0.959000,0.919875,0.255875,0.904000,0.940187,0.912500,0.306438,335444,720,240,20.2 +81,0,0.921969,0.922672,0.921375,0.987375,1.000000,0.967000,0.733500,0.987000,0.990125,0.968750,0.739625,335444,720,240,20.2 +82,0,0.866969,0.897500,0.891516,0.942000,0.997000,0.982750,0.546125,0.940625,0.989812,0.988375,0.647250,335444,720,240,20.2 +83,0,0.894688,0.889031,0.884266,0.991625,0.940750,0.956625,0.689750,0.990750,0.891687,0.964812,0.689813,335444,720,240,20.1 +84,0,0.907594,0.911594,0.906266,0.988875,0.931750,0.962625,0.747125,0.980187,0.927188,0.939937,0.777750,335444,720,240,20.2 +85,0,0.900469,0.950688,0.947953,0.936625,1.000000,0.994125,0.671125,0.950187,0.996375,0.998062,0.847187,335444,720,240,22.0 +86,0,0.911438,0.918641,0.920641,0.988125,0.956750,0.932875,0.768000,0.983250,0.962813,0.928000,0.808500,335444,720,240,20.3 +87,0,0.906969,0.926219,0.926625,0.976625,0.994625,0.960375,0.696250,0.969500,0.999375,0.950438,0.787188,335444,720,240,20.3 +88,0,0.861594,0.915984,0.916703,0.902750,0.987125,0.965250,0.591250,0.922813,0.987437,0.981812,0.774750,335444,720,240,20.8 +89,0,0.876250,0.886719,0.887969,0.921125,1.000000,0.970875,0.613000,0.923125,0.997563,0.972375,0.658813,335444,720,240,20.7 +90,0,0.933250,0.933844,0.929688,0.989625,0.987250,0.989000,0.767125,0.988875,0.982000,0.953875,0.794000,335444,720,240,20.3 +91,0,0.944219,0.933094,0.934234,0.999000,1.000000,0.892500,0.885375,0.995500,0.999000,0.907000,0.835438,335444,720,240,20.2 +92,0,0.932094,0.919953,0.928609,0.990500,0.972375,0.982125,0.783375,0.992375,0.969000,0.974187,0.778875,335444,720,240,20.4 +93,0,0.717594,0.704172,0.710453,0.901250,1.000000,0.831625,0.137500,0.898375,0.997500,0.852438,0.093500,335444,720,240,20.3 +94,0,0.969906,0.986656,0.988906,1.000000,0.974000,0.978375,0.927250,1.000000,0.990125,0.978625,0.986875,335444,720,240,20.2 +95,0,0.792875,0.797063,0.797984,0.930875,0.998250,0.955250,0.287125,0.933250,0.998438,0.954437,0.305812,335444,720,240,20.8 +96,0,0.902062,0.916875,0.907047,0.998250,1.000000,0.809750,0.800250,0.997875,0.990250,0.794000,0.846062,335444,720,240,22.2 +97,0,0.924625,0.925484,0.926500,0.999250,1.000000,0.838875,0.860375,0.995750,0.999875,0.842750,0.867625,335444,720,240,20.9 +98,0,0.886656,0.916906,0.909531,0.958250,0.962875,0.920375,0.705125,0.966375,0.944000,0.930375,0.797375,335444,720,240,20.2 +99,0,0.704406,0.713875,0.714391,0.931125,0.949875,0.873500,0.063125,0.923813,0.923437,0.885875,0.124437,335444,720,240,20.3 +100,0,0.979313,0.978125,0.973250,0.999875,0.986000,0.943625,0.987750,0.999000,0.958937,0.937688,0.997375,335444,720,240,20.9 +101,0,0.947562,0.935062,0.937781,0.999250,0.930625,0.985375,0.875000,0.992938,0.871313,0.956250,0.930625,335444,720,240,20.3 +102,0,0.927844,0.941531,0.938031,0.999250,0.999500,0.828500,0.884125,0.997875,0.999500,0.838000,0.916750,335444,720,240,20.4 +103,0,0.974625,0.963188,0.964219,0.998875,0.932500,0.983750,0.983375,0.994750,0.909750,0.960062,0.992312,335444,720,240,20.8 +104,0,0.951750,0.952953,0.953016,0.989875,0.998500,0.995375,0.823250,0.990125,0.999500,0.993375,0.829063,335444,720,240,20.2 diff --git a/akida1/model_zoo/uored_vafcls/docs/cv_seed_std.csv b/akida1/model_zoo/uored_vafcls/docs/cv_seed_std.csv new file mode 100644 index 0000000..5206826 --- /dev/null +++ b/akida1/model_zoo/uored_vafcls/docs/cv_seed_std.csv @@ -0,0 +1,76 @@ +fold,seed,float_auroc,qat_auroc,akida_auroc,float_auroc_inner,float_auroc_outer,float_auroc_ball,float_auroc_cage,akida_auroc_inner,akida_auroc_outer,akida_auroc_ball,akida_auroc_cage,params,n_train_windows,n_test_windows,seconds +5,0,0.979125,0.967891,0.966000,0.999625,0.970875,0.957875,0.988125,0.996313,0.933688,0.935250,0.998750,335444,720,240,23.8 +5,1,0.946406,0.959844,0.964406,0.998500,0.874125,0.927375,0.985625,0.995250,0.912687,0.954812,0.994875,335444,720,240,20.6 +5,2,0.980594,0.974578,0.976813,0.999875,0.956125,0.975250,0.991125,0.998563,0.941125,0.973375,0.994188,335444,720,240,20.8 +6,0,0.932438,0.926438,0.926406,0.988750,1.000000,0.995750,0.745250,0.986250,0.999875,0.992750,0.726750,335444,720,240,20.2 +6,1,0.943250,0.931000,0.932812,0.986625,1.000000,1.000000,0.786375,0.986000,0.987625,0.995750,0.761875,335444,720,240,20.5 +6,2,0.937000,0.929797,0.928156,0.991500,0.997875,0.922750,0.835875,0.991500,0.997375,0.891688,0.832062,335444,720,240,20.1 +7,0,0.850781,0.859094,0.858578,0.988750,0.894500,0.793875,0.726000,0.989375,0.864688,0.831750,0.748500,335444,720,240,20.5 +7,1,0.861219,0.865828,0.866703,0.991625,0.941375,0.770375,0.741500,0.995250,0.930937,0.785062,0.755563,335444,720,240,20.2 +7,2,0.892188,0.881828,0.881984,0.989000,0.971500,0.873250,0.735000,0.991875,0.976000,0.794937,0.765125,335444,720,240,20.2 +8,0,0.882875,0.894594,0.897125,0.914250,0.989500,0.984625,0.643125,0.947187,0.895500,0.988750,0.757062,335444,720,240,21.2 +8,1,0.927125,0.900172,0.901672,0.945750,0.980125,0.998000,0.784625,0.937125,0.967562,0.990563,0.711438,335444,720,240,20.3 +8,2,0.933125,0.920063,0.921094,0.946375,0.916500,0.929250,0.940375,0.938125,0.912375,0.973750,0.860125,335444,720,240,20.7 +9,0,0.932313,0.959172,0.960812,0.972250,0.960750,0.997375,0.798875,0.984750,0.951875,0.996125,0.910500,335444,720,240,20.6 +9,1,0.976125,0.948141,0.946578,0.989750,0.947250,0.999125,0.968375,0.983000,0.913375,0.997437,0.892500,335444,720,240,20.2 +9,2,0.975875,0.971734,0.972766,0.980375,0.926875,0.996875,0.999375,0.986000,0.935125,0.994063,0.975875,335444,720,240,20.7 +10,0,0.862469,0.881625,0.882266,0.943125,0.897750,0.965750,0.643250,0.949250,0.911375,0.952813,0.715625,335444,720,240,21.3 +10,1,0.850281,0.835594,0.833234,0.965500,0.799375,0.759875,0.876375,0.948813,0.868000,0.722437,0.793688,335444,720,240,20.2 +10,2,0.931312,0.920687,0.922469,0.952750,0.974625,0.832750,0.965125,0.967313,0.977125,0.951812,0.793625,335444,720,240,20.2 +11,0,0.843313,0.794547,0.793922,0.920625,0.951500,0.986250,0.514875,0.908563,0.914438,0.981875,0.370812,335444,720,240,20.2 +11,1,0.900719,0.824562,0.827266,0.924875,0.972375,0.992625,0.713000,0.924000,0.945875,0.991625,0.447563,335444,720,240,21.0 +11,2,0.856375,0.840063,0.837984,0.916750,0.941375,0.950500,0.616875,0.916750,0.936312,0.972625,0.526250,335444,720,240,20.2 +12,0,0.880750,0.918297,0.919672,0.949000,1.000000,0.918875,0.655125,0.955375,0.998125,0.961688,0.763500,335444,720,240,20.3 +12,1,0.957219,0.926156,0.929641,0.999500,1.000000,0.922125,0.907250,0.992312,0.984688,0.972750,0.768813,335444,720,240,20.2 +12,2,0.978125,0.961562,0.956906,0.988750,1.000000,0.953875,0.969875,0.989125,0.999312,0.918375,0.920812,335444,720,240,20.9 +13,0,0.885594,0.867422,0.868688,0.945125,0.957250,0.972375,0.667625,0.935125,0.943375,0.960812,0.635437,335444,720,240,20.3 +13,1,0.891469,0.839375,0.842906,0.968875,0.927500,0.827000,0.842500,0.940000,0.953063,0.778188,0.700375,335444,720,240,20.7 +13,2,0.953219,0.942297,0.942109,0.992125,0.958000,0.911250,0.951500,0.990500,0.966625,0.908687,0.902625,335444,720,240,20.8 +14,0,0.934594,0.931766,0.935609,0.993250,0.934125,0.951000,0.860000,0.988125,0.925812,0.948875,0.879625,335444,720,240,20.8 +14,1,0.912562,0.897922,0.906141,0.990875,0.971375,0.857875,0.830125,0.992625,0.959125,0.846000,0.826812,335444,720,240,21.5 +14,2,0.934344,0.929578,0.936813,0.995000,0.993750,0.872375,0.876250,0.997000,0.985125,0.875250,0.889875,335444,720,240,20.3 +15,0,0.811937,0.774172,0.773297,0.910250,1.000000,0.925875,0.411625,0.904563,1.000000,0.914500,0.274125,335444,720,240,20.8 +15,1,0.804438,0.801859,0.800844,0.920625,1.000000,0.930500,0.366625,0.918000,1.000000,0.951375,0.334000,335444,720,240,20.3 +15,2,0.830594,0.770281,0.765828,0.919750,0.993125,0.908500,0.501000,0.918500,0.995250,0.803750,0.345813,335444,720,240,20.2 +16,0,0.841656,0.829406,0.828625,0.926375,0.994625,0.985000,0.460625,0.931812,0.988000,0.982625,0.412062,335444,720,240,21.1 +16,1,0.864562,0.818828,0.815672,0.927875,0.975000,0.996625,0.558750,0.919500,0.975000,0.995000,0.373188,335444,720,240,20.3 +16,2,0.857719,0.852500,0.851453,0.910250,0.988750,0.973875,0.558000,0.907000,0.976750,0.987313,0.534750,335444,720,240,20.8 +17,0,0.877969,0.866922,0.865719,0.927625,0.982750,0.969375,0.632125,0.937125,0.944750,0.991875,0.589125,335444,720,240,20.8 +17,1,0.893437,0.817438,0.818359,0.924500,0.998750,0.996000,0.654500,0.917625,0.977750,0.992500,0.385562,335444,720,240,20.3 +17,2,0.895312,0.863078,0.863328,0.916125,0.954500,0.961375,0.749250,0.917125,0.931688,0.965625,0.638875,335444,720,240,20.2 +18,0,0.849938,0.813516,0.819438,0.928250,1.000000,0.999250,0.472250,0.937937,0.998500,0.985750,0.355563,335444,720,240,21.1 +18,1,0.891937,0.849562,0.850281,0.934375,0.997000,0.970000,0.666375,0.942500,0.999625,0.939625,0.519375,335444,720,240,20.8 +18,2,0.876594,0.837953,0.847953,0.923625,1.000000,0.991000,0.591750,0.923000,0.994812,0.985750,0.488250,335444,720,240,20.2 +19,0,0.913594,0.888219,0.896406,0.990875,0.995375,0.933125,0.735000,0.990438,0.995500,0.889437,0.710250,335444,720,240,20.8 +19,1,0.865406,0.875297,0.875391,0.994500,0.890125,0.875500,0.701500,0.995375,0.959500,0.845813,0.700875,335444,720,240,20.2 +19,2,0.896656,0.909844,0.908078,0.991375,0.959625,0.944000,0.691625,0.993187,0.933625,0.927687,0.777812,335444,720,240,20.8 +20,0,0.955844,0.973375,0.976031,0.999375,0.933750,0.958125,0.932125,0.999250,0.947063,0.971625,0.986187,335444,720,240,21.4 +20,1,0.957781,0.931953,0.936922,0.993125,0.941125,0.994125,0.902750,0.986125,0.944625,0.990125,0.826812,335444,720,240,20.3 +20,2,0.932000,0.921672,0.927625,0.999875,0.910750,0.963375,0.854000,0.999500,0.934500,0.944375,0.832125,335444,720,240,20.8 +21,0,0.963094,0.951156,0.950563,0.995750,0.995125,0.985500,0.876000,0.994000,0.992563,0.991000,0.824688,335444,720,240,20.3 +21,1,0.929063,0.945109,0.940344,0.992875,0.999875,0.968375,0.755125,0.994375,0.994563,0.965938,0.806500,335444,720,240,20.3 +21,2,0.926375,0.960391,0.960969,0.994375,0.976500,0.972125,0.762500,0.995875,0.990875,0.959125,0.898000,335444,720,240,20.7 +22,0,0.887844,0.919125,0.919766,0.990500,0.901250,0.881000,0.778625,0.990250,0.952750,0.871125,0.864938,335444,720,240,20.7 +22,1,0.805250,0.838719,0.836141,0.993750,0.818125,0.575000,0.834125,0.995813,0.864875,0.679375,0.804500,335444,720,240,20.2 +22,2,0.872938,0.855969,0.844891,0.996375,0.946625,0.683875,0.864875,0.994625,0.902062,0.688000,0.794875,335444,720,240,22.1 +23,0,0.876187,0.904187,0.899844,0.939625,1.000000,0.978125,0.587000,0.940562,0.998750,0.985313,0.674750,335444,720,240,20.8 +23,1,0.901625,0.819031,0.813453,0.917875,1.000000,0.997000,0.691625,0.928562,0.976312,0.999625,0.349313,335444,720,240,20.2 +23,2,0.884500,0.867766,0.859688,0.907125,1.000000,0.976875,0.654000,0.909500,1.000000,0.994563,0.534687,335444,720,240,20.2 +24,0,0.938312,0.936922,0.938719,0.986875,1.000000,0.973625,0.792750,0.986000,0.999875,0.985750,0.783250,335444,720,240,20.2 +24,1,0.921125,0.920844,0.916531,0.991750,0.994250,0.953500,0.745000,0.996938,0.993750,0.902688,0.772750,335444,720,240,20.2 +24,2,0.915750,0.904922,0.898531,0.990000,1.000000,0.923000,0.750000,0.993500,0.999750,0.865188,0.735688,335444,720,240,20.8 +25,0,0.792750,0.800828,0.798906,0.954250,0.938500,0.902875,0.375375,0.975000,0.956688,0.932000,0.331937,335444,720,240,20.3 +25,1,0.795125,0.755359,0.755484,0.965375,0.902125,0.865250,0.447750,0.952250,0.918375,0.879375,0.271937,335444,720,240,21.8 +25,2,0.835219,0.744953,0.747734,0.953000,0.946000,0.866250,0.575625,0.950375,0.947437,0.862313,0.230813,335444,720,240,20.3 +26,0,0.970656,0.981297,0.984141,0.999875,1.000000,0.994375,0.888375,0.999125,0.989500,0.984500,0.963437,335444,720,240,20.2 +26,1,0.974656,0.979938,0.981562,0.992750,1.000000,0.996375,0.909500,0.991750,0.998750,0.994750,0.941000,335444,720,240,20.2 +26,2,0.972344,0.971078,0.973281,0.999875,0.995500,0.986000,0.908000,1.000000,0.983313,0.940187,0.969625,335444,720,240,20.2 +27,0,0.895281,0.910016,0.908906,0.988875,1.000000,0.964375,0.627875,0.989187,0.996875,0.963000,0.686562,335444,720,240,20.2 +27,1,0.854687,0.846656,0.857531,0.989625,0.981750,0.842000,0.605375,0.990375,0.965250,0.846438,0.628062,335444,720,240,20.2 +27,2,0.873250,0.850812,0.847875,0.988625,0.991250,0.867500,0.645625,0.987875,0.974875,0.761813,0.666938,335444,720,240,20.1 +28,0,0.954750,0.933781,0.926016,0.994000,0.997375,0.978875,0.848750,0.994000,0.998500,0.933062,0.778500,335444,720,240,20.4 +28,1,0.937281,0.926234,0.922031,0.991375,0.993875,0.942000,0.821875,0.994562,0.997000,0.915813,0.780750,335444,720,240,21.9 +28,2,0.943437,0.952188,0.955469,0.996500,0.964000,0.992875,0.820375,0.995750,0.978563,0.990625,0.856937,335444,720,240,20.4 +29,0,0.948406,0.935656,0.939234,1.000000,0.844875,0.951250,0.997500,0.998125,0.816688,0.945750,0.996375,335444,720,240,20.4 +29,1,0.855062,0.895766,0.893828,1.000000,0.782000,0.643500,0.994750,1.000000,0.834688,0.753000,0.987625,335444,720,240,20.3 +29,2,0.897812,0.862734,0.857828,0.999875,0.882250,0.755875,0.953250,0.999375,0.815125,0.773938,0.842875,335444,720,240,20.2 diff --git a/akida1/model_zoo/uored_vafcls/docs/metrics.json b/akida1/model_zoo/uored_vafcls/docs/metrics.json new file mode 100644 index 0000000..25b7e04 --- /dev/null +++ b/akida1/model_zoo/uored_vafcls/docs/metrics.json @@ -0,0 +1,44 @@ +{ + "segment_float_auroc": "0.9983", + "segment_qat_auroc": "0.9963", + "segment_akida_auroc": "0.9973", + "segment_params": "335,444", + "segment_sparsity": "39.06%", + "cv_folds": "100", + "cv_seeds": "1", + "cv_float_auroc_mean": "0.8950", + "cv_qat_auroc_mean": "0.8939", + "cv_akida_auroc_mean": "0.8939", + "cv_akida_auroc_std": "0.0666", + "cv_akida_auroc_min": "0.6989", + "cv_akida_auroc_max": "0.9892", + "cv_seed_std": "0.0186", + "cv_seed_std_folds": "25", + "params": "335,444", + "sparsity": "37.81%", + "fixed_fold": "42", + "minimal_nps": "17", + "minimal_passes": "1", + "minimal_cycles": "3930879", + "minimal_latency_ms": "9.827", + "minimal_total_P": "179.2", + "minimal_total_E": "1.782", + "minimal_dyn_P": "67.5", + "minimal_dyn_E": "0.671", + "allnps_nps": "27", + "allnps_passes": "1", + "allnps_cycles": "2932933", + "allnps_latency_ms": "7.332", + "allnps_total_P": "201.9", + "allnps_total_E": "1.506", + "allnps_dyn_P": "89.0", + "allnps_dyn_E": "0.664", + "hwpr_nps": "64", + "hwpr_passes": "3", + "hwpr_cycles": "3063152", + "hwpr_latency_ms": "7.658", + "hwpr_total_P": "216.2", + "hwpr_total_E": "1.791", + "hwpr_dyn_P": "103.1", + "hwpr_dyn_E": "0.854" +} diff --git a/akida1/model_zoo/uored_vafcls/docs/ref_benchmark_results_full.png b/akida1/model_zoo/uored_vafcls/docs/ref_benchmark_results_full.png new file mode 100644 index 0000000..4717b1b Binary files /dev/null and b/akida1/model_zoo/uored_vafcls/docs/ref_benchmark_results_full.png differ diff --git a/akida1/model_zoo/uored_vafcls/docs/ref_benchmark_results_layers.png b/akida1/model_zoo/uored_vafcls/docs/ref_benchmark_results_layers.png new file mode 100644 index 0000000..5d632ab Binary files /dev/null and b/akida1/model_zoo/uored_vafcls/docs/ref_benchmark_results_layers.png differ diff --git a/akida1/model_zoo/uored_vafcls/docs/ref_cv_auroc_distribution.png b/akida1/model_zoo/uored_vafcls/docs/ref_cv_auroc_distribution.png new file mode 100644 index 0000000..ecf735c Binary files /dev/null and b/akida1/model_zoo/uored_vafcls/docs/ref_cv_auroc_distribution.png differ diff --git a/akida1/model_zoo/uored_vafcls/docs/signal_mosaic.png b/akida1/model_zoo/uored_vafcls/docs/signal_mosaic.png new file mode 100644 index 0000000..66a8708 Binary files /dev/null and b/akida1/model_zoo/uored_vafcls/docs/signal_mosaic.png differ diff --git a/akida1/model_zoo/uored_vafcls/models/.gitignore b/akida1/model_zoo/uored_vafcls/models/.gitignore new file mode 100644 index 0000000..f353a40 --- /dev/null +++ b/akida1/model_zoo/uored_vafcls/models/.gitignore @@ -0,0 +1,4 @@ +# Git to Ignore everything in this directory +* +# Except this .gitignore file +!.gitignore diff --git a/akida1/model_zoo/uored_vafcls/pretrained_models/akdcnn_uored_vafcls.h5 b/akida1/model_zoo/uored_vafcls/pretrained_models/akdcnn_uored_vafcls.h5 new file mode 100644 index 0000000..623b289 --- /dev/null +++ b/akida1/model_zoo/uored_vafcls/pretrained_models/akdcnn_uored_vafcls.h5 @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:5f933fed5f6c48586e7170230c048a24d9a881c5e05d923ec7c5653094977510 +size 1434032 diff --git a/akida1/model_zoo/uored_vafcls/pretrained_models/akdcnn_uored_vafcls_qat.fbz b/akida1/model_zoo/uored_vafcls/pretrained_models/akdcnn_uored_vafcls_qat.fbz new file mode 100644 index 0000000..dfbb42b --- /dev/null +++ b/akida1/model_zoo/uored_vafcls/pretrained_models/akdcnn_uored_vafcls_qat.fbz @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:b4b4198b77ed99112f671ec81257d8630484ee6d6a8e737b6636faadf1e32cf2 +size 402863 diff --git a/akida1/model_zoo/uored_vafcls/pretrained_models/akdcnn_uored_vafcls_qat.h5 b/akida1/model_zoo/uored_vafcls/pretrained_models/akdcnn_uored_vafcls_qat.h5 new file mode 100644 index 0000000..9cb66e8 --- /dev/null +++ b/akida1/model_zoo/uored_vafcls/pretrained_models/akdcnn_uored_vafcls_qat.h5 @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:12b460ce62a9a058cee0bb86998f8bdebff0a92cd3bfd81f56c7e9f913a49d4d +size 1386544 diff --git a/akida1/model_zoo/uored_vafcls/uored_vafcls_benchmark.py b/akida1/model_zoo/uored_vafcls/uored_vafcls_benchmark.py new file mode 100644 index 0000000..a5b36ce --- /dev/null +++ b/akida1/model_zoo/uored_vafcls/uored_vafcls_benchmark.py @@ -0,0 +1,197 @@ +#!/usr/bin/env python +# Copyright 2026 Brainchip Holdings Ltd. Apache 2.0 License +""" +UORED-VAFCLS per-layer hardware benchmark. + +Runs a per-layer timing benchmark on an Akida UORED-VAFCLS model, prints a +summary table, and generates summary plots. + +The numbers describe one model trained on one fold. They are a property of the +architecture on this hardware, not of the model's diagnostic performance - for +that see uored_vafcls_cross_validate.py. + +Example +------- + python uored_vafcls_benchmark.py -l models/akdcnn_uored_vafcls_qat.fbz +""" +import argparse +import json +import pathlib +import sys +import time + +import akida +import numpy as np +from akida_models.sparsity import compute_sparsity + +from brainchip_utils.hardware_utils import (full_model_benchmark, + get_akida_device, + get_mapping_stats, + per_layer_benchmark) +from brainchip_utils.plot_utils import (plot_full_model_results, + plot_per_layer_results, + pretty_print_sparsity) +from uored_vafcls_data import FIXED_FOLD, get_samples + +if __name__ == '__main__': + parser = argparse.ArgumentParser( + description='Per-layer hardware benchmark for an Akida UORED-VAFCLS model') + parser.add_argument('-l', '--loadmodel', required=True, + help='Model to load (.fbz akida model)') + parser.add_argument('-d', '--data', default='./data/uored_vafcls', + help='Directory holding the prepared .npz cache') + parser.add_argument('--fold', type=int, default=FIXED_FOLD, + help='Fold whose training recordings the benchmark ' + 'windows are drawn from') + parser.add_argument('--seed', type=int, default=0, + help='Random seed for the benchmark sample draw') + parser.add_argument('--save-metrics', action='store_true', + help='Write benchmark values to metrics.json. Only ' + 'valid on the fixed fold') + args = parser.parse_args() + + # Power measurement on the integrated device is rather low frequency + # (tens of milliseconds). Ensure that enough samples are used so that + # each run is long enough to ensure multiple measurements, ideal about + # 1 second per recording + NUM_SAMPLES = 100 + + if args.save_metrics and args.fold != FIXED_FOLD: + sys.exit(f'--save-metrics writes the README hardware table, which is ' + f'fold {FIXED_FOLD}; refusing to write fold {args.fold}.') + + # ------------------------------------------------------------------------- + # Model + # ------------------------------------------------------------------------- + ak_model = akida.Model(args.loadmodel) + imsize = tuple(ak_model.input_shape) + + # ------------------------------------------------------------------------- + # Device + # ------------------------------------------------------------------------- + device = get_akida_device(target_version=ak_model.ip_version) + if device is None: + sys.exit('No compatible Akida hardware device found. Skipping benchmarking') + + # TODO: Add a check to get clock frequency specific to device + CLOCK_FREQUENCY = 400e6 # 400 MHz for AKD1500 + + # ------------------------------------------------------------------------- + # Sample + # ------------------------------------------------------------------------- + # Processing in Akida is activity dependent (because it exploits sparsity) + # and that activity is dependent on the input. + # That makes it imperative to use real inputs when benchmarking Akida, + # rather than synthetic random samples. + samples = get_samples(args.data, imsize, num_samples=NUM_SAMPLES, + fold=args.fold, seed=args.seed) + + # ------------------------------------------------------------------------- + # Benchmarks + # ------------------------------------------------------------------------- + # Simple benchmarking + # Run a simple benchmark at batch-size=1, Minimal mapping mode + print('Running Simple Benchmark, Minimal map mode, batch-size=1') + + ak_model.map(device, mode=akida.MapMode.Minimal, hw_only=True) + + inf_clks = [] + inf_times = [] + for rr in range(NUM_SAMPLES): + start_t = time.perf_counter_ns() + ak_model.forward(samples[rr:rr + 1], batch_size=1) + inf_times.append(time.perf_counter_ns() - start_t) + # Get the number of on-device clock cycles for that inference + inf_clks.append(ak_model.metrics['inference_clk']) + + mean_inf_clk = np.mean(inf_clks)/CLOCK_FREQUENCY*1e3 # s to ms + mean_inf_time = np.mean(inf_times)*1e-6 # ns to ms + # The timing reported by the device should be very close to that + # measured on the system + print(f'\n Mean inference time (system clock): {mean_inf_time:.3f} ms ') + print(f' Mean on-chip time (via chip clock cycles): {mean_inf_clk:.3f} ms ') + + # ------------------------------------------------------------------------- + # Full-model benchmark (latency + optional power) + # ------------------------------------------------------------------------- + map_modes = ['Minimal', 'AllNps', 'HwPr'] + POWER_REPEATS = 10 + full_results = dict() + for mm in map_modes: + map_mode = getattr(akida.MapMode, mm) + print(f'\nRunning full-model benchmark (MapMode={mm}, {POWER_REPEATS} repeat(s))...') + full_results[mm] = full_model_benchmark(ak_model, device, samples, + map_mode=map_mode, + repeats=POWER_REPEATS) + if full_results[mm] is None: + sys.exit(f'The model did not map fully to hardware in {mm} mode; ' + f'benchmarking cannot proceed.') + # Re-map without hw_only to populate ak_model.sequences for stats + ak_model.map(device, mode=map_mode) + num_nps, num_passes, num_sequences = get_mapping_stats(ak_model) + full_results[mm]['num_nps'] = num_nps + full_results[mm]['num_passes'] = num_passes + print(f' Mapping: {num_nps} NP(s), {num_passes} pass(es), {num_sequences} sequence(s)') + if num_sequences > 1: + print('WARNING: note, model not completely mapped to hardware') + + + # ------------------------------------------------------------------------- + # Per-layer Benchmark. Minimal mapping mode, batch-size 1 + # ------------------------------------------------------------------------- + ak_model.map(device, mode=akida.MapMode.Minimal, hw_only=True) + ak_model.summary() + + # Check sparsity per-layer + sparsity_dict = compute_sparsity(ak_model, samples=samples) + pretty_print_sparsity(sparsity_dict) + + print(f'Running per-layer benchmark ({NUM_SAMPLES} samples)...') + per_layer_results = per_layer_benchmark(ak_model, device, samples, repeats=NUM_SAMPLES) + + # ------------------------------------------------------------------------- + # Plots + # ------------------------------------------------------------------------- + # Map without hw_only so ak_model.sequences is available for plot_mapping + ak_model.map(device, mode=akida.MapMode.Minimal) + perlayer_savepath = 'benchmark_results_layers.png' + if args.save_metrics: + perlayer_savepath = pathlib.Path(__file__).parent / 'docs' / ('ref_'+perlayer_savepath) + plot_per_layer_results(per_layer_results, ak_model, sparsity_dict, + model_name=args.loadmodel, + savepath=perlayer_savepath) + print('\nPer-layer results plot saved to ' + str(perlayer_savepath)) + + full_savepath = 'benchmark_results_full.png' + if args.save_metrics: + full_savepath = pathlib.Path(__file__).parent / 'docs' / ('ref_'+full_savepath) + plot_full_model_results(full_results, ak_model, device, + model_name=args.loadmodel, + savepath=full_savepath) + print('Full model results plot saved to ' + str(full_savepath)) + + + if args.save_metrics: + # The save-metrics argument used to update the stored metrics that are used to generate the + # performance tables in the README of this folder. + # This should only be used for code maintenance, when the model or training + # pipeline is updated and a new trained model integrated. + # + # Note: 'sparsity' is deliberately NOT written here, but is created by + # uored_vafcls_eval.py. This is so that the accuracy table can be regenerated without + # hardware. + metrics_path = pathlib.Path(__file__).parent / 'docs' / 'metrics.json' + metrics = json.loads(metrics_path.read_text()) if metrics_path.exists() else {} + for mm, res in full_results.items(): + prefix = mm.lower() + metrics[f'{prefix}_nps'] = str(res['num_nps']) + metrics[f'{prefix}_passes'] = str(res['num_passes']) + metrics[f'{prefix}_cycles'] = f'{res["mean_inf_clk"]:.0f}' + metrics[f'{prefix}_latency_ms'] = f'{res["mean_clk_ms"]:.3f}' + if res['power'] is not None: + metrics[f'{prefix}_total_P'] = f'{res["power"]["avg_total_mw"]:.1f}' + metrics[f'{prefix}_total_E'] = f'{res["power"]["avg_energy_mj"]:.3f}' + metrics[f'{prefix}_dyn_P'] = f'{res["power"]["avg_dynamic_mw"]:.1f}' + metrics[f'{prefix}_dyn_E'] = f'{res["power"]["avg_dynamic_energy_mj"]:.3f}' + metrics_path.write_text(json.dumps(metrics, indent=4) + '\n') + print(f'Metrics saved to {metrics_path}') diff --git a/akida1/model_zoo/uored_vafcls/uored_vafcls_cross_validate.py b/akida1/model_zoo/uored_vafcls/uored_vafcls_cross_validate.py new file mode 100644 index 0000000..6fc94c8 --- /dev/null +++ b/akida1/model_zoo/uored_vafcls/uored_vafcls_cross_validate.py @@ -0,0 +1,419 @@ +#!/usr/bin/env python +# Copyright 2026 Brainchip Holdings Ltd. Apache 2.0 License +""" +UORED-VAFCLS cross-validation sweep. + +Runs the whole pipeline - float training, quantization, quantization-aware +tuning, conversion, Akida evaluation - independently on every bearing-disjoint +fold, and reports the mean and spread of macro AUROC for each stage. + +This is the script that produces the headline number for this example, and the +reason it exists is that a single fold does not support a conclusion. There are +only 20 bearings; a fold trains on 12 of them and is tested on 8. The AUROC +spread across folds is 0.04-0.07, and individual folds range from about 0.70 to +about 0.99 with the model and the seed held fixed. Two architectures differing +by a couple of AUROC points cannot be distinguished by one fold, and reporting +one fold's score as a model's performance is reporting noise. + +Folds 0-4 are the tuning budget, used to do hyperparameter tuning. +Folds 5-104 are the 100 evaluation folds, and the mean over them is the +number to quote. + +Results accumulate in docs/cv_results.csv, one row per (fold, seed), flushed +after every fold. A run resumes by default, skipping rows already present, so an +interrupted sweep loses at most one fold. + +Runtime is roughly 30-60 s per fold, so about 1-2 hours for the default +100-fold full chain. --skip-akida drops the quantize/convert/Akida stages and is +about four times faster; use it for recipe sweeps on the first 5 folds. + +Example +------- + # the headline run + python uored_vafcls_cross_validate.py --save-metrics + + # a quick recipe comparison on the tuning folds + python uored_vafcls_cross_validate.py --first-fold 0 --last-fold 4 \\ + --skip-akida -o /tmp/tuning.csv +""" + +import argparse +import csv +import os +import pathlib +import time + +import numpy as np +import tensorflow as tf +from cnn2snn import AkidaVersion, convert, quantize, set_akida_version +from tqdm import tqdm + +from uored_vafcls_data import (BATCH_SIZE, EVAL_FOLDS, FIXED_FOLD, + LABEL_COLUMNS, get_data) +from uored_vafcls_eval import (auroc_scores, evaluate_akida_model, + predict_keras_model) +from uored_vafcls_model import build_uored_vafcls_model +from uored_vafcls_train import train_uored_vafcls + + +tf.config.experimental.enable_op_determinism() + +DEFAULT_OUT = pathlib.Path(__file__).parent / 'docs' / 'cv_results.csv' + +FIELDNAMES = ( + ['fold', 'seed', 'float_auroc', 'qat_auroc', 'akida_auroc'] + + [f'float_auroc_{name}' for name in LABEL_COLUMNS] + + [f'akida_auroc_{name}' for name in LABEL_COLUMNS] + + ['params', 'n_train_windows', 'n_test_windows', 'seconds'] +) + + +def run_fold(fold, seed=0, data_path='./data/uored_vafcls', epochs=30, + learning_rate=2e-4, qat_epochs=10, qat_learning_rate=5e-5, + batch_size=BATCH_SIZE, skip_akida=False, models_dir=None, + verbose=0): + """Run the full pipeline on one fold and return a result row. + + The stage order reproduces uored_vafcls_train.sh exactly, so a fold's + numbers here match what the shell pipeline would produce for the same + (fold, seed). + + Args: + fold (int): bearing-disjoint fold index. + seed (int): random seed for weights, crops, gains and shuffling. + data_path (str): directory holding the prepared .npz cache. + epochs (int): float training epochs. + learning_rate (float): peak LR for float training. + qat_epochs (int): quantization-aware tuning epochs. + qat_learning_rate (float): peak LR for tuning. + batch_size (int): batch size. + skip_akida (bool): stop after float training, leaving the quantized and + Akida columns empty. + models_dir (str, optional): write the per-fold model files here. Off by + default - a hundred folds' worth of checkpoints is a lot of noise. + verbose (int): passed through to model.fit. + + Returns: + dict: one row keyed by FIELDNAMES. + """ + started = time.time() + + model = build_uored_vafcls_model(seed=seed) + train_ds, test_ds = get_data(data_path, model.input_shape[1:], batch_size, + fold=fold, seed=seed) + + row = {name: '' for name in FIELDNAMES} + row.update(fold=fold, seed=seed, params=model.count_params(), + n_train_windows=len(train_ds) * batch_size, + n_test_windows=sum(1 for _ in test_ds.unbatch())) + + # --------------------------------------------------------------------------- + # Full precision + # --------------------------------------------------------------------------- + train_uored_vafcls(model, train_ds, epochs, learning_rate, seed=seed) + float_scores = auroc_scores(*predict_keras_model(model, test_ds)) + row['float_auroc'] = f'{float_scores["macro"]:.6f}' + for name in LABEL_COLUMNS: + row[f'float_auroc_{name}'] = f'{float_scores[name]:.6f}' + + if models_dir is not None: + os.makedirs(models_dir, exist_ok=True) + model.save(os.path.join(models_dir, + f'akdcnn_uored_vafcls_fold{fold:03d}_' + f'seed{seed}.h5'), include_optimizer=False) + + if not skip_akida: + # ----------------------------------------------------------------------- + # Quantization, tuning and conversion + # ----------------------------------------------------------------------- + with set_akida_version(AkidaVersion.v1): + qmodel = quantize(model, input_weight_quantization=8, + weight_quantization=4, activ_quantization=4) + + train_uored_vafcls(qmodel, train_ds, qat_epochs, + qat_learning_rate, seed=seed) + qat_scores = auroc_scores(*predict_keras_model(qmodel, test_ds)) + row['qat_auroc'] = f'{qat_scores["macro"]:.6f}' + + ak_model = convert(qmodel) + + if models_dir is not None: + qmodel.save(os.path.join(models_dir, + f'akdcnn_uored_vafcls_fold{fold:03d}_' + f'seed{seed}_qat.h5'), + include_optimizer=False) + ak_model.save(os.path.join(models_dir, + f'akdcnn_uored_vafcls_fold{fold:03d}_' + f'seed{seed}_qat.fbz')) + + # ----------------------------------------------------------------------- + # Akida + # ----------------------------------------------------------------------- + akida_scores = auroc_scores(*evaluate_akida_model(ak_model, test_ds)) + row['akida_auroc'] = f'{akida_scores["macro"]:.6f}' + for name in LABEL_COLUMNS: + row[f'akida_auroc_{name}'] = f'{akida_scores[name]:.6f}' + + row['seconds'] = f'{time.time() - started:.1f}' + return row + + +def _existing_rows(out_path): + """Return the (fold, seed) pairs already present in the results file.""" + if not os.path.exists(out_path): + return set() + with open(out_path, newline='') as handle: + return {(int(r['fold']), int(r['seed'])) for r in csv.DictReader(handle) + if r.get('fold')} + + +def cross_validate(folds, seeds, out_path=DEFAULT_OUT, resume=True, **kwargs): + """Run every (fold, seed) combination, appending results as they complete. + + Args: + folds (iterable): fold indices to run. + seeds (iterable): seeds to run for each fold. + out_path (str): CSV to append results to. + resume (bool): skip (fold, seed) pairs already in out_path. When False, + the file is overwritten. + **kwargs: passed to run_fold. + + Returns: + list: the rows produced by this invocation. + """ + folds, seeds = list(folds), list(seeds) + out_path = str(out_path) + os.makedirs(os.path.dirname(out_path) or '.', exist_ok=True) + + if not resume and os.path.exists(out_path): + os.remove(out_path) + done = _existing_rows(out_path) if resume else set() + + todo = [(f, s) for f in folds for s in seeds if (f, s) not in done] + if done: + print(f'Resuming: {len(done)} of {len(folds) * len(seeds)} ' + f'(fold, seed) results already in {out_path}') + if not todo: + print('Nothing to do.') + return [] + + write_header = not os.path.exists(out_path) + rows = [] + progress = tqdm(todo, desc='Cross-validating', unit='fold') + with open(out_path, 'a', newline='') as handle: + writer = csv.DictWriter(handle, fieldnames=FIELDNAMES) + if write_header: + writer.writeheader() + for fold, seed in progress: + row = run_fold(fold, seed=seed, **kwargs) + writer.writerow(row) + # Flushed per fold so an interrupted sweep loses at most one fold. + handle.flush() + rows.append(row) + recent = [float(r['float_auroc']) for r in rows if r['float_auroc']] + progress.set_postfix(float_mean=f'{np.mean(recent):.4f}') + + return rows + + +def _read_results(out_path): + """Read the results CSV into a dict of column -> list.""" + with open(out_path, newline='') as handle: + return [row for row in csv.DictReader(handle) if row.get('fold')] + + +def _per_fold(rows, column): + """Seed-average a column, returning one value per fold. + + Averaging over seeds before averaging over folds is what the protocol does, + and it matters: it keeps each fold weighted equally regardless of how many + seeds happen to have been run for it. + """ + by_fold = {} + for row in rows: + if row.get(column): + by_fold.setdefault(int(row['fold']), []).append(float(row[column])) + return {fold: float(np.mean(values)) for fold, values in sorted(by_fold.items())} + + +def summarise(rows, quiet=False): + """Report the fold distribution for each stage and return the cv_* metrics. + Args: + rows (list): result rows, as read from the CSV. + quiet (bool): suppress printing. + + Returns: + dict: the cv_* metrics keys the README template reads. + """ + seeds = {int(row['seed']) for row in rows} + folds = {int(row['fold']) for row in rows} + metrics = {'cv_folds': str(len(folds)), 'cv_seeds': str(len(seeds))} + + for stage in ('float', 'qat', 'akida'): + per_fold = _per_fold(rows, f'{stage}_auroc') + if not per_fold: + continue + values = np.array(list(per_fold.values())) + mean, std = values.mean(), values.std(ddof=0) + sem = std / len(values) ** 0.5 + + metrics[f'cv_{stage}_auroc_mean'] = f'{mean:.4f}' + if stage == 'akida': + metrics['cv_akida_auroc_std'] = f'{std:.4f}' + metrics['cv_akida_auroc_min'] = f'{values.min():.4f}' + metrics['cv_akida_auroc_max'] = f'{values.max():.4f}' + + if not quiet: + print(f'\n{stage} macro AUROC over {len(values)} folds x ' + f'{len(seeds)} seed(s), seed-averaged per fold:') + print(f' mean {mean:.4f} std {std:.4f} sem {sem:.4f}') + print(f' min {values.min():.4f} median {np.median(values):.4f} ' + f'max {values.max():.4f}') + if len(seeds) > 1: + within = np.mean([np.std(v, ddof=0) for v in + _by_fold_values(rows, f'{stage}_auroc')]) + print(f' mean within-fold seed std {within:.4f} ' + f'<- the spread a single seed hides') + + return metrics + + +def _by_fold_values(rows, column): + """Group a column's raw values by fold, for within-fold seed statistics.""" + by_fold = {} + for row in rows: + if row.get(column): + by_fold.setdefault(int(row['fold']), []).append(float(row[column])) + return [v for v in by_fold.values() if len(v) > 1] + + +def plot_distribution(rows, fixed_fold, savepath): + """Histogram the per-fold AUROCs, marking the fixed fold and the mean.""" + import matplotlib + matplotlib.use('Agg') + import matplotlib.pyplot as plt + + per_fold = _per_fold(rows, 'akida_auroc') or _per_fold(rows, 'float_auroc') + values = np.array(list(per_fold.values())) + + fig, ax = plt.subplots(figsize=(8, 4.5)) + ax.hist(values, bins=20, color='#4C72B0', edgecolor='white') + ax.axvline(values.mean(), color='#C44E52', linewidth=2, + label=f'mean {values.mean():.4f}') + if fixed_fold in per_fold: + ax.axvline(per_fold[fixed_fold], color='#55A868', linestyle='--', + linewidth=2, + label=f'fold {fixed_fold} (pretrained) ' + f'{per_fold[fixed_fold]:.4f}') + ax.set_xlabel('macro AUROC on the held-out bearings') + ax.set_ylabel('folds') + ax.set_title(f'Per-fold performance over {len(values)} ' + f'bearing-disjoint folds') + ax.legend() + fig.tight_layout() + fig.savefig(savepath, dpi=150) + print(f'Distribution plot saved to {savepath}') + + +if __name__ == '__main__': + parser = argparse.ArgumentParser() + parser.add_argument('-d', '--data', default='./data/uored_vafcls', + help='Directory holding the prepared .npz cache') + parser.add_argument('--first-fold', type=int, default=EVAL_FOLDS.start, + help='First fold to run (inclusive)') + parser.add_argument('--last-fold', type=int, default=EVAL_FOLDS.stop - 1, + help='Last fold to run (inclusive)') + parser.add_argument('--seeds', type=int, default=1, + help='Number of seeds per fold; seeds are range(N)') + parser.add_argument('-b', '--batch_size', type=int, default=BATCH_SIZE) + parser.add_argument('-e', '--epochs', type=int, default=30) + parser.add_argument('-lr', '--learning_rate', type=float, default=2e-4) + parser.add_argument('--qat-epochs', type=int, default=10) + parser.add_argument('--qat-learning-rate', type=float, default=5e-5) + parser.add_argument('-o', '--out', default=str(DEFAULT_OUT), + help='CSV to append per-fold results to. Two machines ' + 'must not append to the same file') + parser.add_argument('--no-resume', action='store_true', + help='Overwrite the results file instead of resuming') + parser.add_argument('--skip-akida', action='store_true', + help='Float training only - no quantization, tuning or ' + 'Akida evaluation. About four times faster') + parser.add_argument('--keep-models', action='store_true', + help='Write each fold\'s models to models/cv/') + parser.add_argument('--summarise-only', action='store_true', + help='Summarise an existing results file without ' + 'running anything') + parser.add_argument('--save-metrics', action='store_true', + help='Write the cv_* keys and the distribution plot to ' + 'docs/. Requires a complete evaluation sweep') + parser.add_argument('--save-seed-std', action='store_true', + help='Write only cv_seed_std, the mean within-fold ' + 'spread across seeds. For the multi-seed subset ' + 'run, which deliberately covers few folds') + args = parser.parse_args() + + folds = range(args.first_fold, args.last_fold + 1) + + if not args.summarise_only: + cross_validate( + folds, range(args.seeds), out_path=args.out, + resume=not args.no_resume, data_path=args.data, + epochs=args.epochs, learning_rate=args.learning_rate, + qat_epochs=args.qat_epochs, + qat_learning_rate=args.qat_learning_rate, + batch_size=args.batch_size, skip_akida=args.skip_akida, + models_dir='./models/cv' if args.keep_models else None) + + rows = _read_results(args.out) + metrics = summarise(rows) + + if args.save_seed_std: + import json + + # Quoted in the README's cross-validation section as the measured + # single-seed uncertainty. It comes from the multi-seed subset run + # rather than the headline sweep, which is one seed per fold. + groups = _by_fold_values(rows, 'akida_auroc') \ + or _by_fold_values(rows, 'float_auroc') + if not groups: + raise SystemExit('--save-seed-std needs at least two seeds for at ' + 'least one fold; run with --seeds 3.') + seed_std = float(np.mean([np.std(v, ddof=0) for v in groups])) + + metrics_path = pathlib.Path(__file__).parent / 'docs' / 'metrics.json' + stored = json.loads(metrics_path.read_text()) if metrics_path.exists() else {} + stored['cv_seed_std'] = f'{seed_std:.4f}' + stored['cv_seed_std_folds'] = str(len(groups)) + metrics_path.write_text(json.dumps(stored, indent=4) + '\n') + print(f'\nMean within-fold seed std {seed_std:.4f} over {len(groups)} ' + f'folds, saved to {metrics_path}') + + if args.save_metrics: + import json + # The save-metrics argument used to update the stored metrics that are used to generate the + # performance tables in the README of this folder. + # This should only be used for code maintenance, when the model or training + # pipeline is updated and a new trained model integrated. + # + # The published aggregate must describe the whole evaluation range, or + # the README would quote a mean over whichever folds happened to run. + have = {int(row['fold']) for row in rows} + missing = set(EVAL_FOLDS) - have + if missing: + raise SystemExit( + f'--save-metrics needs every evaluation fold ' + f'({EVAL_FOLDS.start}-{EVAL_FOLDS.stop - 1}); ' + f'{len(missing)} are missing, e.g. {sorted(missing)[:5]}.') + if not any(row.get('akida_auroc') for row in rows): + raise SystemExit('--save-metrics needs the Akida column; this ' + 'looks like a --skip-akida run.') + + docs = pathlib.Path(__file__).parent / 'docs' + metrics_path = docs / 'metrics.json' + stored = json.loads(metrics_path.read_text()) if metrics_path.exists() else {} + stored.update(metrics) + metrics_path.write_text(json.dumps(stored, indent=4) + '\n') + print(f'\nMetrics saved to {metrics_path}') + + plot_distribution(rows, FIXED_FOLD, + docs / 'ref_cv_auroc_distribution.png') diff --git a/akida1/model_zoo/uored_vafcls/uored_vafcls_data.py b/akida1/model_zoo/uored_vafcls/uored_vafcls_data.py new file mode 100644 index 0000000..9546a3e --- /dev/null +++ b/akida1/model_zoo/uored_vafcls/uored_vafcls_data.py @@ -0,0 +1,723 @@ +#!/usr/bin/env python +# Copyright 2026 Brainchip Holdings Ltd. Apache 2.0 License +""" +UORED-VAFCLS bearing fault data pipeline. + +Turns 10 s raw accelerometer recordings into 1 s uint8 "images" that an Akida +convolutional model can consume directly: + + raw vibration (42 kHz) -> 1 s window (42000 samples) + -> random gain (training only) + -> rescale to uint8-range using a + single per-dataset scale factor + -> reshape to (300, 140, 1) + +BEARING LEVEL DATA SPLIT PROTOCOL + +The dataset split implemented here is derived from the method proposed by +Vieira et al (2026), "Towards a more realistic evaluation of machine learning +models for bearing fault diagnosis" (https://arxiv.org/abs/2509.22267) +The split is bearing-disjoint and drawn *before* any windowing to avoid the +data-leakage that compromises the majority of naive approaches to this dataset. + +Fault mode is determined by bearing id in this dataset (inner=1-5, outer=6-10, +ball=11-15, cage=16-20), so a model given any split that is not bearing-disjoint +can score near-perfectly by re-identifying the bearing without learning anything +about faults. Each bearing contributes three recordings - its own healthy one +plus two severities of a single fault mode - and holding a bearing out removes +all three. + +Folds are enumerated: every way of holding out 2 bearings from +each of the 4 fault modes gives C(5,2)^4 = 10,000 folds, shuffled once with a +fixed seed and indexed by fold number. Folds 0-4 are the tuning budget; folds +5-104 are the 100 evaluation folds. FIXED_FOLD (42) is the one the pretrained +models and the hardware benchmark use. + +SEGMENT LEVEL DATA SPLIT + +The segment split divides each recording by time - the first 60% of every +10 s recording trains, the last 40% tests - and keeps all 60 recordings on both +sides. Windows minutes apart from the same bearing, the same fault and the same +run therefore land on both sides of the train/test split. It is the most severe +form of leakage in Vieira et al's taxonomy, and the most commonly committed. + +CLASS LABELS + +Labels are potentially multi-label, [inner, outer, ball, cage], not one-of-N: +a bearing could in principal suffer from multiple faults (although that case +is not represented in any of the relevant datasets). Note that 'healthy' is +not labelled as a separate class, but is encoded as the all-zero vector. The +reported metric is macro AUROC (see uored_vafcls_eval.py). + +DATA PREPARATION + +The raw input vectors are reshaped to a 2D input. Note that's a plain reshape, +not a spectrogram. It happens here rather than in the model because an Akida graph +cannot contain a reshape - the deployed model receives the already-framed tensor. + + +Usage: + from uored_vafcls_data import get_data, get_test_data, get_samples + + train_ds, test_ds = get_data('./data/uored_vafcls', (300, 140, 1), + batch_size=120, fold=5, seed=0) + samples = get_samples('./data/uored_vafcls', (300, 140, 1), num_samples=100) + + # The naive comparator, for contrast only: + train_ds, test_ds = get_data('./data/uored_vafcls', (300, 140, 1), + split_mode='segment') + +Summarise either split without training anything: + python uored_vafcls_data.py --fold 42 + python uored_vafcls_data.py --split-mode segment + +Rebuild the cache from the raw Mendeley CSVs: + python uored_vafcls_data.py --prepare-raw /path/to/1_CSV_Raw_Data_Files +""" + +import functools +import glob +import os +import random +from collections import namedtuple +from itertools import combinations, product + +import numpy as np +import pandas as pd +import tensorflow as tf + +# --------------------------------------------------------------------------- +# Dataset definition +# --------------------------------------------------------------------------- +DEFAULT_DATA_PATH = './data/uored_vafcls' +CACHE_NAME = 'uored_vafcls_42khz_60rec.npz' + +# The prepared cache is mirrored so the example runs without a manual download. +# The raw dataset is distributed as per-recording CSVs from Mendeley, which is +# not fetchable as a single file; --prepare-raw rebuilds this .npz from them. +MIRROR_URL = ('https://data.brainchip.com/dataset-mirror/uored_vafcls/' + + CACHE_NAME) +MIRROR_HASH = ('sha256:4ba82c0c36a38fb2afc1428c46a84264093da8c6084d4e471db' + 'f00c9679c9c98') + +SAMPLE_RATE = 42_000 # Hz +SIGNAL_LENGTH = 420_000 # 10 s per recording +N_RECORDINGS = 60 # 20 bearings x (1 healthy + 2 fault severities) + +# Fault mode is a property of the bearing in this dataset. This mapping is what +# makes a non-bearing-disjoint split leak, and it is why the split enumerates +# bearings rather than recordings. +FAULT_BEARING_IDS = { + 'inner': (1, 2, 3, 4, 5), + 'outer': (6, 7, 8, 9, 10), + 'ball': (11, 12, 13, 14, 15), + 'cage': (16, 17, 18, 19, 20), +} +LABEL_COLUMNS = ('inner', 'outer', 'ball', 'cage') +NUM_LABELS = 4 + +# --------------------------------------------------------------------------- +# Split protocol +# --------------------------------------------------------------------------- +N_BEARINGS_PER_FAULT_MODE = 2 # 2 of 5 held out per mode -> 8 of 20 bearings +SPLIT_SEED = 42 # fixes the fold numbering; see the note below +TUNING_FOLDS = range(0, 5) # the only folds a hyperparameter may be chosen on +EVAL_FOLDS = range(5, 105) # the 100 folds the reported mean comes from +FIXED_FOLD = 42 # pretrained models + hardware benchmark + +# 'bearing' is the protocol. 'segment' is the naive comparator described in the +# module docstring - a time-wise split within each recording, which leaks badly +# by construction. Anything that publishes a number must reject 'segment'. +SPLIT_MODES = ('bearing', 'segment') +DEFAULT_SPLIT_MODE = 'bearing' + +# The segment split cuts each recording at 60% of its length. +SEGMENT_SPLIT_FRACTION = 0.6 +SEGMENT_BOUNDARY = int(SIGNAL_LENGTH * SEGMENT_SPLIT_FRACTION) # 252_000 + +# --------------------------------------------------------------------------- +# Windowing +# --------------------------------------------------------------------------- +WINDOW_LENGTH = 42_000 # 1.0 s +FRAME_WIDTH = 140 # samples per frame; 42000 / 140 = 300 frames +INPUT_SHAPE = (WINDOW_LENGTH // FRAME_WIDTH, FRAME_WIDTH, 1) # (300, 140, 1) + +TRAIN_WINDOWS_PER_RECORDING = 20 # 36 recordings x 20 = 720 windows per epoch +EVAL_WINDOWS_PER_RECORDING = 10 # 24 recordings x 10 = 240 windows, tiled +GAIN_STD = 0.7 # Random augmentation, a scalar gain applied to the signal, N(1, GAIN_STD) +BATCH_SIZE = 120 # exact divisor of 720 -> 6 steps per epoch + +# The segment split trains on all 60 recordings rather than 36, so it needs its +# own per-recording count to land on the same 720 windows per epoch: 60 x 12. +# Matching the budget is what makes the two modes comparable - at 20 windows per +# recording the naive split would get 1200 windows and 10 steps per epoch, and +# its higher score would confound leakage with 67% more gradient steps. The test +# side needs no such adjustment: the last 40% of a recording holds exactly 4 +# whole windows, and 60 x 4 = 240, the same as 24 held-out recordings x 10. +SEGMENT_TRAIN_WINDOWS_PER_RECORDING = 12 + +# --------------------------------------------------------------------------- +# uint8 input encoding +# --------------------------------------------------------------------------- +# Akida takes 8-bit inputs (uint8 range). Inspection of the UORED-VAFCLS data +# shows that amplitude varies widely across recordings (with, predictably, +# vibration increasing markedly when bearings are faulty). Rather than set the +# range according to the global min-max of the data (which would completely +# flatten the low amplitude vibrations of some recordings when reduced to 8-bit) +# we choose a range that preserves the majority of data points, while clipping +# the worst outliers to preserve resolution for low-amplitude signals. We +# chose a range of [-128, 127], which looks suitable for the data, and has the +# advantage of requiring minimal transformation of the input signal: no rescaling, +# a simple offset of +128 plus rounding and clipping to [0, 255]. +# +ZERO_POINT = 128 + +# --------------------------------------------------------------------------- +# Split protocol +# --------------------------------------------------------------------------- +def held_out_bearings_for_fold(fold, n_per_mode=N_BEARINGS_PER_FAULT_MODE, + seed=SPLIT_SEED): + """Return the sorted bearing ids held out for `fold`. + + Enumerates every way of choosing `n_per_mode` bearings from each of the four + fault modes (2 -> C(5,2)^4 = 10,000 folds), shuffles that list once with a + fixed seed, and indexes it by `fold`. + + The enumeration order and the shuffle are reproduced exactly from the + protocol's reference implementation, so fold N here holds out the same + bearings as fold N there. Do not "tidy" this function: any change to the + ordering silently renumbers all 10,000 folds and every published per-fold + number stops meaning what it says. + + Args: + fold (int): fold index, 0 to 9999. + n_per_mode (int): bearings held out per fault mode. + seed (int): seed for the one-off shuffle of the enumeration. + + Returns: + list: the held-out bearing ids, sorted ascending. + """ + per_mode = [list(combinations(ids, n_per_mode)) + for ids in FAULT_BEARING_IDS.values()] + all_combos = list(product(*per_mode)) + random.Random(seed).shuffle(all_combos) + + if not 0 <= fold < len(all_combos): + raise IndexError(f'fold {fold} out of range for {len(all_combos)} folds') + + selected = all_combos[fold] + return sorted({bearing for mode in selected for bearing in mode}) + + +def fold_rows(bearing_id, fold, n_per_mode=N_BEARINGS_PER_FAULT_MODE, + seed=SPLIT_SEED): + """Split recording indices into train and test rows for `fold`. + + Args: + bearing_id (np.ndarray): per-recording bearing id, shape (N_RECORDINGS,). + fold (int): fold index. + n_per_mode (int): bearings held out per fault mode. + seed (int): seed for the fold enumeration shuffle. + + Returns: + np.ndarray, np.ndarray: training row indices, test row indices. + """ + test_bearings = held_out_bearings_for_fold(fold, n_per_mode, seed) + is_test = np.isin(bearing_id, test_bearings) + return np.flatnonzero(~is_test), np.flatnonzero(is_test) + + +# --------------------------------------------------------------------------- +# The Segment-Level Split Comparator +# --------------------------------------------------------------------------- +# A time region is a half-open (start, stop) in samples. Every window drawn for +# a split comes from one of these, which is the whole of what separates the two +# modes: 'bearing' draws from the full recording and separates the +# rows instead, 'segment' shares every row and separates the time. +FULL_REGION = (0, SIGNAL_LENGTH) +SEGMENT_TRAIN_REGION = (0, SEGMENT_BOUNDARY) +SEGMENT_TEST_REGION = (SEGMENT_BOUNDARY, SIGNAL_LENGTH) + +SplitPlan = namedtuple('SplitPlan', ['mode', 'train_rows', 'test_rows', + 'train_region', 'test_region', + 'train_windows_per_recording']) + + +def check_split_mode(split_mode): + """Validate a split mode, returning it unchanged. + + Args: + split_mode (str): one of SPLIT_MODES. + + Returns: + str: `split_mode`. + """ + if split_mode not in SPLIT_MODES: + raise ValueError(f'split_mode must be one of {SPLIT_MODES}, got ' + f'{split_mode!r}') + return split_mode + + +def split_plan(bearing_id, fold, split_mode=DEFAULT_SPLIT_MODE): + """Resolve one split into rows, time regions and a training window budget. + + Args: + bearing_id (np.ndarray): per-recording bearing id, shape (N_RECORDINGS,). + fold (int): fold index. Ignored by 'segment', which has one split. + split_mode (str): one of SPLIT_MODES. + + Returns: + SplitPlan: rows and time regions for each side, plus the number of + training windows to draw per recording. + """ + check_split_mode(split_mode) + + if split_mode == 'segment': + rows = np.arange(len(bearing_id)) + return SplitPlan(split_mode, rows, rows, SEGMENT_TRAIN_REGION, + SEGMENT_TEST_REGION, + SEGMENT_TRAIN_WINDOWS_PER_RECORDING) + + train_rows, test_rows = fold_rows(bearing_id, fold) + return SplitPlan(split_mode, train_rows, test_rows, FULL_REGION, + FULL_REGION, TRAIN_WINDOWS_PER_RECORDING) + + +# --------------------------------------------------------------------------- +# Input encoding +# --------------------------------------------------------------------------- +def encode_uint8(window): + """Encode a float window as uint8 centred on ZERO_POINT. + Args: + window (np.ndarray): float32 samples, any shape. + + Returns: + np.ndarray: uint8, same shape as `window`. + """ + return np.clip(np.round(ZERO_POINT + window), 0, 255).astype(np.uint8) + + +def frame_window(x, width=FRAME_WIDTH): + """Frame the trailing sample axis into (frames, width, 1). + + Leaves any leading batch axes alone, so it works on one window or a stack. + + Args: + x (np.ndarray): shape (..., WINDOW_LENGTH). + width (int): samples per frame. + + Returns: + np.ndarray: shape (..., WINDOW_LENGTH // width, width, 1). + """ + if x.shape[-1] % width: + raise ValueError(f'window length {x.shape[-1]} is not divisible by the ' + f'frame width {width}') + return np.ascontiguousarray(x.reshape(*x.shape[:-1], -1, width))[..., None] + + +# --------------------------------------------------------------------------- +# Cache +# --------------------------------------------------------------------------- +def _cache_path(data_path): + return os.path.join(data_path, CACHE_NAME) + + +@functools.lru_cache(maxsize=1) +def _load_cache(data_path): + """Load the prepared recordings and derive the encoding constants. + + lru_cached because uored_vafcls_cross_validate.py calls get_data() a hundred + times in one process; without this it would re-read ~100 MB + + Args: + data_path (str): directory holding (or to receive) the .npz cache. + + Returns: + dict: signals, labels, bearing_id, fault_type, severity, rpm, + waveform_id + """ + path = _cache_path(data_path) + if not os.path.exists(path): + _download_cache(data_path) + + with np.load(path, allow_pickle=False) as data: + cache = {key: data[key] for key in data.files} + + signals = cache['signals'] + if signals.shape != (N_RECORDINGS, SIGNAL_LENGTH): + raise ValueError(f'{path}: expected signals of shape ' + f'{(N_RECORDINGS, SIGNAL_LENGTH)}, got {signals.shape}') + + return cache + + +def _download_cache(data_path): + """Fetch the prepared .npz from the BrainChip dataset mirror.""" + import pooch + + os.makedirs(data_path, exist_ok=True) + print(f'Downloading the prepared UORED-VAFCLS cache to {data_path} ...') + pooch.retrieve(url=MIRROR_URL, known_hash=MIRROR_HASH, path=data_path, + fname=CACHE_NAME) + + +def _check_input_shape(input_shape): + if tuple(input_shape) != INPUT_SHAPE: + raise ValueError( + f'This example is built around a fixed input shape of {INPUT_SHAPE} ' + f'(a {WINDOW_LENGTH}-sample window framed {FRAME_WIDTH} samples ' + f'wide), but got {tuple(input_shape)}.') + + +def _labels(cache, rows): + return cache['labels'][rows].astype(np.float32) + + +def _describe(name, cache, rows, region=FULL_REGION): + """Print what a set of recordings contains, by bearing and fault mode.""" + bearings = sorted(set(cache['bearing_id'][rows].tolist())) + counts = {} + for fault_type in cache['fault_type'][rows]: + counts[str(fault_type)] = counts.get(str(fault_type), 0) + 1 + breakdown = ', '.join(f'{k} {v}' for k, v in sorted(counts.items())) + span = '' + if tuple(region) != FULL_REGION: + span = (f', t = {region[0] / SAMPLE_RATE:.1f}-' + f'{region[1] / SAMPLE_RATE:.1f} s') + print(f'{name}: {len(rows)} recordings from {len(bearings)} bearings ' + f'{bearings} ({breakdown}{span})') + + +def _describe_split(cache, plan, fold): + """Print the split, and how much of it leaks.""" + if plan.mode == 'segment': + print(f'Naive segment split: every recording contributes both sides, ' + f'cut at {SEGMENT_SPLIT_FRACTION:.0%} ' + f'({SEGMENT_BOUNDARY / SAMPLE_RATE:.1f} s). There is only one ' + f'such split, so the fold index is ignored.') + else: + print(f'Fold {fold}: holding out bearings ' + f'{held_out_bearings_for_fold(fold)}') + + _describe('Train', cache, plan.train_rows, plan.train_region) + _describe('Held out', cache, plan.test_rows, plan.test_region) + + train_bearings = set(cache['bearing_id'][plan.train_rows].tolist()) + test_bearings = set(cache['bearing_id'][plan.test_rows].tolist()) + shared = train_bearings & test_bearings + verdict = 'LEAKS' if shared else 'leakage-free' + print(f'Train and held out share {len(shared)} of ' + f'{len(train_bearings | test_bearings)} bearings -> {verdict}') + + +# --------------------------------------------------------------------------- +# Window generation +# --------------------------------------------------------------------------- +def _train_dataset(cache, rows, batch_size, dtype, seed, region=FULL_REGION, + windows_per_recording=TRAIN_WINDOWS_PER_RECORDING): + """Random 1 s crops with a random gain, re-drawn every epoch. + + Crops start anywhere in `region` that leaves a whole window inside it. + """ + signals = cache['signals'] + labels = _labels(cache, rows) + n_items = len(rows) * windows_per_recording + region_start, region_stop = region + + # An independent generator, not the global Keras RNG. The shell pipeline + # re-seeds in a fresh process between building and training, so only an + # independent data RNG makes the in-process cross-validation path and the + # multi-process shell path draw the same crops for a given (fold, seed). + rng = np.random.default_rng(seed) + + def generator(): + for idx in range(n_items): + i = idx % len(rows) + row = rows[i] + start = rng.integers(region_start, region_stop - WINDOW_LENGTH) + window = np.array(signals[row, start:start + WINDOW_LENGTH], + dtype=np.float32) + # Random Augmentation: Gain is applied to the float waveform + window *= np.float32(rng.normal(1.0, GAIN_STD)) + window = encode_uint8(window) + yield frame_window(window), labels[i] + + signature = (tf.TensorSpec(shape=INPUT_SHAPE, dtype=tf.uint8), + tf.TensorSpec(shape=(NUM_LABELS,), dtype=tf.float32)) + return ( + tf.data.Dataset.from_generator(generator, output_signature=signature) + # Cardinality is declared so that len(train_ds) works, which is how the training script + # derives steps_per_epoch for the LR schedule. + .apply(tf.data.experimental.assert_cardinality(n_items)) + .shuffle(n_items, seed=seed, reshuffle_each_iteration=True) + .batch(batch_size, drop_remainder=True) + .map(lambda x, y: (tf.cast(x, dtype), y)) + .prefetch(tf.data.AUTOTUNE) + ) + + +def _eval_arrays(cache, rows, region=FULL_REGION): + """Deterministic contiguous tiling of `region`, as many whole windows as fit. + + That is EVAL_WINDOWS_PER_RECORDING (10) over a full recording, or 4 over the + segment split's test region and 6 over its training region. + + Small enough to materialise (240 x 300 x 140 uint8 is about 10 MB). + """ + signals = cache['signals'] + labels = _labels(cache, rows) + region_start, region_stop = region + n_windows = (region_stop - region_start) // WINDOW_LENGTH + windows, targets = [], [] + for i, row in enumerate(rows): + for j in range(n_windows): + start = region_start + j * WINDOW_LENGTH + window = np.array(signals[row, start:start + WINDOW_LENGTH], + dtype=np.float32) + windows.append(encode_uint8(window)) + targets.append(labels[i]) + return frame_window(np.stack(windows)), np.stack(targets) + + +def _eval_dataset(cache, rows, batch_size, dtype, region=FULL_REGION): + x, y = _eval_arrays(cache, rows, region) + return ( + tf.data.Dataset.from_tensor_slices((x, y)) + .batch(batch_size) + .map(lambda a, b: (tf.cast(a, dtype), b)) + .prefetch(tf.data.AUTOTUNE) + ) + + +# --------------------------------------------------------------------------- +# Public API +# --------------------------------------------------------------------------- +def get_data(data_path=DEFAULT_DATA_PATH, input_shape=INPUT_SHAPE, + batch_size=BATCH_SIZE, dtype=tf.uint8, fold=FIXED_FOLD, seed=0, + split_mode=DEFAULT_SPLIT_MODE): + """Load the training and held-out datasets for one fold. + + There is deliberately no validation split. With 36 training recordings, + carving one out would cost about a sixth of the data and would itself have to + be bearing-disjoint to mean anything; nothing in the pipeline consumes it + (the recipe is a fixed epoch count with no early stopping and no best-weight + restore); and the protocol reserves folds 0-4 for exactly the tuning a + validation set would otherwise serve. The second dataset returned is the + held-out fold - the test set - and no decision may be taken from it. + + Args: + data_path (str): directory holding the .npz cache. Downloaded if absent. + input_shape (tuple): model input shape; must be INPUT_SHAPE. + batch_size (int): the batch size. BATCH_SIZE divides the 720 training + windows exactly, giving 6 steps per epoch. + dtype (tf.dtypes.DType, optional): input data type. Defaults to tf.uint8. + fold (int): bearing-disjoint fold index. Folds 0-4 are the tuning + budget; 5-104 are the evaluation folds. Defaults to FIXED_FOLD. + Ignored when `split_mode` is 'segment', which has one split. + seed (int): seed for the crop positions, the gains and the shuffle + order. The split itself does not depend on it. + split_mode (str): 'bearing' for the protocol, 'segment' for the naive + comparator. See the module docstring. Defaults to 'bearing'. + + Returns: + tf.data.Dataset, tf.data.Dataset: training dataset, held-out dataset. + """ + _check_input_shape(input_shape) + cache = _load_cache(data_path) + + plan = split_plan(cache['bearing_id'], fold, split_mode) + _describe_split(cache, plan, fold) + + return (_train_dataset(cache, plan.train_rows, batch_size, dtype, seed, + plan.train_region, + plan.train_windows_per_recording), + _eval_dataset(cache, plan.test_rows, batch_size, dtype, + plan.test_region)) + + +def get_samples(data_path=DEFAULT_DATA_PATH, input_shape=INPUT_SHAPE, + num_samples=1024, fold=FIXED_FOLD, seed=0, + split_mode=DEFAULT_SPLIT_MODE): + """Load a block of training windows as a plain uint8 numpy array. + + Used for activation-sparsity measurement and hardware benchmarking, both of + which need uint8 numpy input rather than a dataset. The draw is over the + fold's *training* data: Akida timings depend on input activity, so the + windows have to be real, but to avoid data leakage at quantization, they must + not come from held-out data. Under 'segment' that means the training side of + the time cut, since every recording appears on both sides. + + The deterministic tiled windows of the training region come first - 360 of + them either way, 36 recordings x 10 under 'bearing' and 60 x 6 under + 'segment'; beyond that the remainder are seeded random crops. + + Args: + data_path (str): directory holding the .npz cache. + input_shape (tuple): model input shape; must be INPUT_SHAPE. + num_samples (int): number of windows to return. Defaults to 1024. + fold (int): bearing-disjoint fold index. Defaults to FIXED_FOLD. + Ignored when `split_mode` is 'segment'. + seed (int): seed for the draw order and for any random crops. Akida + timings and sparsity depend on input activity, so varying this + varies the measurement. + split_mode (str): 'bearing' or 'segment'. Defaults to 'bearing'. + + Returns: + np.ndarray: shape (num_samples,) + INPUT_SHAPE, dtype uint8. + """ + _check_input_shape(input_shape) + cache = _load_cache(data_path) + signals = cache['signals'] + + plan = split_plan(cache['bearing_id'], fold, split_mode) + train_rows = plan.train_rows + region_start, region_stop = plan.train_region + rng = np.random.default_rng(seed) + + tiled, _ = _eval_arrays(cache, train_rows, plan.train_region) + idx = rng.permutation(len(tiled))[:num_samples] + samples = tiled[idx] + + shortfall = num_samples - len(samples) + if shortfall > 0: + extra = np.empty((shortfall, WINDOW_LENGTH), dtype=np.uint8) + for k in range(shortfall): + row = train_rows[rng.integers(0, len(train_rows))] + start = rng.integers(region_start, region_stop - WINDOW_LENGTH) + window = np.array(signals[row, start:start + WINDOW_LENGTH], + dtype=np.float32) + extra[k] = encode_uint8(window) + samples = np.concatenate([samples, frame_window(extra)]) + + return samples.astype(np.uint8) + + +# --------------------------------------------------------------------------- +# Raw data preparation +# --------------------------------------------------------------------------- +FOLDER_FAULT_TYPE = { + '1_Healthy': 'Healthy', + '2_Inner_Race_Faults': 'Inner', + '3_Outer_Race_Faults': 'Outer', + '4_Ball_Faults': 'Ball', + '5_Cage_Faults': 'Cage', +} +FAULT_TYPE_TO_LABEL = {'Inner': 'inner', 'Outer': 'outer', + 'Ball': 'ball', 'Cage': 'cage'} + + +def prepare_raw(raw_dir, data_path=DEFAULT_DATA_PATH): + """Rebuild the .npz cache from the raw Mendeley CSV tree. + + Only needed to reproduce the mirrored cache from the original dataset, or + after changing what the cache stores. Expects the layout of + `1_CSV_Raw_Data_Files (.csv)` from https://data.mendeley.com/datasets/y2px5tg92h/5: + + /1_Healthy/H__0.csv + /2_Inner_Race_Faults/I__.csv + /3_Outer_Race_Faults/O__.csv + /4_Ball_Faults/B__.csv + /5_Cage_Faults/C__.csv + + Folder order and the sorted glob within each folder fix the row ordering, + which is load-bearing: the fold logic indexes recordings by row. + + Args: + raw_dir (str): directory holding the five fault-type subfolders. + data_path (str): directory to write the .npz cache into. + + Returns: + str: path to the written cache. + """ + + signals = np.zeros((N_RECORDINGS, SIGNAL_LENGTH), dtype=np.float32) + records = [] + + for folder, fault_type in FOLDER_FAULT_TYPE.items(): + directory = os.path.join(raw_dir, folder) + if not os.path.isdir(directory): + raise FileNotFoundError(f'missing raw folder {directory}') + + for path in sorted(glob.glob(os.path.join(directory, '*.csv'))): + stem = os.path.splitext(os.path.basename(path))[0] + _, bearing_id, severity = stem.split('_') + frame = pd.read_csv(path, usecols=['Accelerometer', 'Speed']) + vibration = frame['Accelerometer'].to_numpy(dtype=np.float32) + if vibration.size != SIGNAL_LENGTH: + raise ValueError(f'{os.path.basename(path)}: expected ' + f'{SIGNAL_LENGTH} samples, got {vibration.size}') + + row_idx = len(records) + signals[row_idx] = vibration + label = np.zeros(NUM_LABELS, dtype=np.float32) + if fault_type != 'Healthy': + label[LABEL_COLUMNS.index(FAULT_TYPE_TO_LABEL[fault_type])] = 1.0 + + records.append({ + 'waveform_id': f'{bearing_id}_{fault_type}_{severity}', + 'bearing_id': int(bearing_id), + 'fault_type': fault_type, + 'severity': int(severity), + # The tachometer channel is sparsely populated; the first row's + # value is what the protocol uses. Analysis only - never an input. + 'rpm': int(frame['Speed'].iloc[0]), + 'label': label, + }) + + if len(records) != N_RECORDINGS: + raise ValueError(f'expected {N_RECORDINGS} recordings, ' + f'found {len(records)}') + + os.makedirs(data_path, exist_ok=True) + path = _cache_path(data_path) + np.savez( + path, + signals=signals, + labels=np.stack([r['label'] for r in records]), + row_idx=np.arange(N_RECORDINGS, dtype=np.int32), + bearing_id=np.array([r['bearing_id'] for r in records], dtype=np.int32), + severity=np.array([r['severity'] for r in records], dtype=np.int32), + rpm=np.array([r['rpm'] for r in records], dtype=np.int32), + waveform_id=np.array([r['waveform_id'] for r in records]), + fault_type=np.array([r['fault_type'] for r in records]), + label_columns=np.array(LABEL_COLUMNS), + fs=np.array(SAMPLE_RATE, dtype=np.int32), + ) + return path + + +if __name__ == '__main__': + import argparse + + parser = argparse.ArgumentParser( + description='Build, check and report on the UORED-VAFCLS data pipeline') + parser.add_argument('-d', '--data', default=DEFAULT_DATA_PATH, + help='Directory holding the prepared .npz cache') + parser.add_argument('--fold', type=int, default=FIXED_FOLD, + help='Bearing-disjoint fold to summarise') + parser.add_argument('--seed', type=int, default=0, + help='Seed for the crop positions, gains and shuffle') + parser.add_argument('--split-mode', choices=SPLIT_MODES, + default=DEFAULT_SPLIT_MODE, + help="'bearing' is the leakage-free protocol; " + "'segment' is the naive time-wise split within " + 'each recording, for comparison only') + parser.add_argument('--prepare-raw', metavar='RAW_DIR', default=None, + help='Rebuild the .npz cache from the raw Mendeley CSV ' + 'tree instead of downloading it') + args = parser.parse_args() + + if args.prepare_raw is not None: + path = prepare_raw(args.prepare_raw, args.data) + print(f'Wrote {path} ({os.path.getsize(path) / 1e6:.0f} MB)') + _load_cache.cache_clear() + + train_ds, test_ds = get_data(args.data, fold=args.fold, seed=args.seed, + split_mode=args.split_mode) + samples = get_samples(args.data, num_samples=100, fold=args.fold, + seed=args.seed, split_mode=args.split_mode) + print(f'\nBatches: train {len(train_ds)}, held out {len(test_ds)}') + print(f'Windows: train {len(train_ds) * BATCH_SIZE}, ' + f'held out {sum(1 for _ in test_ds.unbatch())}') + print(f'Samples for benchmarking: {samples.shape} {samples.dtype} ' + f'range [{samples.min()}, {samples.max()}]') diff --git a/akida1/model_zoo/uored_vafcls/uored_vafcls_eval.py b/akida1/model_zoo/uored_vafcls/uored_vafcls_eval.py new file mode 100644 index 0000000..d197edf --- /dev/null +++ b/akida1/model_zoo/uored_vafcls/uored_vafcls_eval.py @@ -0,0 +1,252 @@ +#!/usr/bin/env python +# Copyright 2026 Brainchip Holdings Ltd. Apache 2.0 License +""" +UORED-VAFCLS evaluation for tf_keras or akida models. + +The reported metric is macro AUROC (Area Under the Receiver Operating Characteristic Curve) +over the four fault labels. AUROC is threshold free and measures how well the positive and negative +class can be separated by the model. +Note, healthy bearings are encoded here as the all-zero label vector rather than a fifth +class. + +Scores from a single fold are of limited use on their own - the spread across +folds is far wider than the difference between models worth comparing. Use +uored_vafcls_cross_validate.py for a number that means something. + +Example +------- + python uored_vafcls_eval.py -d ./data/uored_vafcls \\ + -l models/akdcnn_uored_vafcls.h5 +""" +import argparse +import json +import pathlib + +import akida +import numpy as np +import tensorflow as tf +from cnn2snn import load_quantized_model +from sklearn.metrics import roc_auc_score +from tqdm import tqdm + +from akida_models.sparsity import compute_sparsity +from brainchip_utils.hardware_utils import get_akida_device +from brainchip_utils.plot_utils import pretty_print_sparsity +from uored_vafcls_data import (DEFAULT_SPLIT_MODE, FIXED_FOLD, LABEL_COLUMNS, + BATCH_SIZE, SPLIT_MODES, get_samples, get_data) + +tf.config.experimental.enable_op_determinism() + +# Windows used to measure activation sparsity. Held under the 360 deterministic +# training windows a fold provides, so the measurement is repeatable. +NUM_SPARSITY_SAMPLES = 240 + + +# --------------------------------------------------------------------------- +# Metrics +# --------------------------------------------------------------------------- +def auroc_scores(logits, labels): + """Macro and per-label AUROC. + Args: + logits (np.ndarray): raw model outputs, shape (N, len(LABEL_COLUMNS)). + labels (np.ndarray): binary indicator labels, same shape. + + Returns: + dict: 'macro' plus one entry per label in LABEL_COLUMNS. + """ + per_label = roc_auc_score(labels, logits, average=None) + scores = {'macro': float(roc_auc_score(labels, logits, average='macro'))} + scores.update({name: float(v) for name, v in zip(LABEL_COLUMNS, per_label)}) + return scores + + +def report(scores, split_label): + """Print the macro and per-label AUROC.""" + print(f'\n{split_label} macro AUROC: {scores["macro"]:.4f}') + print('Per-label AUROC:') + for name in LABEL_COLUMNS: + print(f' {name:<6} {scores[name]:.4f}') + + +# --------------------------------------------------------------------------- +# Inference +# --------------------------------------------------------------------------- +def predict_keras_model(model, dataset): + """Run inference with a tf_keras model and return (logits, labels).""" + logits_all, labels_all = [], [] + for batch, label_batch in dataset: + logits_all.append(model.predict(batch, verbose=0)) + labels_all.append(label_batch.numpy()) + return np.concatenate(logits_all), np.concatenate(labels_all) + + +def evaluate_akida_model(akida_model, dataset): + """Run inference with an Akida model and return (logits, labels). + + Returns logits rather than predictions: AUROC is rank-based and multi-label, + so thresholding here would throw away exactly the information it needs. + """ + device = get_akida_device(target_version=akida_model.ip_version) + if device is not None: + akida_model.map(device, mode=akida.MapMode.Minimal) + print('Running inference on Akida hardware device') + akida_model.summary() + + labels_all = None + logits_all = None + + # Akida can't directly digest the tensorflow dataset, we need to + # manually iterate over the dataset to deliver inputs as numpy arrays + for batch, label_batch in tqdm(dataset, desc="Evaluating on Akida"): + if not isinstance(batch, np.ndarray): + batch = batch.numpy() + + # Inference on Akida + logits_batch = akida_model.predict(batch) + logits_batch = logits_batch.squeeze(axis=(1, 2)) # (B, 1, 1, C) -> (B, C) + + if labels_all is None: + labels_all = label_batch + logits_all = logits_batch + else: + labels_all = np.concatenate([labels_all, label_batch]) + logits_all = np.concatenate([logits_all, logits_batch]) + + return logits_all, np.asarray(labels_all) + + +if __name__ == '__main__': + parser = argparse.ArgumentParser() + parser.add_argument('-l', '--loadmodel', required=True, + help='Model to load (.h5 tf_keras or .fbz akida model)') + parser.add_argument('-d', '--data', default='./data/uored_vafcls', + help='Directory holding the prepared .npz cache') + parser.add_argument('-b', '--batch_size', type=int, default=BATCH_SIZE) + parser.add_argument('--fold', type=int, default=FIXED_FOLD, + help='Bearing-disjoint fold to evaluate on') + parser.add_argument('--seed', type=int, default=0, + help='Random seed for the sparsity sample draw') + parser.add_argument('--split-mode', choices=SPLIT_MODES, + default=DEFAULT_SPLIT_MODE, + help="'bearing' is the leakage-free protocol; " + "'segment' is the naive time-wise split within " + 'each recording, for comparison only') + parser.add_argument('--save-metrics', action='store_true', + help='Write AUROC (and param count for .h5) to ' + 'metrics.json. Only valid on the fixed fold of ' + 'the bearing-disjoint split') + args = parser.parse_args() + + # --------------------------------------------------------------------------- + # Model + # --------------------------------------------------------------------------- + if args.loadmodel.endswith('.h5'): + model = load_quantized_model(args.loadmodel) + isakida = False + imsize = model.input_shape[1:] + elif args.loadmodel.endswith('.fbz'): + model = akida.Model(args.loadmodel) + isakida = True + imsize = tuple(model.input_shape) + else: + raise ValueError(f'Unsupported model file: {args.loadmodel}') + + # --------------------------------------------------------------------------- + # Data loading + # --------------------------------------------------------------------------- + _, test_ds = get_data(args.data, imsize, batch_size=args.batch_size, + fold=args.fold, split_mode=args.split_mode) + + # --------------------------------------------------------------------------- + # Evaluation + # --------------------------------------------------------------------------- + if isakida: + logits, labels = evaluate_akida_model(model, test_ds) + else: + logits, labels = predict_keras_model(model, test_ds) + + scores = auroc_scores(logits, labels) + if args.split_mode == 'segment': + report(scores, 'Naive segment split (LEAKY, for comparison only)') + else: + report(scores, f'Fold {args.fold} held-out') + + # --------------------------------------------------------------------------- + # Activation sparsity + # --------------------------------------------------------------------------- + sparsity = None + if isakida: + samples = get_samples(args.data, imsize, + num_samples=NUM_SPARSITY_SAMPLES, + fold=args.fold, seed=args.seed, + split_mode=args.split_mode) + sparsity_dict = compute_sparsity(model, samples=samples) + pretty_print_sparsity(sparsity_dict) + sparsity = float(np.mean(list(sparsity_dict.values()))) + print(f'Mean activation sparsity over {NUM_SPARSITY_SAMPLES} windows: ' + f'{sparsity * 100:.2f}%') + + # --------------------------------------------------------------------------- + # Persist metrics + # --------------------------------------------------------------------------- + if args.save_metrics: + # The save-metrics argument is used to update the stored metrics that are used to generate the + # performance tables in the README of this folder. + # This should only be used for code maintenance, when the model or training + # pipeline is updated and a new trained model integrated. + # + # What this script may publish depends on the split, and the two cases + # are deliberately not symmetric. + # + # Under the naive split it writes the Model Card's top row outright: one + # train-and-evaluate run is the whole of that row, because the segment + # split is singular and its score is stable to about 0.001. + # + # Under the protocol it writes no AUROC at all. The bottom row is the + # 100-fold mean from uored_vafcls_cross_validate.py, and a single-fold + # number presented beside it is exactly the error this example exists to + # argue against. All that is taken from here is what a single model can + # honestly supply: the parameter count and the measured sparsity. + metrics_path = pathlib.Path(__file__).parent / 'docs' / 'metrics.json' + metrics = json.loads(metrics_path.read_text()) if metrics_path.exists() else {} + auroc_str = f'{scores["macro"]:.4f}' + written = [] + + if args.split_mode == 'segment': + if isakida: + metrics['segment_akida_auroc'] = auroc_str + metrics['segment_sparsity'] = f'{sparsity * 100:.2f}%' + written = ['segment_akida_auroc', 'segment_sparsity'] + elif 'qat' in pathlib.Path(args.loadmodel).stem: + metrics['segment_qat_auroc'] = auroc_str + written = ['segment_qat_auroc'] + else: + metrics['segment_float_auroc'] = auroc_str + metrics['segment_params'] = f'{model.count_params():,}' + written = ['segment_float_auroc', 'segment_params'] + else: + # Guarded on the fold: the hardware table and the sparsity figure + # are specifically the fixed fold, and a stray --fold would + # otherwise silently replace them with a different model's. The + # guard is meaningless under 'segment', which has only one split. + if args.fold != FIXED_FOLD: + raise SystemExit( + f'--save-metrics writes README figures measured on fold ' + f'{FIXED_FOLD}; refusing to write results from fold ' + f'{args.fold}.') + + metrics['fixed_fold'] = str(args.fold) + written = ['fixed_fold'] + if isakida: + metrics['sparsity'] = f'{sparsity * 100:.2f}%' + written.append('sparsity') + elif 'qat' not in pathlib.Path(args.loadmodel).stem: + metrics['params'] = f'{model.count_params():,}' + written.append('params') + + metrics_path.write_text(json.dumps(metrics, indent=4) + '\n') + print(f'Metrics saved to {metrics_path}: {", ".join(written)}') + if args.split_mode != 'segment' and written == ['fixed_fold']: + print('Note: nothing else to write. The README reports no ' + 'single-model AUROC for the bearing-disjoint split - that ' + 'row is the cross-validated mean.') diff --git a/akida1/model_zoo/uored_vafcls/uored_vafcls_model.py b/akida1/model_zoo/uored_vafcls/uored_vafcls_model.py new file mode 100644 index 0000000..65e2df4 --- /dev/null +++ b/akida1/model_zoo/uored_vafcls/uored_vafcls_model.py @@ -0,0 +1,157 @@ +#!/usr/bin/env python +# Copyright 2026 Brainchip Holdings Ltd. Apache 2.0 License +""" +UORED-VAFCLS bearing fault model (akdcnn). + +A 2D convolutional network over a framed 1 s vibration window, descended from +WDCNN but rebuilt to map wholly onto Akida 1 hardware. + +Input representation +-------------------- +The model takes the 42000-sample window already framed to (300, 140, 1) by the +data pipeline, not the raw waveform. The framing is outside the graph because an +Akida model cannot contain a reshape - the deployed network receives the framed +tensor directly. Row j is samples [140j, 140j+140), so the row axis is coarse time +(300 frames, 3.33 ms apart) and the column axis is fine phase within a frame, +much like the framing of a speech signal. The 7x7 stem therefore correlates 7 +consecutive samples across 7 consecutive frames: a lag/phase view. + +Input scaling is part of the model: a Rescaling layer normalizes the data from +the uint8 range furnished by the data preprocessing pipeline down to the roughly +[-1, 1] range the network trains on. Defining this Rescaling within the model +graph means float training, quantized training and Akida inference all see +identical values input values. The actual rescaling op itself is folded into the +weights of the first convolution layer by the cnn2snn package at conversion time. + +Hardware constraints that fixed the geometry +-------------------------------------------- +Some AKD1500 constraints drove the design: +* Most layers on Akida 1 only accept 4-bit inputs; only the specialised Input + Convolutional layer takes uint8 inputs, and we definitely want to use that + here. However, second input dimension on that layer has a hard limit (256 + on Akida 1). Thus the long input dimension has to go first, (300, 140) rather + than (140, 300) +* We found it helpful to use Global Average Pooling at the end of the + convolutional backbone. However, on Akida 1, that operation requires at least + 3 columns. This imposed constraints earlier in the model, limiting the + number of pooling steps, and forcing a relatively wide input. + +Example +------- + python uored_vafcls_model.py -s models/akdcnn_uored_vafcls_untrained.h5 +""" + +import argparse + +from tf_keras import Model +from tf_keras.layers import (BatchNormalization, Conv2D, Dense, Input, + MaxPooling2D, ReLU, Rescaling, GlobalAveragePooling2D, Flatten) +from tf_keras.utils import set_random_seed + +import akida +from cnn2snn import quantize, convert + +from uored_vafcls_data import INPUT_SHAPE + +def build_uored_vafcls_model(input_shape=INPUT_SHAPE, fault_classes=4, seed=0): + """Build the untrained akdcnn model. + + Args: + input_shape (tuple): input shape (H, W, C) + fault_classes (int): number of different fault classes + seed (int): random seed for weight initialization. + + Returns: + tf_keras.Model: the model, expecting uint8 inputs of shape INPUT_SHAPE + and returning fault_classes raw logits (one per fault mode; a healthy + bearing is all four logits low, not a fifth class). + """ + set_random_seed(seed) + + # Model definition parameters + STEM_FILTERS = 16 + STEM_KERNEL_SIZE = 7 + + BLOCK_FILTERS = [32, 64, 64, 64, 96, 128] + BLOCK_POOLING = [True, True, True, True, True, False] + + DENSE_UNITS = 512 + + # BatchNorm settings + # Important in this case - unusually, we find that the tf_keras + # default values (0.99, 0.001) cause problems when the model is quantized + BN_MOMENTUM = 0.9 + BN_EPSILON = 1e-5 + + def _conv_block(x, filters, name, kernel=(3, 3), padding='same', pool=True, + pool_padding='same'): + """Re-usable block with + Conv -> BatchNorm -> ReLU6 [-> MaxPool] + """ + x = Conv2D(filters, kernel, padding=padding, use_bias=False, + name=f'{name}_conv')(x) + x = BatchNormalization(momentum=BN_MOMENTUM, epsilon=BN_EPSILON, + name=f'{name}_bn')(x) + x = ReLU(max_value=6.0, name=f'{name}_relu')(x) + if pool: + x = MaxPooling2D((2, 2), strides=(2, 2), padding=pool_padding, + name=f'{name}_pool')(x) + return x + + inputs = Input(shape=input_shape, name='input') + + # Normalize incoming data from the uint8 range to [-1, 1] + # Folded into stem_conv weights at conversion + x = Rescaling(1.0 / 128.0, -1.0, name='rescaling')(inputs) + + # Stem: a dense convolution with no padding + x = _conv_block(x, STEM_FILTERS, name='stem', kernel=STEM_KERNEL_SIZE, padding='valid', pool=True, pool_padding='valid') + + # Blocks + for i, (filters, pool) in enumerate(zip(BLOCK_FILTERS[:-1], BLOCK_POOLING[:-1]), + start=1): + x = _conv_block(x, filters, f'block{i}', pool=pool) + + # Final block with Global Average Pooling + # Note that for Akida 1, it needs to be placed before the neighbouring ReLU + i = len(BLOCK_FILTERS) + x = Conv2D(BLOCK_FILTERS[-1], 3, padding='same', use_bias=False, + name=f'block{i}_conv')(x) + x = BatchNormalization(momentum=BN_MOMENTUM, epsilon=BN_EPSILON, + name=f'block{i}_bn')(x) + x = GlobalAveragePooling2D(name='gap')(x) # Akida 1: GAP must be placed before the ReLU + x = ReLU(max_value=6.0, name=f'block{i}_relu')(x) + + # Dense Classifier Ending + x = Dense(DENSE_UNITS, name='fc')(x) + x = BatchNormalization(momentum=BN_MOMENTUM, epsilon=BN_EPSILON, + name='fc_bn')(x) + x = ReLU(max_value=6.0, name='fc_relu')(x) + outputs = Dense(fault_classes, name='predictions')(x) + + model = Model(inputs, outputs, name='akdcnn_uored_vafcls') + return model + + +if __name__ == '__main__': + parser = argparse.ArgumentParser() + parser.add_argument('-s', '--savepath', + default='./models/akdcnn_uored_vafcls_untrained.h5', + help='Model save path') + parser.add_argument('--seed', type=int, default=0, + help='Random seed for reproducibility') + args = parser.parse_args() + + model = build_uored_vafcls_model(seed=args.seed) + model.summary() + model.save(args.savepath, include_optimizer=False) + print(f'Model saved to {args.savepath}') + + # Check that the model is hardware-compatible + # This is a very useful check before doing any training if working on model re-design + qmodel = quantize(model, weight_quantization=4, activ_quantization=4, input_weight_quantization=8) + ak_model = convert(qmodel) + device = akida.AKD1500() + + ak_model.map(device, mode=akida.MapMode.Minimal, hw_only=True) + ak_model.summary() diff --git a/akida1/model_zoo/uored_vafcls/uored_vafcls_naive_split.sh b/akida1/model_zoo/uored_vafcls/uored_vafcls_naive_split.sh new file mode 100644 index 0000000..5b66251 --- /dev/null +++ b/akida1/model_zoo/uored_vafcls/uored_vafcls_naive_split.sh @@ -0,0 +1,67 @@ +#!/usr/bin/env bash +# Copyright 2026 Brainchip Holdings Ltd. Apache 2.0 License +# +# The naive comparator: the same pipeline as uored_vafcls_train.sh, run under a +# deliberately leaky split. Build, train, quantize, tune, convert, and evaluate +# This builds the same pipeline as uored_vafcls_train.sh (Build, train, quantize, tune, convert, and evaluate +), run under a +# deliberately leaky data split (see uored_vafcls_data.py docstring for more info). +# +# Usage: bash uored_vafcls_naive_split.sh [DATADIR] [SEED] +# +# THE NUMBER THIS PRINTS IS NOT A RESULT. It is inflated by construction and +# exists only to be compared against uored_vafcls_train.sh, which runs the same +# model and the same recipe under the bearing-disjoint protocol. +# +# This naive split approach divides each recording by time: the first 60% is used for training and the +# last 40% is used for testing. This means that all 20 bearings appear in both train and test, leading to data leakage. +# Both splits draw the same total number of training and test windows, +# so the difference in score is attributable to the leakage and nothing else. +# See the DATA SPLIT PROTOCOL notes in uored_vafcls_data.py. +# +# There is no FOLD argument: unlike the protocol, this split is singular, so +# there is nothing to cross-validate. Its score is stable to about 0.001 where +# the bearing-disjoint folds span 0.04-0.05. Re-run with a different SEED to +# confirm that. +# +# Models are written with a _segment suffix so this never overwrites the +# protocol-trained checkpoints that the README tables describe. + +DATADIR="${1:-}" +DATA_ARG=${DATADIR:+-d "$DATADIR"} +SEED="${2:-${SEED:-0}}" +COMMON="--split-mode segment --seed $SEED $DATA_ARG" + +echo "UORED-VAFCLS NAIVE pipeline: seed ${SEED}, segment split (leaky by design)" + +# 1 - Build the untrained model +python uored_vafcls_model.py -s models/akdcnn_uored_vafcls_segment_untrained.h5 \ + --seed "$SEED" + +# 2 - Full-precision training +python uored_vafcls_train.py -l models/akdcnn_uored_vafcls_segment_untrained.h5 \ + -s models/akdcnn_uored_vafcls_segment.h5 -e 30 -lr 2e-4 -b 120 $COMMON + +# 3 - Evaluate the float model +python uored_vafcls_eval.py -l models/akdcnn_uored_vafcls_segment.h5 $COMMON + +# 4 - Post-training quantization: 4-bit weights and activations, 8-bit input +cnn2snn quantize -m models/akdcnn_uored_vafcls_segment.h5 -i 8 -w 4 -a 4 + +# 5 - Quantization-aware tuning +python uored_vafcls_train.py \ + -l models/akdcnn_uored_vafcls_segment_iq8_wq4_aq4.h5 \ + -s models/akdcnn_uored_vafcls_segment_qat.h5 -e 10 -lr 1e-5 -b 120 $COMMON + +# 6 - Evaluate the tuned quantized model +python uored_vafcls_eval.py -l models/akdcnn_uored_vafcls_segment_qat.h5 $COMMON + +# 7 - Convert to Akida +cnn2snn convert -m models/akdcnn_uored_vafcls_segment_qat.h5 + +# 8 - Evaluate on Akida +python uored_vafcls_eval.py -l models/akdcnn_uored_vafcls_segment_qat.fbz $COMMON + +# No hardware benchmark: latency and sparsity are properties of the graph and +# the input statistics, not of how the data was split. For those numbers run +# uored_vafcls_benchmark.py against the protocol-trained model. diff --git a/akida1/model_zoo/uored_vafcls/uored_vafcls_train.py b/akida1/model_zoo/uored_vafcls/uored_vafcls_train.py new file mode 100644 index 0000000..df9f5e5 --- /dev/null +++ b/akida1/model_zoo/uored_vafcls/uored_vafcls_train.py @@ -0,0 +1,130 @@ +#!/usr/bin/env python +# Copyright 2026 Brainchip Holdings Ltd. Apache 2.0 License +""" +UORED-VAFCLS bearing fault training. + +The loss is binary cross-entropy over four independent fault labels, as a machine can in principle show more than one fault (although it is not the case with the present dataset). Thus +a healthy bearing is the all-zero label vector rather than a fifth class. The +model emits raw logits, so `from_logits=True` throughout. + +There is no validation set, and consequently no early stopping, no +learning-rate-on-plateau and no best-weight restore. This is deliberate. The +only data not used for training is the held-out fold, and selecting weights on +that would be exactly the leakage the bearing-disjoint split exists to prevent. +The recipe is instead fixed - a set epoch count, evaluated at the final epoch - +and folds 0-4 are reserved for tuning that pipeline. Do not "fix" +this by adding a callback that monitors the held-out fold. + +Example +------- + python uored_vafcls_train.py -l models/akdcnn_uored_vafcls_untrained.h5 \\ + -s models/akdcnn_uored_vafcls.h5 -e 30 -lr 2e-4 --fold 5 +""" + +import argparse + +import tensorflow as tf +from cnn2snn import load_quantized_model +from tf_keras import regularizers +from tf_keras.layers import ReLU +from tf_keras.losses import BinaryCrossentropy +from tf_keras.metrics import AUC +from tf_keras.optimizers.legacy import Adam +from tf_keras.optimizers.schedules import CosineDecay +from tf_keras.utils import set_random_seed + +from uored_vafcls_data import (DEFAULT_SPLIT_MODE, FIXED_FOLD, SPLIT_MODES, + get_data) + +# Must be called before any TF ops to make GPU ops deterministic. +tf.config.experimental.enable_op_determinism() + + +def train_uored_vafcls(model, train_ds, epochs, learning_rate, + regularization=None, seed=0): + """Train or quantization-aware tune the model on one fold. + + Args: + model (tf_keras.Model): the model to train, float or quantized. + train_ds (tf.data.Dataset): training dataset. + epochs (int): number of epochs. + learning_rate (float): peak learning rate for the cosine schedule. + regularization (float, optional): activity regularization on the ReLU + layers, to increase activation sparsity. + seed (int): random seed. + + Returns: + tf_keras.callbacks.History: the fit history. + """ + set_random_seed(seed) + + # Optionaly add activity regularization for additional sparsity + if regularization is not None: + print('Adding Activity Regularization to ReLU layers') + regularizer = regularizers.L1L2(regularization, regularization) + for layer in model.layers: + if isinstance(layer, ReLU): + layer.activity_regularizer = regularizer + + # Learning Rate Scheduler + steps_per_epoch = len(train_ds) + total_steps = steps_per_epoch*epochs + warmup_fraction = 0.05 + warmup_steps = max(1, int(warmup_fraction * total_steps)) + lr_schedule = CosineDecay(initial_learning_rate=learning_rate * 0.01, + warmup_target=learning_rate, + warmup_steps=warmup_steps, + decay_steps=max(1, total_steps - warmup_steps)) + + num_labels = model.output_shape[-1] + model.compile(optimizer=Adam(learning_rate=lr_schedule), + loss=BinaryCrossentropy(from_logits=True), + metrics=[AUC(name='auroc', multi_label=True, + num_labels=num_labels)]) + + # Training + return model.fit(train_ds, epochs=epochs) + + +if __name__ == '__main__': + parser = argparse.ArgumentParser() + parser.add_argument('-l', '--loadmodel', required=True, + help='Model to load (.h5 tf_keras or quantized model)') + parser.add_argument('-s', '--savemodel', required=True, + help='Model save path') + parser.add_argument('-d', '--data', default='./data/uored_vafcls', + help='Directory holding the prepared .npz cache') + parser.add_argument('-b', '--batch_size', type=int, default=120, + help='Batch size') + parser.add_argument('-e', '--epochs', type=int, default=30) + parser.add_argument('-lr', '--learning_rate', type=float, default=2e-4, + help='Peak learning rate for the cosine schedule') + parser.add_argument('-reg', '--regularization', type=float, default=None, + help='Activity Regularization to increase sparsity') + parser.add_argument('--fold', type=int, default=FIXED_FOLD, + help='Bearing-disjoint fold index.') + parser.add_argument('--seed', type=int, default=0, + help='Random seed for reproducibility') + parser.add_argument('--split-mode', choices=SPLIT_MODES, + default=DEFAULT_SPLIT_MODE, + help="'bearing' is the leakage-free protocol; " + "'segment' is the naive time-wise split within " + 'each recording, for comparison only') + args = parser.parse_args() + + # Model + model = load_quantized_model(args.loadmodel) + + # Dataset + train_ds, test_ds = get_data(args.data, model.input_shape[1:], + args.batch_size, fold=args.fold, + seed=args.seed, split_mode=args.split_mode) + + # Run Training + train_uored_vafcls(model=model, train_ds=train_ds, + epochs=args.epochs, learning_rate=args.learning_rate, + regularization=args.regularization, seed=args.seed) + + # Save Trained Model + model.save(args.savemodel, include_optimizer=False) + print(f'Model saved as {args.savemodel}.') diff --git a/akida1/model_zoo/uored_vafcls/uored_vafcls_train.sh b/akida1/model_zoo/uored_vafcls/uored_vafcls_train.sh new file mode 100644 index 0000000..0153ee4 --- /dev/null +++ b/akida1/model_zoo/uored_vafcls/uored_vafcls_train.sh @@ -0,0 +1,60 @@ +#!/usr/bin/env bash +# Copyright 2026 Brainchip Holdings Ltd. Apache 2.0 License +# +# Full UORED-VAFCLS pipeline on ONE bearing-disjoint fold: build, train, +# quantize, tune, convert, evaluate at each stage, and benchmark on hardware. +# +# Usage: bash uored_vafcls_train.sh [DATADIR] [FOLD] [SEED] +# +# FOLD defaults to 42 +# This is used as a reference fold, used for the pretrained models +# and the published hardware numbers. +# This fold was picked because performance is very close to the 100-fold +# cross-validation mean, thereby avoiding giving a false impression of +# model accuracy for readers glancing at e.g. the training notebook. +# +# A single fold tells you very little about this model. The across-fold spread +# is 0.04-0.05 macro AUROC and folds range from about 0.70 to about 0.99 with +# everything else held fixed. For the number that means something, run: +# +# python uored_vafcls_cross_validate.py +# +# Folds 0-4 are the tuning folds. Any change to the recipe must be evaluated +# there, never on folds 5-104. + +DATADIR="${1:-}" +DATA_ARG=${DATADIR:+-d "$DATADIR"} +FOLD="${2:-${FOLD:-42}}" +SEED="${3:-${SEED:-0}}" +COMMON="--fold $FOLD --seed $SEED $DATA_ARG" + +echo "UORED-VAFCLS pipeline: fold ${FOLD}, seed ${SEED}" + +# 1 - Build the untrained model +python uored_vafcls_model.py -s models/akdcnn_uored_vafcls_untrained.h5 --seed "$SEED" + +# 2 - Full-precision training +python uored_vafcls_train.py -l models/akdcnn_uored_vafcls_untrained.h5 \ + -s models/akdcnn_uored_vafcls.h5 -e 30 -lr 2e-4 -b 120 $COMMON + +# 3 - Evaluate the float model +python uored_vafcls_eval.py -l models/akdcnn_uored_vafcls.h5 $COMMON + +# 4 - Post-training quantization: 4-bit weights and activations, 8-bit input +cnn2snn quantize -m models/akdcnn_uored_vafcls.h5 -i 8 -w 4 -a 4 + +# 5 - Quantization-aware tuning. +python uored_vafcls_train.py -l models/akdcnn_uored_vafcls_iq8_wq4_aq4.h5 \ + -s models/akdcnn_uored_vafcls_qat.h5 -e 10 -lr 5e-5 -b 120 $COMMON + +# 6 - Evaluate the tuned quantized model +python uored_vafcls_eval.py -l models/akdcnn_uored_vafcls_qat.h5 $COMMON + +# 7 - Convert to Akida +cnn2snn convert -m models/akdcnn_uored_vafcls_qat.h5 + +# 8 - Evaluate on Akida +python uored_vafcls_eval.py -l models/akdcnn_uored_vafcls_qat.fbz $COMMON + +# 9 - Hardware benchmark (needs a physical AKD1500) +python uored_vafcls_benchmark.py -l models/akdcnn_uored_vafcls_qat.fbz $COMMON diff --git a/akida1/model_zoo/uored_vafcls/update_readme.py b/akida1/model_zoo/uored_vafcls/update_readme.py new file mode 100644 index 0000000..82e42d9 --- /dev/null +++ b/akida1/model_zoo/uored_vafcls/update_readme.py @@ -0,0 +1,11 @@ +#!/usr/bin/env python +# Copyright 2025 Brainchip Holdings Ltd. Apache 2.0 License +"""Regenerate README.md from README.md.template + metrics.json.""" +import json +import pathlib + +here = pathlib.Path(__file__).parent +metrics = json.loads((here / "docs" / "metrics.json").read_text()) +template = (here / "docs" / "README.md.template").read_text() +(here / "README.md").write_text(template.format_map(metrics)) +print("README.md updated.")