Data Mining II — Air Quality Time Series

Data Mining 2 coursework project (A.A. 2018/2019), MSc Data Science and Business Informatics, University of Pisa.

Goal: one multivariate air-quality time series, attacked from five angles — clustering, sequential patterns, discretisation, classification and outlier detection.

Dataset

Air quality sensor readings with attributes Date, Time, CO(GT), PT08.S1(CO), NMHC(GT), C6H6(GT), PT08.S2(NMHC), NOx(GT), PT08.S3(NOx), NO2(GT), PT08.S4(NO2), PT08.S5(O3), T, RH, AH. Missing values are encoded as -200.

The working series is PT08.S1(CO) split into daily 24-point slices; days with too many gaps were dropped and remaining gaps interpolated from the adjacent hours of the same day and the same hour of adjacent days — 32 days needed repairing, all ending at 24 valid values.

What's here

File	Contents
dm_2_project.ipynb	Main pipeline: preprocessing, clustering, sequential patterns, discretisation, classification
dm_2_project_addition.ipynb	Contiguous sequential patterns + outlier detection
dm2_report_Aliyev_Gaydamaka_Parshina.pdf	Submitted report

Method

• Clustering over the cleaned daily series
• Sequential pattern mining on the discretised sequences
• Discretisation of the continuous sensors into interpretable bins
• Contiguous sequential patterns — patterns constrained to consecutive days
• Classification — k-NN vs Decision Tree, compared on accuracy and precision / recall / F1
• Outlier detection on the residual series

Result

k-NN with k = 3 gave the strongest F1 of the tested classifiers; the report argues F1 rather than accuracy is the right headline metric here, because the classes are uneven.

Reproducing

pip install pandas numpy matplotlib scikit-learn
jupyter notebook dm_2_project.ipynb
jupyter notebook dm_2_project_addition.ipynb

Team

Anna Gaydamaka · Kseniia Parshina · Gurban Aliyev — University of Pisa, A.A. 2018/2019.