A simple lab covering important Machine Learning Model and Inference Scores.
AI helped generate this lab in prep for the GCP PMLE exam.
Train small models on randomized CSV datasets, then verify the top general ML scoring techniques as unit tests: every score is hand-computed from first principles in scores.py and asserted against scikit-learn. Each test prints the metric, its formula, and both values before asserting they match.
| Group | Scores | Tests |
|---|---|---|
| Classification (binary) | Accuracy, Precision, Recall, Specificity, FPR, FNR, NPV, F1, F2, Balanced Accuracy, MCC, Cohen's κ, ROC AUC, PR AUC, Log Loss | test_classification.py |
| Classification (multi-class) | Per-class TP/FP/FN/TN one-vs-rest; Precision / Recall / F1 micro, macro, weighted | test_classification_averages.py |
| Regression | MAE, MSE, RMSE, MAPE, SMAPE, R², Adjusted R², RMSLE, Explained Variance, Max / Median Error | test_regression.py |
| Ranking | Precision@k, Recall@k (R1 = Recall@1), Hit@k, MRR, MAP, NDCG@k | test_ranking.py |
Stages run sequentially: generate data → train classifiers → train regressor → tests (pytest, runs automatically in docker compose up). Data is written to lab/data, models to lab/models.
Every classification score is built from four counts. Positive = the class you are trying to detect (label 1); for multi-class problems each class gets its own one-vs-rest counts:
| Predicted positive | Predicted negative | |
|---|---|---|
| Actually positive | TP — true positive | FN — false negative (miss) |
| Actually negative | FP — false positive (false alarm) | TN — true negative |
| Score | Also called | Calculation | What it tells you |
|---|---|---|---|
| Accuracy | (TP+TN) / (TP+FP+FN+TN) |
Overall fraction correct. Misleading under class imbalance. | |
| Precision | Positive predictive value | TP / (TP+FP) |
Of everything flagged positive, how much really is. Watch when FPs are costly. |
| Recall | Sensitivity, TPR, hit rate | TP / (TP+FN) |
Of all real positives, how many were caught. Watch when FNs are costly. |
| Specificity | TNR | TN / (TN+FP) |
Of all real negatives, how many were correctly rejected. |
| FPR | Fall-out | FP / (FP+TN) |
= 1 − Specificity. The x-axis of the ROC curve. |
| FNR | Miss rate | FN / (FN+TP) |
= 1 − Recall. |
| NPV | Negative predictive value | TN / (TN+FN) |
Of everything flagged negative, how much really is. |
| F1 | F-score (β=1) | 2TP / (2TP+FP+FN) |
Harmonic mean of Precision and Recall; punishes a lopsided trade-off. |
| Fβ | F2 shown | (1+β²)·TP / ((1+β²)·TP + β²·FN + FP) |
F1 generalized: β>1 weighs Recall over Precision. |
| Balanced accuracy | (TPR+TNR) / 2 = ½·(TP/(TP+FN) + TN/(TN+FP)) |
Accuracy averaged per class; robust to imbalance. | |
| MCC | Matthews correlation | (TP·TN − FP·FN) / √((TP+FP)(TP+FN)(TN+FP)(TN+FN)) |
Correlation in [−1, 1] using all four counts; chance = 0. |
| Cohen's κ | Kappa | (po − pe) / (1 − pe) |
Accuracy corrected for the chance agreement pe implied by the marginals. |
| ROC AUC | AUC | Area under the TPR-vs-FPR curve = P(a random positive is scored above a random negative) |
Threshold-free ranking quality. |
| PR AUC | Average Precision | Mean of Precision at each relevant rank |
More informative than ROC AUC when positives are rare. |
| Log loss | Cross-entropy | −mean( y·log p + (1−y)·log(1−p) ) |
Scores calibrated probabilities, not just the final class. |
With C classes, class c gets one-vs-rest counts TP_c / FP_c / FN_c / TN_c and per-class scores P_c, R_c, F1_c. The three averagings differ only in how the classes are combined:
| Average | Calculation | What it tells you |
|---|---|---|
| Micro | Pool the counts first: F1_micro = 2·ΣTP_c / (2·ΣTP_c + ΣFP_c + ΣFN_c) |
Dominated by the big classes. For single-label multi-class, micro P = micro R = micro F1 = Accuracy. |
| Macro | Unweighted mean: F1_macro = (1/C)·Σ F1_c |
Every class counts equally — exposes rare classes doing badly. |
| Weighted | Support-weighted mean: F1_weighted = Σ (support_c · F1_c) / Σ support_c |
Between the two; accounts for class frequency. |
Residual e = y − ŷ over n rows with p features.
| Score | Calculation | What it tells you |
|---|---|---|
| MAE | mean |e| |
Average error in target units. Robust to outliers. |
| MSE | mean(e²) |
Squared error; large errors dominate. |
| RMSE | √mean(e²) |
MSE back in target units; still outlier-sensitive. |
| MAPE | mean |e / y| (×100 for %) |
Error as a fraction of the actual. Undefined at y=0; skewed by small y. |
| SMAPE | mean( 2|e| / (|y| + |ŷ|) ) |
Symmetric MAPE, bounded in [0, 2]. |
| R² | 1 − Σe² / Σ(y − ȳ)² |
Fraction of variance explained: 1 is perfect, 0 is "predict the mean", <0 is worse. |
| Adjusted R² | 1 − (1−R²)(n−1)/(n−p−1) |
R² penalized for extra features; use when comparing models with different p. |
| RMSLE | √mean( (log(1+y) − log(1+ŷ))² ) |
Log-scale RMSE; penalizes under-prediction more; needs y ≥ 0. |
| Explained variance | 1 − Var(e) / Var(y) |
Like R² but not centered; equals R² when the mean residual is 0. |
| Max error | max |e| |
Worst-case single prediction. |
| Median AE | median |e| |
Typical error; very robust to outliers. |
Computed per query on items sorted by descending model score, then averaged. Graded relevance 0–3; an item counts as relevant when grade ≥ 2.
| Score | Calculation | What it tells you |
|---|---|---|
| Precision@k | (# relevant in top k) / k |
Quality of the first k results. |
| Recall@k | (# relevant in top k) / (total relevant) |
Coverage of all relevant items in the first k. |
| R1 | Recall@1 |
Recall at the very first position — the "R1" of retrieval/ranking. |
| Hit@k | 1 if any relevant in top k, else 0 |
Did the user get at least one good result? |
| MRR | mean(1 / rank of first relevant) |
Rewards putting the first relevant item on top. |
| MAP | Mean over queries of (mean of Precision at each relevant rank) |
Precision across the whole ranked list. |
| NDCG@k | DCG@k / IDCG@k, with DCG = Σ relᵢ / log₂(i+1) |
Graded relevance with a position discount; 1 = ideal ordering. |
Note: R1 (Recall@1) is a ranking score — don't confuse it with R², the regression score above. Both are demonstrated here.
docker compose up
The test stage runs automatically as the last container. To run the tests directly (after stages 1–3 have produced data/ and models/):
cd lab
python -m pytest tests -v -s # -s shows each test's metric / formula / values
- Hand vs scikit-learn: every score is computed twice and asserted equal (
pytest.approx) — the formulas above are exactly what runs. Follow the suite withdocker compose logs -f tests. - Binary classification: the data is
70/30imbalanced, so Accuracy0.79looks fine while Recall0.53and F10.60tell the real story.test_threshold_sweepprints TP/FP/FN/TN and the Precision–Recall trade-off at each decision threshold (recall 0.95att=0.1 → 0.03att=0.9). - Multi-class:
test_micro_f1_equals_accuracyshows micro-F1= Accuracy = 0.67exactly (the single-label identity); macro-F10.57 < weighted-F1 0.65because the rare classes (support41–61) score worst — macro is the average that exposes them. - Regression: same errors, different stories — MAE
58.0vs RMSE72.1(outliers weigh more in RMSE), MAPE8.2%, R²0.877, max error207vs median AE47. - Ranking: the ranker almost always surfaces relevant items somewhere in the top
10(Recall@10= 0.97) but R1= Recall@1 = 0.30and Precision@1= 0.80— being right at the very top is the hard part.
Generated data and models are included for study; they are recreated on each run.
The above frequently show up in the GCP PMLE but there aren't that many hands-on labs for these in skills.google.