Predict Customer Churn with ML.NET

This exercise teaches .NET developers how to reason about an imbalanced binary-classification problem, compare controlled modeling experiments, and convert a model score into a business decision.

It is Exercise 3 in the AInDotNet Forecasting and Predictive AI series. The central lesson is that model training is only part of a predictive system: precision, recall, validation design, classification thresholds, and the business cost of mistakes determine whether a churn prediction is useful.

https://aindotnet.com/2026/09/predict-customer-churn-csharp-mlnet/

Prediction Contract

• Business question: Which current customers are at elevated risk of churning?
• Model output: A churn score/probability and a threshold-based churn classification.
• Historical label: Churn, represented as Yes or No in the source data.
• Prediction point: A snapshot of an existing customer before a retention decision is made.
• Available information: Customer demographics, tenure, subscribed services, contract, billing, payment method, and charges represented in the snapshot.
• Excluded feature: customerID is retained for diagnostics but excluded from the feature vector because it is an identifier rather than a generalizable customer characteristic.
• Possible action: Prioritize selected customers for a retention workflow.
• Success criterion: Detect enough true churners to make intervention worthwhile without targeting so many non-churners that the program becomes uneconomic.

Important limitation

The dataset does not define when the snapshot was taken, a future prediction horizon such as “churn within 30 days,” or whether every feature was captured before the churn outcome. Therefore, this exercise demonstrates classification methodology but cannot establish prospective production performance. A real churn system needs a time-indexed label and strict proof that every feature was available at the operational prediction point.

What the Experiments Test

Run	Question	Major change
A	How well do four numeric features predict churn?	Establishes the first ML baseline with SDCA logistic regression.
B	Do selected business-context categories help?	Adds contract, internet service, payment method, and paperless billing while keeping the trainer and data populations fixed.
C	Does the full customer/service context help further?	Adds the remaining categorical features while keeping the trainer and data populations fixed.
D	Can AutoML find a stronger trainer and hyperparameters?	Keeps the full features and changes the modeling algorithm/search process.
E	Can a lower classification threshold find more churners?	Keeps the Run D model fixed and changes only the decision threshold.

The application uses a reproducible 60/20/20 training/validation/final-test design. Runs A–D and threshold selection use validation data. The final test population remains untouched until the model and threshold have been selected.

Dataset

The lab uses the commonly distributed IBM Telco Customer Churn sample with 7,043 customer rows and 21 columns. IBM publishes a copy in its archived telco-customer-churn-on-icp4d repository:

• Dataset: https://github.com/IBM/telco-customer-churn-on-icp4d/blob/master/data/Telco-Customer-Churn.csv
• Source repository: https://github.com/IBM/telco-customer-churn-on-icp4d
• Source repository license: Apache License 2.0

The same data is widely mirrored on Kaggle, commonly under the filename WA_Fn-UseC_-Telco-Customer-Churn.csv. Dataset copies can differ, so verify that your file contains 7,043 data rows and the 21 columns profiled by the application. Review and comply with the terms of the source from which you obtain the data. The AInDotNet MIT license applies to this exercise’s source code, not to independently obtained third-party data.

Install the data

1. Download the CSV from the IBM source above or another source you are authorized to use.

2. Rename it to WA_Fn-UseC_-Telco-Customer-Churn.csv if necessary.

3. Place it in the project’s Data folder:

   ConsoleApp1/Data/WA_Fn-UseC_-Telco-Customer-Churn.csv

The CSV is intentionally excluded from this repository so users obtain it from its source and review the applicable terms.

Requirements

• Visual Studio 2026 or the .NET 10 SDK
• ML.NET 5.0.0
• ML.NET AutoML 0.23.0

Package versions are pinned in ConsoleApp1.csproj for repeatability.

Build and Run

From the ConsoleApp1 folder:

dotnet restore
dotnet run

The application profiles the raw data, creates fixed training/validation/final-test populations, runs five experiments, reports classification metrics and confusion matrices, saves the winning model, reloads it, and scores one hypothetical customer.

AutoML uses a time budget, so the exact winning trial, trainer, and metrics may differ across machines or ML.NET versions. Fixed split seeds and deterministic SDCA settings stabilize the controlled baseline experiments, but time-limited AutoML is not guaranteed to be bit-for-bit reproducible.

Reading the Metrics

• Accuracy: Fraction of all customers classified correctly. Compare this with the 73.5% majority-class baseline.
• Precision: Among customers predicted to churn, the fraction that actually churned.
• Recall: Among actual churners, the fraction the model identified.
• F1: Harmonic balance of precision and recall.
• ROC-AUC: Ranking/separation quality across thresholds. It does not select a business threshold.

The probability column is useful for ranking and threshold experiments, but it should not automatically be interpreted as a perfectly calibrated real-world churn probability.

Generated Model

Running the application creates Models/CustomerChurn-Model.zip. The trainer-neutral name is deliberate because AutoML can select a different trainer on another run or machine. This generated artifact is excluded from source control; regenerate it locally so it matches the code, data, package versions, and AutoML run on your machine.

Try It Yourself

1. Add one feature group at a time and explain which metric moved and why.
2. Remove TotalCharges; compare the result with removing MonthlyCharges or tenure.
3. Add thresholds above 0.50 and below 0.30; plot the precision/recall tradeoff.
4. Replace F1 threshold selection with an explicit cost model for false positives and false negatives.
5. Repeat the split with several seeds and report the range of final-test metrics.
6. Add PR-AUC and compare what it reveals with ROC-AUC.
7. Investigate high-confidence errors by customer segment rather than only by customer ID.
8. Design a production label such as “churn within the next 30 days” and list which fields would truly exist at that prediction point.

Production Reality Check

A production implementation would require time-based labels and validation, documented feature availability, representative and monitored data, calibrated probabilities where needed, a cost-based decision policy, batch or online scoring infrastructure, model/version lineage, drift and outcome monitoring, security and privacy controls, and an experiment proving that retention actions improve outcomes. Those omissions are deliberate for a focused teaching application.

License

The exercise source code is licensed under the MIT License. Third-party data retains its own terms.