Big Idea • RFM • Dataset • Engineering • Choosing K • The Duel • Personas
Note
In one sentence: this project reads about a million store receipts, turns them into one simple profile per customer, and lets two clustering algorithms sort everyone into 4 groups, each with its own marketing plan.
Explain it like I'm five: Imagine a shop owner holding a giant pile of receipts. Sending the same discount email to everyone is wasteful: your best customer, a first-time visitor, and someone who left two years ago need very different messages. So we sort customers into groups of "people who behave alike" without telling the computer what the groups are. That is called unsupervised learning.
~1M receipts ➜ One profile per customer (RFM) ➜ Clustering ➜ 4 personas + a plan for each
Three questions describe almost any shopper:
| Letter | The question | Measured as | |
|---|---|---|---|
| Recency | How recently did they shop? | Days since last completed purchase | |
| Frequency | How often do they shop? | Number of distinct completed purchases | |
| Monetary | How much do they spend? | Total revenue from that customer |
RFM was chosen because three business-friendly numbers are easy to explain to a marketing team, and that matters as much as the math.
Source: Online Retail II Dataset on Kaggle
Scale: ~1,000,000 raw invoice transactions, collapsed into one RFM profile per unique customer.
flowchart LR
A[" Raw invoices<br/>~1M rows"] --> B[" Clean<br/>drop missing Customer ID<br/>and cancelled orders"]
B --> C[" Aggregate<br/>one RFM row per customer"]
C --> D[" Log transform<br/>np.log1p"]
D --> E[" StandardScaler"]
E --> F[" K-Means and<br/>Hierarchical Clustering"]
classDef step fill:#4c1d95,stroke:#a78bfa,color:#fff;
classDef out fill:#9d174d,stroke:#f472b6,color:#fff;
class A,B,C,D,E step;
class F out;
Before and after (an illustrative example, not real rows from the dataset):
| Before: many receipt lines | After: one customer profile | ||
|---|---|---|---|
| Customer 17850, 3 Dec, £20 | Recency | 12 days | |
| Customer 17850, 9 Dec, £35 | ➜ | Frequency | 3 purchases |
| Customer 17850, 15 Dec, £45 | Monetary | £100 |
Step 1: Cleaning (click to expand)
Dropped rows missing Customer ID and removed cancelled or negative-quantity rows (Quantity > 0), so RFM reflects only real, completed purchases.
Step 2: Aggregation (click to expand)
Computed spend per line (Quantity × Price), then grouped by Customer ID to calculate Recency, Frequency, and Monetary. About 1M rows became one row per customer.
Step 3: Log transform, the "billionaire in the room" fix (click to expand)
Monetary and Frequency were heavily right-skewed. Picture ten people in a room and one billionaire walks in: the average wealth explodes, and it tells you nothing about the typical person. A few mega-spenders would do the same to the clustering distances, so np.log1p() was applied to shrink the extremes and let the typical behavior show.
Step 4: Scaling (click to expand)
StandardScaler puts Recency, Frequency, and Monetary on the same scale. Both algorithms are distance-based, so without this, a feature measured in big numbers (money) would quietly bully a feature measured in small ones (order counts).
Instead of guessing, two independent methods were used to cross-check the number of clusters.
With ward linkage and Euclidean distance, customers are merged step by step into one big tree. Cutting where the vertical gaps are largest reveals 4 natural groupings.
The WCSS plot (how tightly packed the clusters are) drops fast, then flattens. The bend, the "elbow", sits at K = 4: past that point, extra clusters barely help.
Important
Two different methods, one answer. The dendrogram (linkage distances) and the elbow (variance minimization) reach K = 4 by completely different math. Agreement between them is what makes the choice trustworthy.
Both algorithms ran on the same scaled RFM data. There are no "correct answer" labels in unsupervised learning, so they are judged with two internal scores.
| Silhouette ↑ | Davies-Bouldin ↓ | |
|---|---|---|
| K-Means | 0.3663 | 0.9355 |
| Hierarchical | 0.3314 | 0.9317 |
Reading the scores in plain words:
- Silhouette (higher is better) asks "does each customer feel at home in their own group, far from the others?" K-Means scores a bit higher, so its groups are slightly tighter and more distinct.
- Davies-Bouldin (lower is better) asks "how much does each group overlap with its closest neighbor?" Hierarchical edges it by a hair.
Each algorithm wins one round, which is common when comparing these two families. K-Means was used for the final personas because of its stronger Silhouette Score and its much better scalability to large customer bases.
How each algorithm splits customers on the Recency vs Monetary plane:
Champions / VIP Recent, frequent, big spenders VIP rewards, early access, dedicated support |
Loyal Regulars Steady, frequent, good spenders Cross-sell and a loyalty points program |
Recent / New Buyers Just arrived, spending lightly Welcome discount, onboarding emails, popular picks |
At-Risk / Lost Gone quiet for a long time Win-back campaigns and re-engagement offers |
| Segment | Avg. Recency | Avg. Frequency | Avg. Monetary | Strategy |
|---|---|---|---|---|
| Champions / VIP | Low (< 30 days) | High (> 15 orders) | High (> $10,000) | Exclusive VIP rewards, early access to new collections, dedicated account support |
| Loyal Regulars | Moderate (30–90 days) | High (> 10 orders) | Moderate–High | Cross-selling recommendations, loyalty points to maximize Customer Lifetime Value (CLV) |
| Recent / New Buyers | Low recency, low order count | Low | Low–Moderate | Welcome discount codes, onboarding email sequences, popular product suggestions |
| At-Risk / Lost | High–Very High (> 120–250+ days) | Low (1–2 orders) | Low–Moderate | Targeted win-back campaigns, re-engagement promotions, low-cost automated emails |
quadrantChart
title Customer personas (positions are illustrative)
x-axis Bought long ago --> Bought recently
y-axis Low spend --> High spend
quadrant-1 Recent big spenders
quadrant-2 Old big spenders
quadrant-3 Old small spenders
quadrant-4 Recent small spenders
Champions: [0.86, 0.9]
Loyal Regulars: [0.6, 0.66]
New Buyers: [0.85, 0.2]
At-Risk and Lost: [0.14, 0.3]
Tip
Why this matters: every cluster is more than a statistical blob. It is a group a marketing team can act on, each with its own message and budget, instead of one blanket campaign for everyone.
| Category | Tools |
|---|---|
| Language | Python |
| Data Handling | Pandas, NumPy |
| Machine Learning | Scikit-learn (K-Means, Agglomerative Clustering) |
| Visualization | Matplotlib, Seaborn, SciPy (dendrogram) |
Customer-Segmentation-RFM-Clustering/
│
├── images/
│ ├── dendrogram.png
│ ├── elbow_method.png
│ └── cluster_comparison.png
├── data/
│ └── online_retail_ii.csv
├── notebooks/
│ └── customer_segmentation_rfm.ipynb
├── README.md
└── requirements.txt
# 1. Clone the repository
git clone https://github.com/khaled-amireh/Customer-Segmentation-RFM-Clustering.git
cd Customer-Segmentation-RFM-Clustering
# 2. Create and activate a virtual environment (recommended)
python -m venv venv
source venv/bin/activate # On Windows: venv\Scripts\activate
# 3. Install dependencies
pip install -r requirements.txt
# 4. Launch the notebook
jupyter notebook notebooks/customer_segmentation_rfm.ipynbClick to see what this project does not do (yet)
- Clustering has no ground-truth labels, so "correct" segments can't be proven. Evaluation relies on internal metrics (Silhouette, Davies-Bouldin) and on whether the personas make business sense.
- RFM alone doesn't capture product preferences or sales channel behavior, which could sharpen the segments.
- The groups come from a snapshot in time. Customers change, so the clustering would need to be re-run periodically in production.
- Add product-category and channel features alongside RFM
- Try DBSCAN or Gaussian Mixture Models
- Build a recurring re-clustering pipeline to track segment drift
- A/B test each persona's marketing strategy to prove real-world impact
- Add a customer lifetime value (CLV) prediction layer on top of the segments
Khaled Amireh
If this project helped you, drop a ⭐ on the repo.