🧹 Data Preprocessing Pipeline (DPP)

An end-to-end data preprocessing pipeline built on the Breast Cancer dataset - covering data cleaning, missing value handling, outlier detection, feature encoding, feature scaling, and a baseline classification model, all using Python and scikit-learn.

🔗 Repository: github.com/FathimaNufla2000/data-preprocessing-pipeline

---

📌 Project Overview

This project walks through the complete data preprocessing workflow needed before feeding data into a machine learning model. It uses the Breast Cancer dataset to demonstrate cleaning, transforming, and preparing raw tabular data, and finishes with a Logistic Regression model to test the pipeline end-to-end.

---

✨ Features

• 🧹 Single-Value Column Removal: Detects and drops non-informative columns with only one unique value
• 🔁 Duplicate Removal: Identifies and removes duplicate rows
• ❓ Missing Value Handling:
  • Row-drop approach (dropna())
  • Imputation approach — mean/median for numerical features, mode for categorical features
• 🎯 Outlier Detection: Detects outliers using the IQR (Interquartile Range) method and removes them
• 🔢 Encoding Techniques:
  • OrdinalEncoder for ordered categorical data
  • OneHotEncoder for nominal categorical data
  • LabelEncoder for target variable encoding
• 📏 Feature Scaling:
  • MinMaxScaler (normalization)
  • StandardScaler (standardization)
• 🤖 Baseline Model: Logistic Regression classifier trained on the fully preprocessed dataset, with train/test split and accuracy evaluation

---

🛠 Technology Stack

• Language: Python 3
• Libraries: Pandas, NumPy, scikit-learn
• Environment: Jupyter Notebook

---

📂 Project Structure

data-preprocessing-pipeline/
├── DPP.ipynb                          # Main preprocessing & modeling notebook
├── breast-cancer-1.csv                # Breast cancer dataset (raw)
├── breast-cancer-1 dataset.csv        # Breast cancer dataset (used for model training)
└── README.md

---

📊 Dataset

The dataset used is the Breast Cancer dataset, containing patient records with a mix of categorical features (age group, menopause status, tumor size, breast side, breast quadrant) and a boolean/categorical target variable indicating recurrence of cancer (class).

---

⚙️ Prerequisites

• Python 3.x
• Jupyter Notebook
• Pandas, NumPy, scikit-learn

---

🚀 Installation & Running Instructions

Method 1: Clone and Run Locally

git clone https://github.com/FathimaNufla2000/data-preprocessing-pipeline.git
cd data-preprocessing-pipeline
pip install pandas numpy scikit-learn jupyter
jupyter notebook DPP.ipynb

Make sure both CSV files are in the same folder as the notebook before running.

Method 2: Using an IDE (VS Code)

1. Open the project folder in VS Code
2. Install the Jupyter extension
3. Open DPP.ipynb
4. Select a Python kernel and run cells sequentially

---

🔑 Key Steps Explained

• Outlier Detection (IQR Method): Values below Q1 - 1.5×IQR or above Q3 + 1.5×IQR are flagged and removed
• OrdinalEncoder vs OneHotEncoder vs LabelEncoder:
  • OrdinalEncoder — for input features with a natural order
  • OneHotEncoder — for nominal input features with no order
  • LabelEncoder — reserved for encoding the target variable only
• MinMaxScaler vs StandardScaler: MinMaxScaler rescales values to a [0,1] range; StandardScaler centers data around mean 0 with unit variance
• Model Evaluation: A Logistic Regression model is trained on the encoded/split data and evaluated using accuracy_score

---

🩺 Troubleshooting

FileNotFoundError → Ensure both breast-cancer-1.csv and breast-cancer-1 dataset.csv are in the same directory as the notebook.

ModuleNotFoundError: No module named 'sklearn' → Run pip install scikit-learn.

SettingWithCopyWarning from Pandas → Safe to ignore for this exercise, but consider using .loc[] for cleaner assignments in future versions.

---

🚧 Future Enhancements

• Combine both CSV files into a single clean dataset to avoid duplication
• Wrap the full pipeline using scikit-learn's Pipeline and ColumnTransformer
• Add cross-validation for more robust model evaluation
• Try additional classifiers (Decision Tree, Random Forest, SVM) for comparison
• Add confusion matrix and classification report for deeper evaluation

---

👩‍💻 Author

Fathima Nufla GitHub · LinkedIn

---

📄 License

This project was developed for educational and academic purposes.