Ames Housing Price Prediction

[Disclaimer: I mainly made this project to learn modelling, project structure and workflow. You may not find anything particularly innovative here, and if you think this project is not special in any way, you would be right. I still enjoyed doing this though!]

End-to-end machine learning pipeline using scikit-learn to predict house prices using the Ames Housing dataset. Mainly intended to understand flow, analysis, pipeline design and experimentation.

---

Objective

• Build a strong regression model
• Understand dataset through EDA, and perform feature engineering for better performance
• Implement cleaning, preprocessing and proper pipelines for robust working of model
• Experiment with multiple models, and understand why their results came out to as they did
• Use visualisations to better understand data as well as models

---

Dataset

Ames Housing Dataset (2930 rows, 82 features) Kaggle link: https://www.kaggle.com/datasets/marcopale/housing
[Since it's a small dataset, I have included the dataset in the repo. I have also included a small CSV file(12 rows) named test_input_ames.csv for testing user predictions]

---

Approach

Cleaning

• Dropped 'PID' and 'Order' (identifiers)
• Dropped 'Alley' as 2732 out 2930 rows have missing value for this feature. Too little data to perform imputation confidently
• Dropped 'Misc Feature', as 2824 houses lack Misc Features, and the mean price of houses with Misc Features($181,000) and without Misc Features($177,000) is nearly same, hence no point in converting column to binary feature either
• Replaced missing values in categorical cols ['Fireplace Qu', 'Mas Vnr Type', 'Pool QC', 'Fence', 'Garage Type', 'Garage Finish', 'Garage Cond', 'Garage Qual'] with 'NA'
• Replaced missing values in numerical cols ['Garage Yr Blt', 'Garage Area', 'Garage Cars'] with 0

Preprocessing

• num_pipeline to treat numeric columns that don't need log transformation. Applies KNNImputer and StandardScaler
• log_pipeline for skewed and non Gaussian columns ['Lot Area', 'Bsmt Unf SF', 'Total Bsmt SF', '1st Flr SF', 'Gr Liv Area', 'Open Porch SF'], that seemed to benefit from log transformation
• cat_pipeline to apply SimpleImputer with 'most_frequent' strategy, FunctionTransformer to convert potential numeric values in these columns to string, and apply OneHotEncoding
• Column Transformer to apply above pipelines to respective columns

Pipeline

• Built sklearn Pipeline using ColumnTransformer and Pipeline
• Includes:
  • Initial Cleaning transformations
  • Preprocessing transformations through ColumnTransformer
  • Final Model with optimally tuned hyperparameters
• Ensures consistent transformations during training and inference
• Enables direct prediction on raw input data

Models tried and evaluated with Optuna tuned hyperparameters

• Decision Tree
• Random Forest
• Gradient Boosting (final model)
• K Neighbors Regressor
• ElasticNet
• Voting ensemble of tuned RF, GB, KNN, ElasticNet
• Bagging ensemble of tuned Gradient Boosting
• Base model GB + Residual Model RF
  

Chose Gradient Boosting to be the final model, since ensemble techniques did not improve performance significantly, and Gradient Boosting provided the best individual results

Hyperparameter tuning

• Used Optuna for optimised and automated tuning
• Performed tuning for multiple models and compared results to choose best candidate

Results

• Gradient Boosting performed best
• RMSE on test data: ~20,000-23,000
• R2 score on test data: ~0.92-0.93
• RMSE on cross validation: ~22,000
• R2 score on cross validation: ~0.92-0.93

Key Insights

• During later stages, Feature engineering had stronger impact than model complexity
• PCA with around 95 components explained 95% variance in original data, but was not used, since it showed a ~15% drop in prediction performance
• Ensemble methods such as Bagging and Voting did not significantly improve performance
• Performance of model plateaued after tuning
• Simple ElasticNet Regression performed significantly well, beating KNN and DT, but fell slightly fell short to RF and GradientBoosting

Project Structure

src/                 # Core pipeline and logic
datasets/        # raw, processed and test data
models/           # saved models
notebooks/     # experimentation
reports/           # visualisations

How to Run

1. Install dependencies:
   

pip install -r requirements.txt

2. Run program:
   

python main.py

3. Options:
   • Build and train model
   • Evaluate model on test data
   • Feed CSV files to obtain predictions