Fully automated Databricks PySpark DLT pipeline for integrating with Kafka Topics & Kafka Schema Registry

An enterprise Apache Spark ML pipeline benchmarking ensemble methods, featuring MLOps tracking (MLflow), Docker deployment, and autonomous Gen AI evaluation.

Web Data Processing Application involving Big Data NLP and entity linking(disambiguation)

Comprehensive Jupyter notebook designed for analyzing and predicting pharmaceutical sales with PySpark

Python package for Spark programming and its powerful, higher-level libraries such as SparkSQL and MLlib (for Machine Learning), very useful for general Big Data analysis.

Big Data Analytics module (UEL-CN-7031), featuring Hive and PySpark analysis on the UNSW-NB15 dataset, with detailed tasks, scripts, visualizations, and reports