miniHive

A distributed SQL query engine built on Hadoop MapReduce that parses and executes SQL-like queries against HDFS-stored datasets using custom mapper and reducer pipelines.

---

Architecture

                        ┌─────────────────────────┐
                        │   HDFS Schema Registry   │
                        │   index.txt - metadata   │
                        └────────────┬────────────┘
                                     │ schema lookup
                                     ▼
┌──────────────┐   SQL query   ┌─────────────────┐
│     User     │ ─────────────▶│  Query Parser   │
│  SQL query   │               │    main.py      │
└──────────────┘               │ LOAD/DELETE/    │
                               │ SELECT          │
                               └────────┬────────┘
                                        │
                        ┌───────────────▼────────────────┐
                        │       Hadoop Streaming          │
                        │  ┌──────────────┐  ┌─────────┐ │
                        │  │    Mapper    │─▶│ Reducer │ │
                        │  │ filter+proj  │  │SUM/MAX/ │ │
                        │  └──────┬───────┘  │ COUNT   │ │
                        │         │ input    └────┬────┘ │
                        └─────────│───────────────│──────┘
                                  │               │
                        ┌─────────▼──────┐        │
                        │   HDFS Data    │        │
                        │  CSV tables    │        ▼
                        └────────────────┘  ┌──────────┐
                                            │ Console  │
                                            │ results  │
                                            └──────────┘

---

Features

• SQL interface - interactive miniHive> prompt supporting LOAD, DELETE, and SELECT queries
• HDFS-backed schema registry - persists table metadata in index.txt across sessions, enabling dynamic column resolution and type-aware query execution
• Predicate pushdown - WHERE clause filtering at the map phase, minimizing data shuffled across the network
• Aggregations - SUM, MAX, COUNT supported via reducer pipeline
• Hadoop Streaming - mapper and reducer are plain Python scripts, executed via the Hadoop Streaming API

---

Query Syntax

LOAD

Load a CSV file into HDFS as a table with a defined schema.

LOAD db/table.csv AS (col1:type,col2:type) INPATH local_file.csv

Example:

LOAD mydb/employees.csv AS (id:int,name:string,salary:int) INPATH /data/employees.csv

DELETE

Delete a database and all its tables from HDFS.

DELETE db

Example:

DELETE mydb

SELECT

Query a table with optional WHERE filter and aggregation.

SELECT col FROM db/table.csv
SELECT col FROM db/table.csv WHERE col>value
SELECT col FROM db/table.csv aggr=SUM
SELECT col FROM db/table.csv WHERE col>value aggr=MAX

Examples:

SELECT salary FROM mydb/employees.csv
SELECT salary FROM mydb/employees.csv WHERE salary>50000
SELECT salary FROM mydb/employees.csv aggr=SUM
SELECT name FROM mydb/employees.csv aggr=COUNT

---

Project Structure

bigdata_proj/
├── main.py       # SQL parser and query controller - entry point
├── map.py        # Hadoop mapper - row filtering and column projection
├── red.py        # Hadoop reducer - SUM, MAX, COUNT aggregations
└── readme.md     # Project documentation

---

How It Works

Query Parsing (main.py)

The parser tokenizes the input query, validates syntax, and routes to the appropriate handler. For SELECT queries, it reads the schema from the HDFS index file, resolves column indices, and dynamically constructs the Hadoop Streaming job command.

Mapper (map.py)

Receives each row from HDFS via stdin. Applies the WHERE filter (predicate pushdown) and projects the requested column. Outputs column_name,value pairs to stdout.

Reducer (red.py)

Receives mapper output and applies the aggregation function:

• SUM - sums all integer values
• MAX - returns the maximum integer value
• COUNT - counts all rows
• No aggregation - passes values through directly

Schema Registry

On LOAD, the table schema is written to index.txt in the HDFS database directory. On SELECT, the schema is read from index.txt to resolve column names to indices and determine datatypes for type-aware comparisons.

---

Setup and Requirements

• Hadoop 3.2.0
• Python 3
• HDFS running locally or on a cluster

Run

python3 main.py

This starts the interactive miniHive> prompt.

---

Resume Highlights

• Engineered miniHive, a distributed SQL engine on Hadoop MapReduce executing SELECT, WHERE, and SUM/MAX/COUNT aggregations on HDFS datasets via custom Python mapper-reducer pipelines
• Designed HDFS-backed schema registry using index files for persistent table metadata, enabling dynamic column resolution and type-aware query execution
• Implemented predicate pushdown at the map phase to filter rows before shuffle, minimizing network I/O across the MapReduce pipeline
• Built a SQL parser from scratch supporting DDL/DML operations, dynamically constructing Hadoop Streaming job commands with resolved column indices and inferred datatypes