Skip to content
BranchesTags
 
 

Repository files navigation

FacultyGraph

University Expertise Discovery & Research Search Engine

Public URL: https://infraglyph.vercel.app

A reproducible web-to-data pipeline that transforms fragmented university faculty webpages into structured, queryable academic expertise data and exposes the resulting dataset through a REST API.

Python FastAPI SQLite Web Data Data Quality Project

Table of Contents

Overview

FacultyGraph is an end-to-end data engineering and academic information discovery project.

It addresses a practical data problem:

University faculty expertise is distributed across multiple webpages, directories, inconsistent HTML structures, and semi-structured profile content.

FacultyGraph builds a reproducible pipeline that converts those webpages into structured, queryable faculty records.

University Website
        ↓
Faculty Directory Discovery
        ↓
Profile URL Discovery
        ↓
Profile-Level Extraction
        ↓
Raw HTML Preservation
        ↓
Cleaning & Normalization
        ↓
SQLite Storage
        ↓
FastAPI Read-Only APIs
        ↓
Downstream Applications

The current implementation is primarily the data engineering foundation required for reliable faculty discovery. It prepares structured faculty data for downstream search, NLP, recommendation, and semantic discovery without over-claiming future capabilities as part of the validated core system.

Why FacultyGraph?

Faculty expertise is difficult to discover because information is distributed across:

  • Department directories
  • Faculty profile pages
  • Different faculty categories
  • Semi-structured HTML
  • Long biographies
  • Research and teaching descriptions
  • Publication lists
  • Pages with inconsistent or missing fields

The first requirement for any downstream intelligence layer is a clean, reproducible data foundation.

FacultyGraph focuses on that foundation:

Fragmented Web Information
          ↓
Reliable Extraction
          ↓
Structured Academic Data
          ↓
Queryable Faculty Knowledge Base

Problem Statement

Traditional faculty discovery has several practical problems:

  • Faculty pages are distributed across multiple directories.
  • URL structures may vary across departments or categories.
  • Profile pages are semi-structured rather than database-ready.
  • Fields may be missing or inconsistently populated.
  • HTML contains navigation and presentation noise.
  • Repeated collection requires resilient crawling.
  • Downstream applications need structured and stable records.

The project therefore treats the university website as a data source rather than as an application interface.

Web Pages
   ↓
Raw Data
   ↓
Structured Records
   ↓
Clean Data
   ↓
Database
   ↓
API

Project Objective

FacultyGraph is designed to:

  1. Discover faculty profiles across multiple university directories.
  2. Extract structured and semi-structured faculty information.
  3. Preserve raw source HTML for reproducibility and future reprocessing.
  4. Normalize noisy and inconsistent text.
  5. Persist structured information in SQLite.
  6. Expose the dataset through a lightweight FastAPI service.
  7. Quantify data quality and missingness.
  8. Provide a stable foundation for future academic search and NLP applications.

What the Current Codebase Implements

✅ Web ingestion

Multiple faculty directories are crawled and profile URLs are discovered dynamically.

✅ Profile extraction

Faculty profile pages are parsed into structured fields.

✅ Raw-source preservation

Raw HTML is retained for later cleaning, reprocessing, and downstream analysis.

✅ Cleaning and normalization

HTML noise is removed and text fields are normalized.

✅ SQLite persistence

Faculty records are stored in a schema-driven relational structure.

✅ Read-only REST API

The stored faculty data is exposed through FastAPI endpoints.

✅ Data quality analysis

The project includes scripts for category statistics, missing-value analysis, dataset overview, and text-length statistics.

✅ Frontend integration

The repository contains a frontend application for browsing faculty information.

System Architecture

                     ┌──────────────────────────┐
                     │   University Web Pages   │
                     └────────────┬─────────────┘
                                  │
                                  ▼
                     ┌──────────────────────────┐
                     │ Faculty Directory        │
                     │ Discovery                │
                     └────────────┬─────────────┘
                                  │
                                  ▼
                     ┌──────────────────────────┐
                     │ Faculty Profile URL      │
                     │ Discovery & Normalization│
                     └────────────┬─────────────┘
                                  │
                                  ▼
                     ┌──────────────────────────┐
                     │ Profile-Level Scraping   │
                     │ & Extraction             │
                     └────────────┬─────────────┘
                                  │
                                  ▼
                     ┌──────────────────────────┐
                     │ Raw HTML / Bronze Data   │
                     └────────────┬─────────────┘
                                  │
                                  ▼
                     ┌──────────────────────────┐
                     │ Cleaning & Transformation│
                     └────────────┬─────────────┘
                                  │
                                  ▼
                     ┌──────────────────────────┐
                     │ SQLite Structured Store  │
                     └────────────┬─────────────┘
                                  │
                         ┌────────┴─────────┐
                         │                  │
                         ▼                  ▼
               ┌─────────────────┐  ┌─────────────────┐
               │ FastAPI REST API│  │ Frontend Client │
               └─────────────────┘  └─────────────────┘

End-to-End Data Flow

1. University Website
        ↓
2. Directory Discovery
        ↓
3. Faculty URL Discovery
        ↓
4. HTTP Fetch
        ↓
5. HTML Extraction
        ↓
6. Raw Record Creation
        ↓
7. Data Validation
        ↓
8. Text Cleaning
        ↓
9. Structured Transformation
        ↓
10. SQLite Persistence
        ↓
11. REST API
        ↓
12. Faculty Discovery Interface

This makes the project a complete web-to-database-to-API pipeline rather than a standalone scraper.

Data Engineering Pipeline

1. Faculty Directory Discovery

The ingestion layer discovers faculty profiles from multiple faculty directories.

The documented categories include:

  • Regular Faculty
  • Adjunct Faculty
  • International Adjunct Faculty
  • Distinguished Professors
  • Professors of Practice

The crawler dynamically discovers profile URLs rather than relying entirely on a manually maintained list.

Handling real-world website variation

The ingestion layer accounts for:

  • Inconsistent URL taxonomy
  • Absolute and relative URLs
  • Missing profile fields
  • Dynamic profile discovery
  • Network failures

2. Profile Extraction

For each faculty profile, the pipeline extracts structured and semi-structured attributes such as:

  • Faculty name
  • Image
  • Education
  • Phone
  • Address
  • Email
  • Biography
  • Specialization
  • Teaching
  • Research
  • Publications

The raw page content is also preserved so the pipeline does not lose source information during transformation.

3. Raw Data Preservation

FacultyGraph intentionally separates source preservation from cleaned data.

University Profile Page
        ↓
Raw HTML
        ↓
Extraction
        ↓
Cleaned Fields

Why preserve raw HTML?

It enables:

  • Reprocessing when extraction rules change
  • Debugging extraction failures
  • Auditing source content
  • Re-running cleaning logic
  • Building future NLP datasets

4. Cleaning & Transformation

The cleaning layer converts semi-structured web content into usable text.

Main transformations

  • Remove HTML tags
  • Remove noisy markup
  • Normalize whitespace
  • Normalize encoding and formatting
  • Handle missing values
  • Handle null/irregular fields
  • Produce cleaner text for downstream analysis

Missing data remains missing rather than being fabricated.

5. Storage

FacultyGraph uses SQLite for lightweight persistence.

Storage characteristics

  • Schema-driven database
  • Structured faculty records
  • Raw HTML preserved as source data
  • Unique constraints for safer reruns
  • Lightweight local deployment
  • Simple API integration

Logical storage layers

Bronze
──────
Raw HTML / Source Content

        ↓

Clean
──────
Normalized Faculty Fields

        ↓

Serving
───────
SQLite Records
        ↓
FastAPI API

6. API Serving

FacultyGraph exposes faculty information through a read-only FastAPI API.

Endpoint Purpose
GET /faculty Retrieve faculty records
GET /faculty/{id} Retrieve a faculty member by ID
GET /faculty/category/{category} Filter faculty by category

Swagger UI:

http://127.0.0.1:8000/docs

Data Quality & Dataset Profile

The current processed dataset contains:

Metric Value
Faculty records 110
Attributes per record 12

The dataset contains multiple faculty categories and meaningful variation in field completeness.

This makes it a useful example of real-world semi-structured web data engineering, where missingness and inconsistency are part of the problem rather than anomalies to hide.

Faculty Distribution

Faculty Category Count Percentage
Regular Faculty 67 60.91%
Adjunct Faculty 26 23.64%
International Adjunct Faculty 11 10.00%
Professor of Practice 4 3.64%
Distinguished Professor 2 1.82%

Missing Data Analysis

Field Missing %
Education 1.82%
Phone 29.09%
Address 31.82%
Email 0.91%
Biography 38.18%
Specialization 1.82%
Publications 33.64%
Teaching 36.36%
Research 87.27%

What the Missingness Tells Us

Not all fields are equally complete.

For example:

  • Email is comparatively complete.
  • Education and specialization are also relatively well populated.
  • Biography, teaching, and publications have moderate missingness.
  • Research has very high missingness in the current dataset.

This matters for downstream applications because:

Search Quality
     ↓
depends on
     ↓
Field Availability

A faculty record with no research text cannot provide research-based relevance as reliably as a complete record.

Therefore:

Data quality is part of application quality.

Text Statistics & NLP Readiness

Field Non-Empty Records Avg Length Max Length
Biography 68 541 2,439
Specialization 108 111 503
Teaching 70 123 1,885
Research 14 42 1,897
Publications 73 1,831 16,202

These statistics are useful for downstream:

  • Search indexing
  • Text chunking
  • Embedding generation
  • NLP preprocessing
  • Profile summarization

The current repository should primarily be understood as the data engineering foundation, while advanced semantic discovery should only be treated as implemented when the corresponding module has been executed and validated.

Reliability & Robustness

Web scraping fails in ways that ordinary database pipelines do not.

FacultyGraph accounts for several real-world collection issues.

URL Robustness

Supports:

Absolute URLs
Relative URLs
Inconsistent URL taxonomy

Missing-Field Handling

Profiles may omit:

Phone
Address
Biography
Teaching
Research
Publications

The pipeline continues rather than assuming every profile is complete.

Network Robustness

Request
  ↓
Failure?
  ├── No → Continue
  └── Yes
        ↓
      Retry
        ↓
     Backoff
        ↓
      Retry

This prevents transient HTTP failures from unnecessarily terminating the collection process.

Data Lineage

A core design principle is preserving the relationship between:

Source URL
    ↓
Raw HTML
    ↓
Extracted Record
    ↓
Cleaned Record
    ↓
Stored Record
    ↓
API Response

This gives the project a simple but important lineage model.

A data consumer can understand where a faculty record originated rather than treating the database as an unexplained black box.

API Layer

Get all faculty

GET /faculty

Get a faculty member

GET /faculty/1

Filter by category

GET /faculty/category/Regular%20Faculty

The exact JSON field names should remain synchronized with the Pydantic schema implemented in api/schema.py.

Example API Requests

cURL

curl http://127.0.0.1:8000/faculty
curl http://127.0.0.1:8000/faculty/1
curl "http://127.0.0.1:8000/faculty/category/Regular%20Faculty"

Swagger

Open:

http://127.0.0.1:8000/docs

Swagger provides an interactive interface for inspecting and testing the available API operations.

Repository Structure

Faculty_Finder/
│
├── api/
│   ├── main.py
│   └── schema.py
│
├── ingestion/
│   ├── discover_urls.py
│   ├── http_client.py
│   └── scrape_faculty.py
│
├── cleaning/
│   ├── __init__.py
│   └── clean_faculty_records.py
│
├── storage/
│   ├── db.py
│   ├── schema.sql
│   ├── init_db.py
│   ├── insert_faculty.py
│   ├── fetch_faculty.py
│   └── faculty.db
│
├── pipeline/
│   ├── step_1_ingestion.py
│   ├── step_2_storage.py
│   └── step_3_cleaning.py
│
├── rag/
│   ├── artifacts/
│   ├── vector_store/
│   ├── step_1_text_construction.py
│   ├── step_2_bm25_retrieval.py
│   ├── step_3_semantic_index.py
│   ├── step_4_semantic_retrieval.py
│   ├── step_5_hybrid_retrieval.py
│   ├── step_6_llm_explainability.py
│   └── utils.py
│
├── data/
│   ├── raw/
│   │   └── raw_faculty_data.csv
│   └── processed/
│       └── clean_faculty_data.csv
│
├── data_analysis/
│   ├── category_statistics.py
│   ├── data_overview.py
│   ├── data_quality_checks.py
│   ├── load_data.py
│   ├── missing_value_statistics.py
│   ├── statistics.py
│   ├── summary_report.py
│   └── text_statistics.py
│
├── frontend/
│   ├── src/
│   │   ├── assets/
│   │   ├── components/
│   │   ├── pages/
│   │   ├── services/
│   │   ├── data/
│   │   ├── App.tsx
│   │   └── main.tsx
│   ├── index.html
│   ├── package.json
│   ├── tailwind.config.js
│   ├── tsconfig.json
│   └── vite.config.ts
│
├── logs/
│   └── llm_usage.md
│
├── run_pipeline.py
├── requirements.txt
├── LICENSE
└── README.md

Installation

Prerequisites

  • Python 3.10+
  • Node.js 18+
  • Git

If using optional downstream retrieval/LLM modules, configure the corresponding model/API dependencies separately.

1. Clone the Repository

git clone https://github.com/harshpatel080503/Faculty_Finder.git
cd Faculty_Finder

2. Create the Python Environment

Windows

python -m venv venv
.\venv\Scripts\activate

Linux / macOS

python3 -m venv venv
source venv/bin/activate

3. Install Dependencies

pip install -r requirements.txt

Running the Pipeline

Complete Pipeline

Initialize the database:

python storage/init_db.py

Then run:

python run_pipeline.py

End-to-end flow:

Ingestion
   ↓
Storage
   ↓
Cleaning

Running Individual Pipeline Stages

1. Data Ingestion

python pipeline/step_1_ingestion.py

Purpose:

  • Discover faculty directories
  • Discover profile URLs
  • Fetch profile pages
  • Extract raw profile information

2. Data Storage

python pipeline/step_2_storage.py

Purpose:

  • Initialize / populate SQLite
  • Persist structured records
  • Maintain database consistency

3. Data Cleaning

python pipeline/step_3_cleaning.py

Purpose:

  • Clean HTML-derived fields
  • Normalize text
  • Handle missing/irregular values
  • Produce downstream-friendly data

Running the API

Start FastAPI:

uvicorn api.main:app --reload

Open:

http://127.0.0.1:8000/docs

Then test:

GET /faculty
GET /faculty/{id}
GET /faculty/category/{category}

Frontend

The repository also contains a Vite/React frontend.

From the frontend directory:

cd frontend
npm install
npm run dev

The frontend contains pages/components for:

  • Home
  • Search Results
  • Faculty Detail
  • Faculty List
  • Featured Faculty
  • Data Insights
  • How It Works
  • About

The frontend acts as a presentation layer over the faculty information and API-backed data.

Design Decisions

Why Preserve Raw HTML?

Because extraction logic changes.

Keeping the original source allows the pipeline to be reprocessed without repeatedly depending on the upstream website.

Why SQLite?

SQLite is appropriate for the current project because:

  • Dataset scale is manageable.
  • The database is portable.
  • Local development is simple.
  • API integration is straightforward.
  • No distributed database infrastructure is required.

Why Separate Ingestion, Cleaning, Storage and Serving?

Each layer has a distinct responsibility:

Ingestion
→ Get the data

Storage
→ Persist the data

Cleaning
→ Improve the data

API
→ Serve the data

This separation also makes the system easier to debug and re-run.

Why Analyze Missingness?

Because downstream search quality depends on field completeness.

A faculty record with no research text cannot provide research-based relevance as reliably as a complete profile.

Therefore:

Data quality is part of application quality.

What This Project Demonstrates

Data Engineering

  • Web data acquisition
  • ETL pipeline design
  • Semi-structured data handling
  • Data cleaning
  • Persistence
  • Reproducible processing

Data Quality

  • Missing-value profiling
  • Field completeness analysis
  • Text statistics
  • Category distributions
  • Source-data inspection

Backend Engineering

  • FastAPI
  • REST APIs
  • Pydantic schemas
  • Database integration

Reliability

  • URL normalization
  • Retry and backoff
  • Missing-field handling
  • Safe reruns
  • Source preservation

Full-Stack Integration

Web Data
   ↓
Data Pipeline
   ↓
SQLite
   ↓
FastAPI
   ↓
Frontend

Current Scope vs Future Opportunities

The repository contains a rag/ module with retrieval/LLM-oriented scripts.

For portfolio accuracy, the project should be understood primarily as the data engineering + API foundation, while advanced semantic discovery should only be described as a validated current capability when the corresponding module has been executed and evaluated as part of the active implementation.

Current Core Capability

Web Crawling
    ↓
Profile Extraction
    ↓
Cleaning
    ↓
SQLite
    ↓
FastAPI
    ↓
Frontend

Future Opportunity

Clean Faculty Data
       ↓
Hybrid Search
   ├── BM25
   └── Semantic Retrieval
       ↓
Research Intent Understanding
       ↓
Faculty Ranking
       ↓
Evidence-backed Recommendations

Potential future capabilities include:

  • Semantic faculty search
  • Research-interest matching
  • Hybrid lexical + semantic retrieval
  • Faculty similarity graphs
  • Research collaboration discovery
  • Explainable recommendations
  • Publication-aware ranking

These should be considered future opportunities unless explicitly validated in the active implementation.

Limitations

1. Source Dependence

The pipeline depends on the structure and availability of university webpages.

Changes to upstream HTML can require extractor maintenance.

2. Missing Data

Several faculty fields contain substantial missingness, particularly research, biography, teaching, and publication information.

3. Website Dynamics

Dynamic or JavaScript-dependent content may require future crawler adaptation.

4. SQLite Scale

SQLite is appropriate for the current project scope but is not positioned as a distributed production data platform.

5. Search Intelligence

Advanced semantic search and recommendation capabilities should not be treated as production guarantees without validation of their respective modules.

Conclusion

FacultyGraph demonstrates how an unstructured web source can be transformed into a usable data product:

Messy Web Data
      ↓
Reliable Ingestion
      ↓
Raw Data Preservation
      ↓
Cleaning & Transformation
      ↓
Data Quality Analysis
      ↓
Structured Storage
      ↓
REST API
      ↓
Application / Discovery Layer

The core achievement is not simply scraping faculty pages.

It is building a reproducible, inspectable, queryable data pipeline that converts fragmented academic information into a structured foundation for search and discovery applications.

FacultyGraph — from university webpages to a queryable academic expertise dataset.

Team

Infraglyph

Member Contribution
Urvi Kava Data ingestion, crawling, retrieval design, system integration
Patel Harsh Satishkumar Data cleaning, storage, API layer, documentation, output validation, frontend alignment

FacultyGraph

University Expertise Discovery & Research Search Engine

Collect. Clean. Structure. Serve. Discover.

About

Web-to-data pipeline for discovering, structuring, storing and serving university faculty expertise through SQLite and FastAPI.

Topics

data-engineeringdata-qualityetlfastapipythonsqliteweb-scraping

Resources

Readme
License
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages