3D Face Reconstruction — Project Plan

📌 Objective

Build a pipeline that transforms smartphone-captured 2D images (photos or video) into a high-quality 3D face model suitable for 3D printing and game-ready use — targeting quality comparable to NBA 2K scanned players.

---

🎯 Goals

#	Goal	Success Criteria
1	Capture face via smartphone	Protocol yields ≥ 30 sharp, calibrated photos
2	Run AI/photogrammetry pipeline	< 30 min processing time on local hardware
3	Output print-ready 3D model	Clean mesh, < 500k polys, UV-mapped
4	Match reference quality	Passes visual comparison against Polycam/Luma output

---

🗂️ Repository Structure

3d-face-project/
├── README.md
├── .cursorrules                  # Cursor AI context file (see below)
├── capture/
│   ├── protocol.md               # Photo capture instructions
│   └── calibration/              # Camera calibration assets
├── pipeline/
│   ├── 01_preprocess.py          # Image validation, resize, EXIF strip
│   ├── 02_reconstruct.py         # SfM / NeRF / Gaussian Splatting runner
│   ├── 03_postprocess.py         # Mesh cleanup, decimation, UV unwrap
│   └── 04_export.py              # Export to .obj / .glb / .stl
├── models/
│   └── .gitkeep                  # Output 3D models go here
├── evaluation/
│   ├── compare.py                # Metric comparison vs. reference scans
│   └── assets/                   # Reference meshes for quality benchmarking
├── docker/
│   └── Dockerfile                # Reproducible environment
├── requirements.txt
└── config.yaml                   # Pipeline configuration

---

🔬 Technical Architecture

Data Inputs (Priority Order)

Sensor	Availability	Weight in Pipeline
RGB Camera	Universal	Primary
LiDAR (iPhone Pro / iPad Pro)	Optional	Depth prior
ToF (Android flagship)	Optional	Depth prior
Infrared / TrueDepth	Optional	Landmark anchor
Gyroscope / IMU	Universal	Camera pose hint

Processing Pipeline

Input Images
    │
    ▼
[01] Preprocessing
    • Validate sharpness (Laplacian variance)
    • Normalize exposure
    • Face detection + crop (MediaPipe)
    │
    ▼
[02] Reconstruction Engine (selectable via config.yaml)
    ├── Option A: SfM — Meshroom / COLMAP (best for 30+ photos)
    ├── Option B: Monocular AI — TripoSR / One-2-3-45 (single photo fallback)
    └── Option C: Neural — 3DGS / DreamGaussian (best perceptual quality)
    │
    ▼
[03] Post-Processing (Blender / Open3D)
    • Mesh decimation to target poly count
    • Hole filling
    • Smoothing (bilateral filter)
    • UV unwrap + texture bake
    │
    ▼
[04] Export
    .obj (universal) | .glb (game/web) | .stl (3D printing)

AI Model Options

Model	Input	Output	Quality	Speed	Notes
TripoSR	1 image	Mesh	Medium	Fast	Best single-shot option
PiFuHD	1-2 images	Mesh	High	Medium	Best for clothing/body
One-2-3-45	1 image	Multi-view → Mesh	Medium-High	Slow	Good geometry
DreamGaussian	1 image	3DGS → Mesh	High	Medium	Best texture
COLMAP + MVSNet	20+ images	Point Cloud → Mesh	Very High	Slow	Best overall quality

Recommended default: COLMAP SfM → OpenMVS → Blender post-process

---

🏗️ Implementation Phases

Phase 1 — Environment & Baselines (Week 1–2)

• Set up Docker environment with COLMAP, Open3D, Blender CLI
• Run Meshroom on reference face dataset (e.g., FaceScape)
• Run TripoSR on same reference images
• Run Luma AI / Polycam on smartphone capture for baseline
• Document quality gaps vs. NBA 2K reference

Phase 2 — Capture Protocol (Week 3)

• Write capture/protocol.md with lighting, distance, angle specifications
• Develop capture/calibration/ assets for consistent smartphone setup
• Implement 01_preprocess.py:
  • Sharpness filtering (cv2.Laplacian)
  • Face crop with MediaPipe
  • EXIF metadata extraction for camera intrinsics
• Test protocol with 3 subjects, ≥ 30 photos each

Phase 3 — Pipeline Implementation (Week 4–6)

• Implement 02_reconstruct.py with engine selection via config.yaml
• COLMAP integration (feature extraction → matching → SfM → MVS)
• TripoSR integration as single-image fallback
• Implement 03_postprocess.py:
  • Mesh decimation (target: 100k–500k tris)
  • Blender Python API for UV unwrap + bake
• Implement 04_export.py for .obj / .glb / .stl
• End-to-end test on Phase 2 captures

Phase 4 — Quality & Iteration (Week 7)

• Implement evaluation/compare.py (Chamfer distance, SSIM on textures)
• 3D print one output, assess surface quality
• Tune pipeline parameters based on print feedback
• Document final recommended settings in config.yaml

---

⚙️ Configuration (config.yaml)

reconstruction:
  engine: colmap           # colmap | triposr | dreamgaussian
  num_images_min: 20
  num_images_max: 80

postprocess:
  target_poly_count: 200000
  smooth_iterations: 3
  fill_holes: true

export:
  formats: [obj, glb, stl]
  output_dir: ./models/

capture:
  recommended_distance_cm: 40
  recommended_images: 36
  lighting: natural_diffuse  # natural_diffuse | ring_light | studio

---

📦 Dependencies (requirements.txt)

# Core
opencv-python>=4.8
numpy>=1.24
open3d>=0.17
trimesh>=4.0
mediapipe>=0.10

# AI Models
torch>=2.0
torchvision
huggingface-hub

# Utilities
Pillow
tqdm
pyyaml
pycolmap          # Python bindings for COLMAP
bpy               # Blender Python (install separately via Blender's Python)

---

🛠️ Key Tools

Tool	Purpose	License
COLMAP	SfM + MVS reconstruction	BSD
Meshroom	GUI SfM pipeline	MPL-2.0
TripoSR	Single-image AI reconstruction	MIT
PiFuHD	AI implicit surface	BSD
DreamGaussian	Gaussian splatting reconstruction	MIT
Open3D	Point cloud & mesh processing	MIT
Blender (CLI)	Post-processing, UV, export	GPL
MediaPipe	Face detection & landmarks	Apache 2.0

---

📊 Quality Benchmarks

Level	Poly Count	Texture Res	Capture Method	Reference
Consumer	50k	2K	Single phone photo	TripoSR
Prosumer	200k	4K	30 phone photos + SfM	Polycam, Meshroom
Target	500k	4K–8K	36+ photos + LiDAR	Luma AI
Studio	2M+	8K+	Multi-camera rig + LiDAR	NBA 2K

---

🔗 Reference & Research

• FaceScape Dataset — benchmark face scans
• COLMAP Docs
• TripoSR (Stability AI)
• PiFuHD
• DreamGaussian
• Open3D Tutorials

---

.cursorrules (place in project root)

# 3D Face Reconstruction Project — Cursor AI Context

## Stack
- Python 3.11
- COLMAP (CLI via subprocess), Open3D, Trimesh, Blender Python API
- PyTorch for AI model inference
- MediaPipe for face detection

## Conventions
- All pipeline scripts accept --input and --output CLI args via argparse
- Config is loaded from config.yaml at project root
- Intermediate outputs go to ./tmp/, final outputs go to ./models/
- Log with Python logging module (not print)
- Type hints required on all function signatures

## Domain Knowledge
- 3D meshes are represented as (vertices, faces) numpy arrays unless using Open3D objects
- Coordinate system: Y-up, right-handed
- Target output: watertight, manifold mesh for 3D printing

## Do Not
- Do not use closed-source APIs (no Luma AI API, no Polycam API)
- Do not hardcode file paths; use pathlib.Path throughout
- Do not commit model weights; use huggingface-hub downloads

---

🚀 Quick Start

# 1. Clone and install
git clone https://github.com/yourname/3d-face-project
cd 3d-face-project
pip install -r requirements.txt

# 2. Capture photos following capture/protocol.md
# Place images in ./input/

# 3. Run pipeline
python pipeline/01_preprocess.py --input ./input/ --output ./tmp/preprocessed/
python pipeline/02_reconstruct.py --input ./tmp/preprocessed/ --output ./tmp/mesh/
python pipeline/03_postprocess.py --input ./tmp/mesh/ --output ./tmp/clean/
python pipeline/04_export.py --input ./tmp/clean/ --output ./models/