PointCloud Converter

A self-contained, production-oriented system that accepts .laz / .las LiDAR files, converts them into a web-optimized Potree 2 octree, and renders them in an interactive React point cloud viewer.

Everything runs with a single command:

docker compose up --build

No manual installation of PDAL, PotreeConverter, PostgreSQL, or Redis is required on the host.

---

Table of contents

• Features
• Architecture
• Technology stack
• Prerequisites
• Quick start
• Step-by-step usage
• Docker services
• Project structure
• Processing pipeline
• API reference
• Configuration
• Local development
• Troubleshooting
• Security

---

Features

Area	Capabilities
Upload	Drag-and-drop, resumable chunked uploads (multi-GB), progress tracking, cancellation
Validation	Extension/MIME checks, LAS signature, PDAL integrity, point-count limits
Processing	Async Celery jobs, PDAL metadata extraction, PotreeConverter 2 octree generation
Viewer	Orbit/zoom/pan, elevation & classification coloring, clipping, measurement, FPS overlay
Ops	Health checks, structured logging, WebSocket job progress, OpenAPI docs

---

Architecture

flowchart TB
    subgraph Browser
        UI[React SPA]
    end

    subgraph Docker["Docker Compose"]
        NGX[Nginx :8080]
        API[FastAPI API]
        WRK[Celery Worker]
        PG[(PostgreSQL)]
        RD[(Redis)]
        FE[Frontend static]
        VOL[(storage volume)]
    end

    UI --> NGX
    NGX --> FE
    NGX --> API
    NGX -->|/tiles range requests| VOL
  API --> PG
    API --> RD
    API --> VOL
    WRK --> PG
    WRK --> RD
    WRK -->|PDAL + PotreeConverter| VOL
    WRK -->|progress| RD
    API -->|WebSocket| RD

Loading

Request flow (upload → view):

1. User uploads .laz via the web UI (chunked or single-shot).
2. API validates the file and stores it under /storage/uploads/.
3. A Celery task is queued in Redis.
4. The worker runs PDAL (metadata, CRS, validation) then PotreeConverter 2 (octree).
5. Output is written to /storage/processed/{job_id}/potree/.
6. Nginx serves tiles with HTTP range requests (required for Potree 2).
7. The React viewer loads metadata.json via @pnext/three-loader (Potree v2) with LOD streaming.

---

Technology stack

Layer	Technologies
API	Python 3.12, FastAPI, SQLAlchemy, Alembic, Pydantic, SlowAPI
Worker	Celery, PDAL (conda-forge), PotreeConverter 2.1.2
Database	PostgreSQL 16
Queue	Redis 7
Proxy	Nginx (upload limits, rate limiting, tile caching, range requests)
Frontend	React 18, TypeScript, Vite, Tailwind CSS, Zustand
3D viewer	Three.js, React Three Fiber, @pnext/three-loader (Potree 2)
Containers	Docker Compose, multi-stage Dockerfiles

Rendering format: Potree 2 octree (metadata.json, hierarchy.bin, octree.bin) — chosen for proven LOD, frustum culling, and streaming at 10M–500M+ points.

---

Prerequisites

• Docker Desktop (or Docker Engine + Compose v2) — must be running
• 8 GB+ RAM recommended (worker + PDAL + PotreeConverter are memory-heavy)
• Disk space for LiDAR uploads and processed tiles (multi-GB per file is normal)

Apple Silicon (M1/M2/M3): The worker image uses platform: linux/amd64 because PotreeConverter only ships x86_64 Linux binaries. Docker runs this under emulation (slower but functional).

---

Quick start

1. Clone and configure

cd PointCloudConverter
cp .env.example .env

Edit .env if needed (defaults work for local use). Change SECRET_KEY before any public deployment.

2. Build and start all services

docker compose up --build

The first build often takes 10–20 minutes (downloads PDAL via conda-forge, PotreeConverter binary, npm build).

Wait until logs show:

• api — Application startup complete
• worker — celery@... ready
• nginx — healthy

3. Open the application

URL	Description
http://localhost:8080	Web UI (upload, jobs, viewer)
http://localhost:8080/api/docs	Swagger / OpenAPI
http://localhost:8080/health	Health check (API + Postgres + Redis)

4. Run in the background (optional)

docker compose up --build -d

5. Stop services

docker compose down

Remove all data (database + uploads + processed clouds):

docker compose down -v

---

Step-by-step usage

Upload a LAZ file

1. Open http://localhost:8080
2. Drag and drop a .laz or .las file onto the upload area (or click to browse).
3. Watch upload progress on the same page.
4. When upload completes, the job is queued for processing (WebSocket updates).

Tips:

• Start with a smaller file (50–200 MB) to verify the pipeline before multi-GB uploads.
• Large files use 16 MB chunks by default (CHUNK_SIZE_BYTES in .env).

Monitor jobs

1. Go to Jobs in the navigation bar.
2. Status flows: uploading → queued → processing → completed (or failed).
3. The list auto-refreshes every 5 seconds.

View a point cloud

1. On the Jobs page, click View on a completed job.
2. Use the sidebar controls:
   • Point budget — max points rendered (lower if the browser is slow)
   • Point size — visual size of points
   • Color mode — Elevation, Classification, or RGB (if available)
   • Elevation clipping — slice by height
   • Measurement tool — click two points for distance
   • Reset camera — reframe the cloud

Viewer tips:

• Use Elevation or Classification if the file has no RGB.
• If Chrome shows “Aw, Snap!”, lower the point budget to 500k, close other tabs, hard-refresh (Cmd+Shift+R), and ensure you rebuilt the frontend after the latest fixes.

Delete a job

On the Jobs page, click Delete to remove the job record and associated files from storage.

---

Docker services

Service	Image / build	Role	Host port
nginx	docker/nginx	Reverse proxy, static UI, API, WebSocket, /tiles	8080 → 80
api	docker/api	REST API, WebSocket, Alembic migrations	internal
worker	docker/worker	Celery + PDAL + PotreeConverter	internal
frontend	docker/frontend	Built React SPA (served via nginx)	internal
postgres	postgres:16-alpine	Jobs, metadata, logs	internal
redis	redis:7-alpine	Celery broker + progress pub/sub	internal

Volumes:

Volume	Purpose
postgres_data	Database persistence
redis_data	Redis AOF persistence
storage_data	Uploads, processed Potree tiles, metadata JSON

Useful commands:

# Service status
docker compose ps

# Follow logs
docker compose logs -f api worker nginx

# Rebuild one service after code changes
docker compose build worker
docker compose up -d worker

# Rebuild frontend only
docker compose build frontend && docker compose up -d frontend nginx

---

Project structure

PointCloudConverter/
├── docker-compose.yml          # Orchestrates all services
├── .env.example                # Environment template
├── README.md
│
├── docker/
│   ├── api/                    # API Dockerfile + entrypoint (migrations, uvicorn)
│   ├── worker/                 # Worker Dockerfile (Ubuntu 24.04, PDAL, PotreeConverter)
│   ├── frontend/               # Frontend build + nginx SPA config
│   └── nginx/                  # Reverse proxy config
│
├── backend/
│   ├── pyproject.toml
│   ├── alembic/                # Database migrations
│   └── app/
│       ├── main.py             # FastAPI application
│       ├── config.py
│       ├── database.py
│       ├── models/             # Upload, metadata, processing logs
│       ├── schemas/              # Pydantic request/response models
│       ├── api/routes/           # upload, jobs, health
│       ├── services/             # storage, validation, upload, progress
│       ├── worker/               # Celery app + tasks
│       │   └── pipeline/       # pdal_metadata.py, potree_convert.py
│       └── websocket/            # Job progress WebSocket
│
├── frontend/
│   ├── package.json
│   ├── vite.config.ts
│   └── src/
│       ├── pages/                # UploadPage, JobsDashboard, ViewerPage
│       ├── components/viewer/    # Potree / R3F viewer
│       ├── api/client.ts         # API client
│       └── store/                # Zustand state
│
└── storage/                      # Bind-mounted via Docker volume at runtime
    ├── uploads/
    ├── processed/                # {job_id}/potree/metadata.json, octree.bin, ...
    ├── metadata/
    └── logs/

---

Processing pipeline

Step	Component	Action
1	API	Receive upload (chunked or single), store as {uuid}.laz
2	API	Validate extension, size, MIME
3	API	Enqueue Celery task process_pointcloud
4	Worker	Verify LAS LASF signature
5	Worker	PDAL — full file validation + metadata (bounds, EPSG, point count, elevation, classifications)
6	Worker	PotreeConverter 2 — build octree LOD
7	Worker	Save to /storage/processed/{job_id}/potree/
8	Worker	Update PostgreSQL (status=completed, metadata row)
9	Worker	Publish progress via Redis → WebSocket
10	Nginx	Serve tiles at /tiles/{job_id}/potree/ with byte-range support
11	Browser	Stream LOD nodes via @pnext/three-loader

Retries: Failed tasks retry up to 3 times with exponential backoff (CELERY_TASK_MAX_RETRIES).

---

API reference

Base URL: http://localhost:8080

Method	Path	Description
POST	/api/upload/init	Start resumable upload → returns job_id, chunk_size
PUT	/api/upload/{job_id}/chunk/{index}	Upload one chunk (raw body)
POST	/api/upload/{job_id}/complete	Finalize upload and enqueue processing
DELETE	/api/upload/{job_id}	Cancel in-flight upload
POST	/api/upload	Single-shot upload (multipart file)
GET	/api/job/{job_id}	Get job status and progress
GET	/api/jobs	List jobs (?page=1&page_size=20)
GET	/api/metadata/{job_id}	Point cloud metadata (bounds, EPSG, counts, …)
GET	/api/pointcloud/{job_id}	Viewer manifest (tile URLs)
DELETE	/api/job/{job_id}	Delete job and files
GET	/health	Liveness + Postgres + Redis checks
WS	/ws/jobs/{job_id}	Real-time job progress (JSON messages)

Interactive documentation: http://localhost:8080/api/docs

Example — check health:

curl http://localhost:8080/health

Example — list jobs:

curl http://localhost:8080/api/jobs

---

Configuration

Copy .env.example to .env:

cp .env.example .env

Variable	Default	Description
SECRET_KEY	(change me)	App secret — must change in production
MAX_UPLOAD_BYTES	10737418240 (10 GB)	Maximum upload size
CHUNK_SIZE_BYTES	16777216 (16 MB)	Resumable upload chunk size
MAX_POINT_COUNT	2000000000	Reject files exceeding this point count
POSTGRES_USER / POSTGRES_PASSWORD / POSTGRES_DB	pointcloud / …	Database credentials
POTREE_SPACING	0.0	PotreeConverter spacing (0 = auto; increase to subsample for web)
WORKER_CONCURRENCY	1	Parallel Celery tasks per worker container
CELERY_TASK_MAX_RETRIES	3	Processing retry attempts
RATE_LIMIT_PER_MINUTE	120	API rate limit (also nginx limit_req)
CORS_ORIGINS	localhost variants	Allowed browser origins
VITE_API_BASE_URL	empty	Leave empty when using nginx on same host

---

Local development

Docker is the recommended path. For native development:

Backend

Requires Python 3.12, PostgreSQL, Redis, PDAL, and PotreeConverter on PATH.

cd backend
pip install .
# Set DATABASE_URL, REDIS_URL, etc. in .env or environment
alembic upgrade head
uvicorn app.main:app --reload --host 0.0.0.0 --port 8000

Celery worker (separate terminal):

cd backend
celery -A app.worker.celery_app worker --loglevel=INFO --concurrency=1

Frontend

cd frontend
npm install
npm run dev

Vite dev server proxies /api, /ws, and /tiles to http://localhost:8080 — run the full Docker stack (or nginx) for API and tiles.

# Terminal 1 — infrastructure + API
docker compose up postgres redis api worker nginx

# Terminal 2 — hot-reload UI
cd frontend && npm run dev

Open http://localhost:5173 (Vite) or http://localhost:8080 (Docker nginx).

---

Troubleshooting

localhost:8080 connection refused

Docker is not running, or the web stack is stopped.

docker compose ps          # nginx, api, frontend should be "Up"
docker compose up -d     # start all services

Only starting the worker does not expose the UI:

docker compose up -d       # not: docker compose up -d worker

Migration error: upload_status already exists

Database was left in a partial state from a failed first run.

docker compose down -v
docker compose up --build

Worker: celery: not found or libtbb.so.12 / PotreeConverter errors

Rebuild the worker image (Ubuntu 24.04 + libtbb + PotreeConverter 2.1.2):

docker compose build worker
docker compose up -d worker

Job failed in dashboard

Check worker logs:

docker compose logs worker --tail 100

Common causes: corrupt LAZ, unsupported point format, out-of-memory during conversion. Re-upload after fixing the file or lowering dataset size.

Viewer empty / FPS 0

• Confirm job status is completed.
• Hard-refresh the browser (Cmd+Shift+R).
• Rebuild frontend: docker compose build frontend && docker compose up -d frontend nginx
• Verify tiles: curl -I http://localhost:8080/tiles/{job_id}/potree/metadata.json

Chrome “Aw, Snap!” (tab crash)

Usually GPU/memory pressure from the 3D viewer.

1. Rebuild frontend (includes reload-loop fix).
2. Quit Chrome completely and reopen.
3. Lower Point budget to 500k in the viewer sidebar.
4. Close other heavy tabs.

Apple Silicon: slow processing

Expected — worker runs x86_64 under emulation. Processing still works; large files take longer.

---

Security

• Upload validation (extension, MIME, LAS signature, PDAL probe, point-count ceiling)
• UUID-based storage paths (no user-controlled filenames on disk)
• Rate limiting (nginx + SlowAPI)
• Security headers (CORS, X-Content-Type-Options, etc.)
• Non-root container users (API); worker runs as UID 1000
• Auth-ready API structure (JWT can be added; not enabled in v1)

Before production: change SECRET_KEY, use strong Postgres passwords, restrict CORS_ORIGINS, put TLS in front of nginx, and consider virus scanning for uploads.

---

License

Portfolio / demonstration project. See repository for license terms if applicable.