Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,9 +44,9 @@ Perfect for music researchers, data scientists, developers, and music enthusiast
| Service | Purpose | Key Technologies |
| ------------------------------------------------------------- | -------------------------------------- | ------------------------------------------------- |
| **[🔐](docs/emoji-guide.md#service-identifiers) API** | User accounts and JWT authentication | `FastAPI`, `psycopg3`, `redis`, Discogs OAuth 1.0 |
| **[🗂️](docs/emoji-guide.md#service-identifiers) Curator** | Discogs collection & wantlist sync | `FastAPI`, `psycopg3`, `neo4j-driver` |
| **[🗂️](docs/emoji-guide.md#service-identifiers) Curator** | Background collection & wantlist sync | `FastAPI`, `psycopg3`, `neo4j-driver` |
| **[📊](docs/emoji-guide.md#service-identifiers) Dashboard** | Real-time system monitoring | `FastAPI`, WebSocket, reactive UI |
| **[🔍](docs/emoji-guide.md#service-identifiers) Explore** | Interactive graph exploration & trends | `FastAPI`, `D3.js`, `Plotly.js`, Neo4j |
| **[🔍](docs/emoji-guide.md#service-identifiers) Explore** | Serves graph exploration frontend (static files) | `FastAPI`, `D3.js`, `Plotly.js` |
| **[⚡](docs/emoji-guide.md#service-identifiers) Extractor** | High-performance Rust-based extractor | `tokio`, `quick-xml`, `lapin` |
| **[🔗](docs/emoji-guide.md#service-identifiers) Graphinator** | Builds Neo4j knowledge graphs | `neo4j-driver`, graph algorithms |
| **[🔧](docs/emoji-guide.md#service-identifiers) Schema-Init** | One-shot database schema initializer | `neo4j-driver`, `psycopg3` |
Expand Down
62 changes: 40 additions & 22 deletions api/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ POSTGRES_USERNAME=discogsography
POSTGRES_PASSWORD=discogsography
POSTGRES_DATABASE=discogsography

# Redis (OAuth state storage)
# Redis (OAuth state + JTI blacklist storage)
REDIS_URL=redis://redis:6379/0

# JWT signing secret (shared with Curator)
Expand All @@ -42,6 +42,17 @@ JWT_SECRET_KEY=your-secret-key-here
# Discogs API
DISCOGS_USER_AGENT="Discogsography/1.0 +https://github.com/SimplicityGuy/discogsography"

# OAuth token encryption (Fernet symmetric key — generate with:
# python -c 'from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())')
OAUTH_ENCRYPTION_KEY=your-fernet-key-here

# Optional — CORS
CORS_ORIGINS="http://localhost:8003,http://localhost:8006" # Comma-separated allowed origins

# Optional — Snapshot settings
SNAPSHOT_TTL_DAYS=28 # Default: 28 days
SNAPSHOT_MAX_NODES=100 # Default: 100 nodes per snapshot

# Optional
JWT_EXPIRE_MINUTES=1440 # Default: 24 hours
LOG_LEVEL=INFO
Expand Down Expand Up @@ -115,11 +126,12 @@ If a user attempts to start the Discogs OAuth flow before credentials are config

### Authentication

| Method | Path | Auth Required | Description |
| ------ | -------------------- | ------------- | --------------------------- |
| POST | `/api/auth/register` | No | Register a new user account |
| POST | `/api/auth/login` | No | Login and receive JWT token |
| GET | `/api/auth/me` | Yes | Get current user details |
| Method | Path | Auth Required | Rate Limit | Description |
| ------ | -------------------- | ------------- | ---------- | ------------------------------- |
| POST | `/api/auth/register` | No | 3/min | Register a new user account |
| POST | `/api/auth/login` | No | 5/min | Login and receive JWT token |
| POST | `/api/auth/logout` | Yes | — | Revoke JWT token (JTI blacklist) |
| GET | `/api/auth/me` | Yes | — | Get current user details |

### Discogs OAuth

Expand All @@ -134,20 +146,20 @@ If a user attempts to start the Discogs OAuth flow before credentials are config

All graph query endpoints are served by the API service and consumed by the Explore frontend.

| Method | Path | Auth Required | Description |
| ------ | --------------------- | ------------- | ------------------------------------ |
| GET | `/api/autocomplete` | No | Search entities with autocomplete |
| GET | `/api/explore` | No | Get center node with category counts |
| GET | `/api/expand` | No | Expand a category node (paginated) |
| GET | `/api/node/{node_id}` | No | Get full details for a node |
| GET | `/api/trends` | No | Get time-series release counts |
| Method | Path | Auth Required | Rate Limit | Description |
| ------ | --------------------- | ------------- | ---------- | ------------------------------------ |
| GET | `/api/autocomplete` | No | 30/min | Search entities with autocomplete |
| GET | `/api/explore` | No | — | Get center node with category counts |
| GET | `/api/expand` | No | — | Expand a category node (paginated) |
| GET | `/api/node/{node_id}` | No | — | Get full details for a node |
| GET | `/api/trends` | No | — | Get time-series release counts |

### Collection Sync

| Method | Path | Auth Required | Description |
| ------ | ------------------ | ------------- | ------------------------------- |
| POST | `/api/sync` | Yes | Trigger a full Discogs sync |
| GET | `/api/sync/status` | Yes | Get sync history (last 10 jobs) |
| Method | Path | Auth Required | Rate Limit | Description |
| ------ | ------------------ | ------------- | ---------- | ------------------------------- |
| POST | `/api/sync` | Yes | 2/10min | Trigger a full Discogs sync |
| GET | `/api/sync/status` | Yes | — | Get sync history (last 10 jobs) |

### User Collection

Expand All @@ -167,7 +179,7 @@ Save and restore graph exploration states as shareable URLs.

| Method | Path | Auth Required | Description |
| ------ | ----------------------- | ------------- | --------------------------- |
| POST | `/api/snapshot` | No | Save current graph snapshot |
| POST | `/api/snapshot` | Yes | Save current graph snapshot |
| GET | `/api/snapshot/{token}` | No | Restore a saved snapshot |

### Health
Expand Down Expand Up @@ -221,10 +233,16 @@ The API service uses the following tables (created by schema-init):

## Security

- Passwords hashed with PBKDF2-SHA256 (100,000 iterations, random 32-byte salt)
- JWT signatures use `hmac.compare_digest` for constant-time comparison
- OAuth state stored in Redis with TTL to prevent replay attacks
- All endpoints run as non-root container user (UID 1000)
- **Passwords**: PBKDF2-SHA256 (100,000 iterations, random 32-byte salt)
- **Constant-time auth**: Login and registration use constant-time comparison to prevent user enumeration via timing attacks
- **Blind registration**: Duplicate email registration returns the same 201 response to prevent enumeration
- **JWT revocation**: Logout blacklists the JWT's `jti` claim in Redis with TTL matching the token expiry
- **OAuth tokens encrypted at rest**: Discogs OAuth access tokens are encrypted with Fernet symmetric encryption before database storage (`OAUTH_ENCRYPTION_KEY`)
- **Rate limiting**: register (3/min), login (5/min), sync (2/10min), autocomplete (30/min) via slowapi; per-user sync cooldown (600s) in Redis
- **Security response headers**: `X-Content-Type-Options`, `X-Frame-Options`, `Referrer-Policy`, `Permissions-Policy`
- **CORS**: Configurable via `CORS_ORIGINS` env var (disabled by default)
- **Snapshots require auth**: `POST /api/snapshot` requires a valid JWT
- **Container**: All endpoints run as non-root container user (UID 1000)

## Monitoring

Expand Down
125 changes: 99 additions & 26 deletions api/api.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,18 +10,23 @@
import json
import os
from pathlib import Path
import secrets
from typing import Annotated, Any

from fastapi import Depends, FastAPI, HTTPException, status
from fastapi import Depends, FastAPI, HTTPException, Request, status
from fastapi.middleware.cors import CORSMiddleware
from fastapi.responses import ORJSONResponse
from fastapi.security import HTTPAuthorizationCredentials, HTTPBearer
from psycopg.rows import dict_row
from pydantic import BaseModel
import redis.asyncio as aioredis
from slowapi import _rate_limit_exceeded_handler
from slowapi.errors import RateLimitExceeded
import structlog
import uvicorn

from api.auth import encrypt_oauth_token
from api.limiter import limiter
from api.models import LoginRequest, RegisterRequest
import api.routers.explore as _explore_router
import api.routers.snapshot as _snapshot_router
Expand Down Expand Up @@ -106,6 +111,11 @@ def _verify_password(plain_password: str, hashed_password: str) -> bool:
return False


# Pre-computed dummy hash for timing attack protection in login (H4)
# Ensures user-not-found path takes the same time as wrong-password path
_DUMMY_HASH: str = _hash_password("__dummy_password_for_timing__")


def _create_access_token(user_id: str, email: str) -> tuple[str, int]:
"""Create a HS256 JWT access token. Returns (token, expires_in_seconds)."""
if _config is None:
Expand All @@ -118,6 +128,7 @@ def _create_access_token(user_id: str, email: str) -> tuple[str, int]:
"email": email,
"exp": int(expire.timestamp()),
"iat": int(datetime.now(UTC).timestamp()),
"jti": secrets.token_hex(16),
}

header = _b64url_encode(json.dumps({"alg": "HS256", "typ": "JWT"}, separators=(",", ":")).encode())
Expand Down Expand Up @@ -169,6 +180,16 @@ async def _get_current_user(
status_code=status.HTTP_401_UNAUTHORIZED,
detail="Invalid token",
)
# Check jti blacklist (revoked tokens via logout)
jti: str | None = payload.get("jti")
if jti and _redis:
revoked = await _redis.get(f"revoked:jti:{jti}")
if revoked:
raise HTTPException(
status_code=status.HTTP_401_UNAUTHORIZED,
detail="Token has been revoked",
headers={"WWW-Authenticate": "Bearer"},
)
return payload
except ValueError as exc:
raise HTTPException(
Expand Down Expand Up @@ -207,22 +228,28 @@ async def lifespan(_app: FastAPI) -> AsyncGenerator[None]: # pragma: no cover
await _pool.initialize()
logger.info("💾 Database pool initialized")

# Initialize Redis for OAuth state storage
# Initialize Redis for OAuth state storage and token blacklist
_redis = await aioredis.from_url(_config.redis_url, decode_responses=True)
logger.info("🔴 Redis connected", url=_config.redis_url)
redis_host = _config.redis_url.split("@")[-1] if "@" in _config.redis_url else _config.redis_url.split("://")[-1]
logger.info("🔴 Redis connected", host=redis_host)

if _config.neo4j_address and _config.neo4j_username and _config.neo4j_password:
_neo4j = AsyncResilientNeo4jDriver(
uri=_config.neo4j_address,
auth=(_config.neo4j_username, _config.neo4j_password),
max_retries=5,
encrypted=False,
encrypted=False, # M3: Set encrypted=True in production with TLS-enabled Neo4j
)
logger.info("🔗 Neo4j driver initialized")
jwt_secret_for_neo4j = _config.jwt_secret_key if _config.neo4j_address else None
_sync_router.configure(_pool, _neo4j, _config, _running_syncs)
_sync_router.configure(_pool, _neo4j, _config, _running_syncs, _redis)
_explore_router.configure(_neo4j, jwt_secret_for_neo4j)
_user_router.configure(_neo4j, jwt_secret_for_neo4j)
_snapshot_router.configure(
jwt_secret=_config.jwt_secret_key,
ttl_days=_config.snapshot_ttl_days,
max_nodes=_config.snapshot_max_nodes,
)
logger.info("✅ API service ready", port=API_PORT)

yield
Expand All @@ -242,6 +269,10 @@ async def lifespan(_app: FastAPI) -> AsyncGenerator[None]: # pragma: no cover
logger.info("✅ API service stopped")


# Read CORS origins at module load time (config not available yet at this point)
_cors_origins_raw = os.environ.get("CORS_ORIGINS", "")
_cors_origins = [o.strip() for o in _cors_origins_raw.split(",") if o.strip()] if _cors_origins_raw else None

app = FastAPI(
title="Discogsography API",
version="0.1.0",
Expand All @@ -250,13 +281,28 @@ async def lifespan(_app: FastAPI) -> AsyncGenerator[None]: # pragma: no cover
lifespan=lifespan,
)

app.state.limiter = limiter
app.add_exception_handler(RateLimitExceeded, _rate_limit_exceeded_handler) # type: ignore[arg-type]

app.add_middleware(
CORSMiddleware,
allow_origins=["*"],
allow_origins=_cors_origins or ["http://localhost:3000", "http://localhost:8003"],
allow_methods=["*"],
allow_headers=["*"],
)


@app.middleware("http")
async def security_headers(request: Request, call_next: Any) -> Any:
"""Add security headers to all responses."""
response = await call_next(request)
response.headers["X-Content-Type-Options"] = "nosniff"
response.headers["X-Frame-Options"] = "DENY"
response.headers["Referrer-Policy"] = "strict-origin-when-cross-origin"
response.headers["Permissions-Policy"] = "geolocation=(), microphone=(), camera=()"
return response


app.include_router(_sync_router.router)
app.include_router(_explore_router.router)
app.include_router(_snapshot_router.router)
Expand All @@ -270,15 +316,16 @@ async def health_check() -> ORJSONResponse:


@app.post("/api/auth/register", status_code=status.HTTP_201_CREATED)
async def register(request: RegisterRequest) -> ORJSONResponse:
@limiter.limit("3/minute")
async def register(request: Request, body: RegisterRequest) -> ORJSONResponse: # noqa: ARG001
"""Register a new user account."""
if _pool is None:
raise HTTPException(
status_code=status.HTTP_503_SERVICE_UNAVAILABLE,
detail="Service not ready",
)

hashed_password = _hash_password(request.password)
hashed_password = _hash_password(body.password)

try:
async with _pool.connection() as conn, conn.cursor(row_factory=dict_row) as cur:
Expand All @@ -288,16 +335,18 @@ async def register(request: RegisterRequest) -> ORJSONResponse:
VALUES (%s, %s)
RETURNING id, email, is_active, created_at
""",
(request.email, hashed_password),
(body.email, hashed_password),
)
row = await cur.fetchone()
except Exception as exc:
exc_str = str(exc).lower()
if "unique" in exc_str or "duplicate" in exc_str:
raise HTTPException(
status_code=status.HTTP_409_CONFLICT,
detail="Email address already registered",
) from exc
# L1: Return same response for duplicate email to prevent user enumeration
logger.info("i Registration attempt for existing email (blind)")
return ORJSONResponse(
content={"message": "Registration processed"},
status_code=status.HTTP_201_CREATED,
)
logger.error("❌ Registration failed", error=str(exc))
raise HTTPException(
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
Expand All @@ -310,20 +359,16 @@ async def register(request: RegisterRequest) -> ORJSONResponse:
detail="Registration failed",
)

logger.info("✅ User registered", email=request.email)
logger.info("✅ User registered", email=body.email)
return ORJSONResponse(
content={
"id": str(row["id"]),
"email": row["email"],
"is_active": row["is_active"],
"created_at": row["created_at"].isoformat(),
},
content={"message": "Registration processed"},
status_code=status.HTTP_201_CREATED,
)


@app.post("/api/auth/login")
async def login(request: LoginRequest) -> ORJSONResponse:
@limiter.limit("5/minute")
async def login(request: Request, body: LoginRequest) -> ORJSONResponse: # noqa: ARG001
"""Authenticate and receive a JWT access token."""
if _pool is None or _config is None:
raise HTTPException(
Expand All @@ -334,19 +379,28 @@ async def login(request: LoginRequest) -> ORJSONResponse:
async with _pool.connection() as conn, conn.cursor(row_factory=dict_row) as cur:
await cur.execute(
"SELECT id, email, hashed_password, is_active FROM users WHERE email = %s",
(request.email,),
(body.email,),
)
user = await cur.fetchone()

if user is None or not user["is_active"] or not _verify_password(request.password, user["hashed_password"]):
# H4: Constant-time check to prevent user enumeration via timing
if user is None:
_verify_password(body.password, _DUMMY_HASH) # consume same time as real verify
raise HTTPException(
status_code=status.HTTP_401_UNAUTHORIZED,
detail="Incorrect email or password",
headers={"WWW-Authenticate": "Bearer"},
)

if not user["is_active"] or not _verify_password(body.password, user["hashed_password"]):
raise HTTPException(
status_code=status.HTTP_401_UNAUTHORIZED,
detail="Incorrect email or password",
headers={"WWW-Authenticate": "Bearer"},
)

access_token, expires_in = _create_access_token(str(user["id"]), user["email"])
logger.info("✅ User logged in", email=request.email)
logger.info("✅ User logged in", email=body.email)

return ORJSONResponse(
content={
Expand All @@ -357,6 +411,21 @@ async def login(request: LoginRequest) -> ORJSONResponse:
)


@app.post("/api/auth/logout")
async def logout(
current_user: Annotated[dict[str, Any], Depends(_get_current_user)],
) -> ORJSONResponse:
"""Logout and revoke the current JWT token."""
if _redis:
jti: str | None = current_user.get("jti")
exp: int | None = current_user.get("exp")
if jti:
now = int(datetime.now(UTC).timestamp())
ttl = max((exp - now), 60) if exp else 3600
await _redis.setex(f"revoked:jti:{jti}", ttl, "1")
return ORJSONResponse(content={"logged_out": True})


@app.get("/api/auth/me")
async def get_me(
current_user: Annotated[dict[str, Any], Depends(_get_current_user)],
Expand Down Expand Up @@ -546,8 +615,12 @@ async def verify_discogs(
""",
(
user_id,
access_data["oauth_token"],
access_data["oauth_token_secret"],
encrypt_oauth_token(access_data["oauth_token"], _config.oauth_encryption_key)
if _config.oauth_encryption_key
else access_data["oauth_token"],
encrypt_oauth_token(access_data["oauth_token_secret"], _config.oauth_encryption_key)
if _config.oauth_encryption_key
else access_data["oauth_token_secret"],
discogs_username,
discogs_user_id,
),
Expand Down
Loading
Loading