chore(docker): adopt Docker Compose runtime secrets for credential management

[SimplicityGuy]

Overview

This issue tracks adopting Docker Compose runtime secrets to replace bare environment variables for sensitive credentials. Secrets mount as in-memory files inside containers — they are never written to disk, never appear in `docker inspect` environment output, and are flushed from memory when the container stops.

How Runtime Secrets Work

Docker Compose mounts each secret as a file at `/run/secrets/<secret_name>` inside the container. They are not set as environment variables automatically (by design — env vars leak between processes and appear in `docker inspect`).

Compose Definition

```yaml

Top-level block — defines the secret sources

secrets:
jwt_secret_key:
file: ./secrets/jwt_secret_key.txt # or: environment: JWT_SECRET_KEY

services:
api:
secrets:
- jwt_secret_key # grants access; mounts at /run/secrets/jwt_secret_key
```

How Applications Read Them

There are two approaches:

Option A — `_FILE` convention (preferred)

The application is updated to check for a `_FILE` env var and read the secret from that path:

```python
import os

def get_secret(env_var: str) -> str:
file_path = os.environ.get(f"{env_var}_FILE")
if file_path:
with open(file_path) as f:
return f.read().strip()
return os.environ[env_var]

jwt_secret = get_secret("JWT_SECRET_KEY")
```

```yaml
services:
api:
environment:
JWT_SECRET_KEY_FILE: /run/secrets/jwt_secret_key
secrets:
- jwt_secret_key
```

Option B — Entrypoint script sourcing

A shell entrypoint reads the secret files and exports them before starting the process:

```sh
#!/bin/sh
set -e
[ -f /run/secrets/jwt_secret_key ] && export JWT_SECRET_KEY=$(cat /run/secrets/jwt_secret_key)
[ -f /run/secrets/postgres_password ] && export POSTGRES_PASSWORD=$(cat /run/secrets/postgres_password)
exec "$@"
```

Option A is cleaner and more secure (secrets never touch process environment), but requires changes to application code. Option B is simpler to adopt with no app changes.

---

What Needs to Change in This Project

Secrets That Should Be Migrated

Secret	Services	Current Mechanism	Generation
`jwt_secret_key`	api, curator, explore	`${JWT_SECRET_KEY}` env var	`openssl rand -hex 32`
`oauth_encryption_key`	api	`${OAUTH_ENCRYPTION_KEY}` env var	`python -c 'from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())'`
`postgres_password`	api, curator, tableinator, dashboard, postgres	`${POSTGRES_PASSWORD}` env var	`openssl rand -base64 24`
`postgres_user`	api, curator, tableinator, dashboard, postgres	`${POSTGRES_USER}` env var	static: `discogsography`
`rabbitmq_pass`	rabbitmq, dashboard	`${RABBITMQ_PASS}` env var	`openssl rand -base64 24`
`rabbitmq_user`	rabbitmq, dashboard	`${RABBITMQ_USER}` env var	static: `discogsography`
`neo4j_password`	neo4j, graphinator, dashboard, explore, curator, api	`${NEO4J_PASSWORD}` env var	`openssl rand -base64 24`

Notes:

• `oauth_encryption_key` is a Fernet symmetric key that encrypts Discogs consumer keys and OAuth tokens stored in the PostgreSQL `app_config` and `oauth_tokens` tables (introduced in security: encrypt Discogs consumer key/secret at rest #76). If unset, tokens are stored in plaintext (migration fallback). Required in production.
• `rabbitmq_pass` and `rabbitmq_user` also serve as the RabbitMQ management API credentials used by the Dashboard service (`RABBITMQ_MANAGEMENT_PASSWORD` / `RABBITMQ_MANAGEMENT_USER`).
• Neo4j `NEO4J_AUTH` is set as `/` — the entire value is constructed from `NEO4J_USER` + `NEO4J_PASSWORD`.

1. Create Secret Files (gitignored)

```
secrets/
jwt_secret_key.txt
oauth_encryption_key.txt
postgres_password.txt
postgres_user.txt
rabbitmq_pass.txt
rabbitmq_user.txt
neo4j_password.txt
```

Add to `.gitignore`:
```
secrets/
```

2. Update `docker-compose.prod.yml`

```yaml
secrets:
jwt_secret_key:
file: ./secrets/jwt_secret_key.txt
oauth_encryption_key:
file: ./secrets/oauth_encryption_key.txt
postgres_password:
file: ./secrets/postgres_password.txt
postgres_user:
file: ./secrets/postgres_user.txt
rabbitmq_pass:
file: ./secrets/rabbitmq_pass.txt
rabbitmq_user:
file: ./secrets/rabbitmq_user.txt
neo4j_password:
file: ./secrets/neo4j_password.txt

services:
api:
environment:
JWT_SECRET_KEY_FILE: /run/secrets/jwt_secret_key
OAUTH_ENCRYPTION_KEY_FILE: /run/secrets/oauth_encryption_key
POSTGRES_PASSWORD_FILE: /run/secrets/postgres_password
POSTGRES_USER_FILE: /run/secrets/postgres_user
secrets:
- jwt_secret_key
- oauth_encryption_key
- postgres_password
- postgres_user

curator:
environment:
JWT_SECRET_KEY_FILE: /run/secrets/jwt_secret_key
POSTGRES_PASSWORD_FILE: /run/secrets/postgres_password
POSTGRES_USER_FILE: /run/secrets/postgres_user
secrets:
- jwt_secret_key
- postgres_password
- postgres_user

explore:
environment:
JWT_SECRET_KEY_FILE: /run/secrets/jwt_secret_key
secrets:
- jwt_secret_key

postgres:
environment:
POSTGRES_PASSWORD_FILE: /run/secrets/postgres_password
POSTGRES_USER_FILE: /run/secrets/postgres_user
secrets:
- postgres_password
- postgres_user

rabbitmq:
environment:
RABBITMQ_DEFAULT_PASS_FILE: /run/secrets/rabbitmq_pass
RABBITMQ_DEFAULT_USER_FILE: /run/secrets/rabbitmq_user
secrets:
- rabbitmq_pass
- rabbitmq_user

neo4j:
# Neo4j doesn't natively support _FILE convention — needs entrypoint wrapper
# NEO4J_AUTH must be set as "/" — construct from secret files in entrypoint
secrets:
- neo4j_password

dashboard:
environment:
RABBITMQ_MANAGEMENT_USER_FILE: /run/secrets/rabbitmq_user
RABBITMQ_MANAGEMENT_PASSWORD_FILE: /run/secrets/rabbitmq_pass
secrets:
- rabbitmq_user
- rabbitmq_pass

tableinator:
environment:
POSTGRES_PASSWORD_FILE: /run/secrets/postgres_password
POSTGRES_USER_FILE: /run/secrets/postgres_user
secrets:
- postgres_password
- postgres_user
```

Note: The official `postgres` and `rabbitmq` images natively support `_FILE` suffixed variables. The `neo4j` image does not — it will need an entrypoint wrapper or the Option B approach. The Python services will require application-level `get_secret()` support (see below).

3. Update Python Services to Support `_FILE` Convention

Add a `get_secret()` helper to `common/` and update each service's settings/config module to use it for all sensitive values. This applies to:

• `JWT_SECRET_KEY` — api, curator, explore
• `OAUTH_ENCRYPTION_KEY` — api
• `POSTGRES_USER` / `POSTGRES_PASSWORD` — api, curator, tableinator, dashboard
• `RABBITMQ_MANAGEMENT_USER` / `RABBITMQ_MANAGEMENT_PASSWORD` — dashboard

```python
import os

def get_secret(env_var: str) -> str | None:
"""Read secret from _FILE path if set, otherwise fall back to env var."""
file_path = os.environ.get(f"{env_var}_FILE")
if file_path:
with open(file_path) as f:
return f.read().strip()
return os.environ.get(env_var)
```

4. Add `secrets.example/` Directory

Provide placeholder files documenting what each secret should contain:

```
secrets.example/
jwt_secret_key.txt # A 256-bit random key: openssl rand -hex 32
oauth_encryption_key.txt # Fernet key: python -c 'from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())'
postgres_password.txt # Strong password for PostgreSQL
postgres_user.txt # PostgreSQL username (e.g. discogsography)
rabbitmq_pass.txt # RabbitMQ password (also used for management API)
rabbitmq_user.txt # RabbitMQ username (also used for management API)
neo4j_password.txt # Neo4j password only (entrypoint constructs neo4j/)
```

5. Add `scripts/create-secrets.sh` Admin Script

Provide a shell script that operators run once on a new production host to generate the `secrets/` directory and populate each file with a cryptographically secure value. The script must:

• Create the `secrets/` directory with mode `700` (owner-only access)
• Generate each secret file with mode `600`
• Use `openssl rand` (or `/dev/urandom`) to produce random values — never hardcoded defaults
• Skip any file that already exists (idempotent — safe to re-run without overwriting)
• Prompt the operator for values that cannot be auto-generated (e.g. `postgres_user`, `rabbitmq_user`)
• Print a summary of which files were created vs. skipped
• Be executable and include a usage comment at the top

Example structure:

```sh
#!/usr/bin/env bash

create-secrets.sh — generate runtime secret files for production deployment

Usage: ./scripts/create-secrets.sh

Run once on the production host before first docker compose up.

Re-running is safe — existing files are never overwritten.

set -euo pipefail

SECRETS_DIR="$(dirname "$0")/../secrets"
mkdir -p "$SECRETS_DIR"
chmod 700 "$SECRETS_DIR"

create_secret() {
local name="$1"
local value="$2"
local file="$SECRETS_DIR/${name}.txt"
if [ -f "$file" ]; then
echo " [skip] $name (already exists)"
else
printf '%s' "$value" > "$file"
chmod 600 "$file"
echo " [created] $name"
fi
}

create_secret "jwt_secret_key" "$(openssl rand -hex 32)"
create_secret "oauth_encryption_key" "$(python3 -c 'from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())')"
create_secret "postgres_password" "$(openssl rand -base64 24)"
create_secret "postgres_user" "discogsography"
create_secret "rabbitmq_pass" "$(openssl rand -base64 24)"
create_secret "rabbitmq_user" "discogsography"
create_secret "neo4j_password" "$(openssl rand -base64 24)"

echo ""
echo "Done. Secrets written to $SECRETS_DIR"
echo "Add secrets/ to .gitignore and never commit these files."
```

---

Security Properties Gained

Property	Env Vars	Runtime Secrets
Visible in `docker inspect`	✅ Yes	❌ No
Written to disk	Depends on `.env` file	❌ Never (in-memory tmpfs)
Persists in image if committed	✅ Yes	❌ No
Flushed when container stops	❌ No	✅ Yes
Per-service access control	❌ No	✅ Yes

---

Acceptance Criteria

• Add `secrets/` directory to `.gitignore`
• Add `secrets.example/` directory with placeholder files and generation instructions for all 7 secrets
• Add `get_secret(env_var)` helper to `common/` supporting the `_FILE` convention
• Update all Python service config/settings modules to use `get_secret()` for `JWT_SECRET_KEY`, `OAUTH_ENCRYPTION_KEY`, `POSTGRES_USER`, `POSTGRES_PASSWORD`, `RABBITMQ_MANAGEMENT_USER`, `RABBITMQ_MANAGEMENT_PASSWORD`
• Update `docker-compose.prod.yml` to define and grant secrets to all applicable services (including `oauth_encryption_key` for api, `rabbitmq_user`/`rabbitmq_pass` for dashboard)
• Verify official images (postgres, rabbitmq) work with `_FILE` variables
• Add entrypoint wrapper for neo4j since `_FILE` convention is not supported natively
• Update `docker-compose.yml` (dev) to document the pattern, keeping hardcoded dev values
• Add `scripts/create-secrets.sh` that generates all 7 secret files with secure random values, sets correct file permissions (700 dir / 600 files), is idempotent (skips existing files), and prints a created/skipped summary

---

References

• Docker Compose secrets how-to
• Docker Swarm secrets
• PostgreSQL `_FILE` variable support — supports `POSTGRES_PASSWORD_FILE`
• RabbitMQ `_FILE` variable support — supports `RABBITMQ_DEFAULT_PASS_FILE`
• Cryptography Fernet — symmetric encryption for `OAUTH_ENCRYPTION_KEY`