This guide uses Windows and PowerShell for local development. Equivalent Linux commands are noted where they differ.
Second Pass Library targets Python 3.14.x for development and deployment.
.python-version selects that minor version, and the local startup and
verification scripts reject a different minor version. The React workspace uses
the Node.js version in .node-version and npm version declared by
frontend/package.json.
python -m venv .venv
.\.venv\Scripts\Activate.ps1
pip install -r requirements-dev.txtrequirements.txt contains runtime dependencies. requirements-dev.txt
adds local test/type tooling such as Django/DRF typing packages.
Pillow is included for cover image validation. Current EPUB, ZIP/OPF metadata,
cover precedence, normalization, and safety rules are in Imports.
The React Product UI has its npm workspace under frontend/; see
Frontend. There is no repository-root npm project.
The first-party migration history was flattened before release. Existing development databases created from the old migration history are intentionally unsupported. Recreate the database and run the new initial migrations; do not fake the new initial migrations onto an old database.
Existing files under userdata/media/ may be retained, but their old database
rows are not reusable. Books must be re-imported into the fresh database before
the retained media can be treated as library content again.
Create normal schema changes with migrations from the current initial state. Do not add compatibility migrations, fake initial state, or application-level shims to make databases from the deleted migration history appear compatible.
Book stores its EPUB and cover fields directly.
Do not add compatibility re-export modules or wrapper imports for deleted or
renamed modules. Update callers to the current module boundary instead.
Runtime files belong under userdata/ or a test-isolated temporary root.
Development and other non-Docker installations may set
SECOND_PASS_USERDATA_DIR to a different runtime root. The standard Docker
image always uses /app/userdata as its internal runtime root.
Committed files under TestFiles/ or tests/fixtures/ are fixtures only, not
runtime storage or a destination for generated artifacts.
- API conventions and external contracts
- Architecture
- Permissions, with immutable Advanced/Simple Mode and Book visibility policy
- Imports
- Marginalia, with immutable visibility and preservation
- Production startup
- Operator maintenance and repair
- Frontend architecture
Use the standard Python logging module with a module-level logger:
logging.getLogger(__name__). Add a log only when it gives an operator or
developer useful information about runtime behavior. A touched file does not
need a new log merely because it was changed.
Choose the least severe useful level:
DEBUG: expected ignored edge cases, optional enrichment that was skipped, and fallback paths useful during development. These are normal conditions, not operator alerts.INFO: successful operator-level actions and concise management-command, import, or cleanup summaries. Do not use it for routine request success.WARNING: recoverable corruption, self-healing, ignored unsafe archive members, or failure of a best-effort feature after the requested primary action succeeded.ERROR: an unexpected failure that blocks the requested action. Preserve useful exception context where it helps diagnose the failure, without exposing sensitive input.
Use structured or consistently bounded messages. Never log secrets, passwords, authentication or client tokens, email addresses, marginalia content, raw uploads or payloads, unsafe or absolute paths, file hashes, or storage keys. For unsafe archive input, log a safe summary or count rather than repeating attacker-controlled text. Avoid high-volume per-request success messages; Django and the serving stack already provide request-level diagnostics where configured.
When reviewing a runtime change, explicitly decide whether logging would help operate or diagnose it. If not, leave the code quiet and state in the implementation report that no useful logging was warranted. Deployment output continues to use standard stdout/stderr logging as described in Production startup; application code should not invent its own file-log directory.
Operator log-level changes and production log review are documented in Operations.
Cover images (and other user media) are addressed under MEDIA_URL (default: /media/) and stored under MEDIA_ROOT (default: userdata/media).
In local development, Django serves only /media/covers/ so cover images render in the product UI.
Django owns cover cache headers in development and production-like direct
serving. Canonical SHA-256 cover paths are public and immutable for one year,
and non-content-addressed cover paths are not served. The route retains
Last-Modified and conditional 304 handling for canonical covers.
Production deployments must handle durable media separately from WhiteNoise.
WhiteNoise serves packaged Product UI assets under /static/ only. Django
serves only the public cover namespace, /media/covers/; stored EPUB files,
imports, exports, marginalia, and other protected user data are never served as
raw media URLs.
Production collectstatic output goes to backend/var/static/, which is generated and
can be rebuilt. The canonical backup unit is documented in
Operations.
Docker deployment options live in docker/compose.example.yml. The local
PowerShell helpers define their own environment in the script files.
.\scripts\start-dev.ps1The script sets DJANGO_DEBUG=1, disables WhiteNoise runtime caching for faster
template/static iteration, runs python backend/manage.py migrate --noinput, and then
starts Django on port 8000 and the Product UI Vite server on port 5174. Run npm.cmd --prefix frontend install before using it for the first time. Vite uses the proxy configuration
documented in Frontend and is stopped when the Django
process exits. The script's Python executable and application environment are
defined in the script and do not inherit configuration choices from the calling
shell. Edit the values near the top of the script when local settings need to
change. Additional arguments are passed through to Django's runserver, for
example:
.\scripts\start-dev.ps1 --noreloadRaw python backend/manage.py runserver uses the normal settings defaults. Because
DEBUG defaults to false, local development behavior requires either the
development startup script or an explicit DJANGO_DEBUG=1 in the shell before
running raw runserver.
For production-likeness, use .\scripts\start-local-production.ps1; it keeps
DJANGO_DEBUG=0, builds the React workspace, runs collectstatic, and uses
WhiteNoise in manifest-backed mode. After collection it retains the Product UI
index and removes the redundant source-asset copy. It requires npm.cmd install to have been
run for frontend/, but it does not run a Vite or Node server. It runs one
direct Uvicorn worker against
secondpass.asgi:application, matching the Docker application target and
runtime path. Access logs are disabled in both deployment-like paths. Static
and media routing is unchanged: covers remain public display assets, while
EPUB downloads remain authenticated application/API responses. Proxy-header
trust remains deployment-owned and is not broadened by these startup helpers.
The development script sets DJANGO_ALLOWED_HOSTS to
localhost,127.0.0.1,[::1] and enables the Django Admin service hatch. Edit
the script if a LAN hostname or IP is needed during development.
For a clean local reset:
- Stop the server.
- Delete
userdata/if the database and all local runtime/user data may be discarded. - Run the development startup script.
- Visit
/. - Complete the first-run setup wizard.
As an optional headless alternative to step 4-5, run
python backend/manage.py setup_server --username owner from an interactive
terminal after startup. Raw or Django-encoded credentials can instead be piped
through stdin for automation; see
Optional command-line setup for
the secure input modes and Docker examples.
The setup page initializes:
- Server Name:
Second Pass Library - Server Description: blank
- Public Group Name:
Common Room - Public Group Description:
Main Public Library Room for everyone - Enable Advanced Library Group Usage: off
- The first active Owner account, its Manager
UserProfile, and its reader membership in the Public group
Email and the display-name fields are optional account metadata. Common Room
is the default display name for the protected shared public library space
managed by librarians and managers. Enabling advanced library groups presents
separate curator-managed rooms as a first-class UI feature and enables normal
non-Public group mutation workflows. Product UI does not provide a disable
control after enablement; the operator recovery flow is documented in
Operations. Once an active Owner
exists, /setup/ is disabled and normal Product UI login at /login/ is used.
Raw python backend/manage.py runserver remains available, but it does not create or
migrate the database schema. If using raw runserver, set DJANGO_DEBUG=1 for
local debug/static/media behavior and run python backend/manage.py migrate --noinput
first. The setup wizard assumes migrations already exist; it does not create
database tables during an HTTP request.
start-dev.ps1 and start-local-production.ps1 start and supervise the Huey
maintenance consumer automatically. When using raw runserver or Uvicorn,
start it in a second terminal:
python backend/manage.py run_hueyThe consumer uses userdata/db/huey.sqlite3. Existing maintenance management
commands remain synchronous and do not require Huey to be running.
The setup wizard is not development-only. The same migrated-database/no-active-
Owner condition is used in production. See docs/deployment.md.
The first reader client is expected to be a standalone browser app (e.g. React) running on http://localhost:5173 during development.
The Django server enables open CORS for API + discovery endpoints only:
/.well-known/*/api/*
This supports standalone reader clients from arbitrary origins. Cross-origin
cookie credentials are not enabled; clients must use Authorization: Bearer ...
tokens for protected API calls. The Product UI remains same-origin and uses
session auth with CSRF.
The Product UI is the Vite React workspace under frontend/. Django serves the
authenticated React shell at /; ordinary development should use the Vite
server on port 5174.
Django continues to render /setup/, /login/, /logout/, and the optional
/admin/ service hatch. DRF browsable pages and /api-auth/ are disabled.
React owns pairing approval at /profile/client-pairing; there are no alternate
Product UI mounts or compatibility routes.
HTML and API missing-route behavior intentionally differs:
- Non-API missing pages return styled HTML error pages.
/api/missing routes return JSON 404 responses shaped as{"detail": "Not found."}.- API route-level 404 tests should assert JSON content type and response body.
Useful focused checks:
.\.venv\Scripts\python.exe -m pytest tests/core/test_api_route_errors.py -q
.\.venv\Scripts\python.exe tools\static_hygiene.py
.\.venv\Scripts\python.exe -m ruff check .Second Pass Library uses Django/DRF authentication:
- Django session authentication for the React Product UI
- Client API bearer token authentication on selected reader-client endpoints
- Django admin authentication at
/admin/(a service hatch; not the product UI)
Practical notes:
- Use
/login/for Product UI session authentication. APIs return JSON rather than DRF browsable pages. - Use
/admin/to access the Django admin only whenSECOND_PASS_ENABLE_DJANGO_ADMIN=1is set. The local production helper sets this for operator testing only when the variable is unset and respects an explicit0; the settings default leaves the route unregistered. - Admin repair and recovery procedures are documented in Operations.
- HTTP Basic authentication is not enabled. Non-browser reader clients should use the Client API bearer-token flow where supported.
Use the smallest workflow that matches the change:
| Work | Canonical command |
|---|---|
| Run the development stack | .\scripts\start-dev.ps1 |
| Focused backend test | .\.venv\Scripts\python.exe -m pytest <test-path> -q |
| Focused frontend test | npm.cmd --prefix frontend run test:vitest -- --run <test-path> |
| Normal cross-project check | .\.venv\Scripts\python.exe tools\verify.py --lane fast |
| Backend release confidence | .\.venv\Scripts\python.exe tools\verify.py --lane backend |
| Frontend release confidence | .\.venv\Scripts\python.exe tools\verify.py --lane frontend |
| Complete release confidence | .\.venv\Scripts\python.exe tools\verify.py --lane all |
VS Code exposes these common workflows as Develop: Full Stack, the three
Verify scope tasks, and Verify: All. Coverage remains available through the
explicit command-line lanes below; it is intentionally absent from the short
task list because it is not an every-change workflow.
Run the complete project-owned verification entry point with the active Python environment:
.\.venv\Scripts\python.exe tools\verify.pyOn Linux, use .venv/bin/python tools/verify.py. Select a narrower supported
lane with --lane:
fast: lint, Django checks, migration consistency, an explicitly owned backend push-confidence sample, complete Product UI/SDK tests with boundary enforcement, and static hygiene. This is the normal maintainer check before pushes and pull-request reviews and is designed to finish in a few minutes;backend: the complete Python suite;backend-coverage: the complete Python suite with branch coverage undertest-results/coverage/python/;frontend: Vitest, TypeScript, the boundary checker, and production build;frontend-coverage: separate Product UI and SDK coverage undertest-results/coverage/frontend/;security-deps: opt-in live dependency advisory checks usingpip-auditfor pinned Python requirements andnpm auditfor the frontend lockfile;all: the release-confidence aggregate and default.
The advisory lane requires network access and a developer-installed
pip-audit (python -m pip install pip-audit). It is intentionally separate
from fast and release-confidence verification so registry outages do not block
offline development. pip check and npm ls remain dependency-integrity
checks, not vulnerability scans.
Application verification belongs to local maintainers through the repository-owned
tools/verify.py lanes. GitHub Actions only builds, smoke-tests, verifies, and
publishes container artifacts; it does not run application tests or advisory scans.
The manual .github/workflows/build-image.yml workflow builds and smoke-tests
the selected commit as a development image. Its publish input defaults to
false. When enabled, it publishes the immutable dev-sha-<full-commit-sha> tag,
verifies that artifact from the registry, and then moves dev to the same
digest. The dev channel therefore identifies the latest successfully built,
smoke-tested, and published development image. It does not create a release or
publish latest.
Pushing a release tag matching alpha-rc<integer> or
alpha-rc<integer>.<integer> starts .github/workflows/release.yml. That
workflow validates the exact tag, then builds, smoke-tests, and publishes the
production image. Before creating the tag, the maintainer must already have run
appropriate local verification through tools/verify.py. It publishes both
sha-<full-commit-sha> and the exact Git tag as image tags, then deletes its
local image and pulls the SHA tag back from the registry to smoke-test runtime
startup and embedded version/date metadata again. It also pulls the exact release
tag and checks that both tags resolve to the same digest. The registry image is
the release artifact; the workflow does not create a GitHub Release.
Both workflows share .github/actions/build-production-image for Buildx setup,
image labels, embedded metadata, local loading, and the production smoke handoff.
Publishing remains in each workflow because development channel promotion and
release tags have different policies. Buildx layer caching through type=gha is
deferred until build timings justify it; no application dependency cache is used.
Images publish to ghcr.io/orionshock/secondpasslibrary. Workflows authenticate
with github.actor and the automatic GITHUB_TOKEN; no registry repository
variables or custom token are required. Only publishing jobs receive
packages: write, alongside contents: read. The manual workflow selects
mutually exclusive build-only and publishing jobs sharing the same anchored
steps; publish=false has no package-write permission, login, or push.
A new GHCR package defaults to private. Anonymous pulls require a later, intentional change of package visibility to public; these workflows do not change visibility. Until then, deployment hosts need authenticated GHCR access.
Tags should be intentional release identifiers. Existing releases establish
the alpha-rc1 and alpha-rc1.1 forms. The tagged source tree contains the
corresponding self-contained docker/compose.example.yml.
Individual checks remain available below for focused work.
python backend/manage.py check
python -m pytest
npm --prefix frontend run test:vitest -- tests/app/AppBootstrap.test.tsx
npm --prefix frontend run typecheck
npm --prefix frontend run build
node frontend/scripts/check-boundaries.mjsPyright uses the checked-in pyrightconfig.json with backend/ as the Python
import root. The frontend production build runs the TypeScript project checks
before Vite emits the Product UI artifact.
Prefer pytest for focused test runs. Run the smallest package, module, class,
or test that covers the change. Broad suites are intentional, not the default.
When tests change, report each test or coherent test group as an invariant,
contract, regression, or implementation detail. Do not weaken an
invariant or contract assertion without explicitly identifying and justifying
the weaker guarantee.
Use the smallest relevant path or node first. The supported broad backend lanes
are explicit selections; unfiltered pytest still runs every test.
# Broad developer lane: everything except the measured/inherently slow slice.
.\.venv\Scripts\python.exe -m pytest -m "not slow" -q
# Purpose-based lanes. Markers overlap intentionally.
.\.venv\Scripts\python.exe -m pytest -m security -q
.\.venv\Scripts\python.exe -m pytest -m filesystem -q
.\.venv\Scripts\python.exe -m pytest -m integration -q
.\.venv\Scripts\python.exe -m pytest -m slow -q
# Full backend release/comprehensive run. No tests are excluded by default.
.\.venv\Scripts\python.exe -m pytest -qMarker meanings:
push: the deliberately selected backend semantic sample used byfast. It covers representative authentication/token, password-policy, authority, API error/shape, immutable identity, canonical visibility, Group fallback, cache publication, maintenance registration, Marginalia lifecycle, and Shelf authorization/model contracts. It is allowlisted so newly added tests cannot silently make normal push feedback unbounded.security: owning authentication, authorization, anti-enumeration, credential, archive-safety, destructive-operation, and immutable-policy boundaries. It is narrower than the complete set of permission assertions.concurrency: real threads, executors, barriers, or competing transactions.subprocess: a real server, subprocess, or separate interpreter.filesystem: behavior whose contract depends on files, media, archive bytes, temporary directories, static output, or storage cleanup.integration: several real application layers whose boundary is not replaced by a lower-level test.slow: tests measured as expensive or inherently unsuitable for normal push feedback. Important coverage remains authoritative inbackendandall.
The fast lane intentionally leaves exhaustive matrices and expensive
families to backend and all: real concurrency and subprocess scheduling,
filesystem/archive integration (including hostile-file cases), broad
multi-layer integration, password-heavy role matrices, and complete
import/export workflows. Those tests are not optional; they are excluded from
routine push feedback because their boundary requires the slower execution
model. Purpose markers continue to make those families directly runnable.
Run the concurrency lane with SQLite resource warnings promoted to errors:
.\.venv\Scripts\python.exe -X dev -m pytest -q -m concurrency `
-W "error::ResourceWarning" `
-W "error::pytest.PytestUnraisableExceptionWarning"Coherent backend domain suites:
# Accounts and browser/client security
.\.venv\Scripts\python.exe -m pytest tests/accounts -q
# Core runtime, settings, server shell, static, and media behavior
.\.venv\Scripts\python.exe -m pytest tests/core -q
# Library catalog, Groups, bearer reads, models, queries, and Admin
.\.venv\Scripts\python.exe -m pytest tests/library/catalog tests/library/groups tests/library/bearer `
tests/library/test_models.py tests/library/test_queries.py tests/library/test_roles.py `
tests/library/test_admin_book_changelist.py tests/library/test_admin_book_group_assignment.py `
tests/library/test_admin_book_layout.py tests/library/test_admin_catalog_tag.py -q
# Library imports and stored-file repair
.\.venv\Scripts\python.exe -m pytest tests/library/imports `
tests/library/test_file_repair.py tests/library/test_admin_file_repair.py -q
.\.venv\Scripts\python.exe -m pytest tests/shelves -q
.\.venv\Scripts\python.exe -m pytest tests/marginalia -q
.\.venv\Scripts\python.exe -m pytest tests/testenv -qFrontend selections use paths instead of a second marker system:
# Focused file, SDK tests, Product UI, then complete Vitest.
npm.cmd --prefix frontend run test:vitest -- --run tests/app/AppBootstrap.test.tsx
npm.cmd --prefix frontend run test:vitest -- --run tests/sdk
npm.cmd --prefix frontend run test:vitest -- --run tests/app tests/components tests/domain tests/features tests/shared
npm.cmd --prefix frontend run test:vitest -- --run
npm.cmd --prefix frontend run typecheck
node frontend/scripts/check-boundaries.mjs
npm.cmd --prefix frontend run buildThese commands use the dedicated test root under frontend/tests/, with SDK
wire-contract tests under frontend/tests/sdk/ and Product UI tests grouped by
their runtime owner. Production SDK source remains under
frontend/packages/spl-api/src/.
Use npm instead of npm.cmd on shells where the executable shim is not
blocked by PowerShell execution policy.
Hygiene:
.\.venv\Scripts\python.exe -m ruff check tests
.\.venv\Scripts\python.exe tools\static_hygiene.py
.\.venv\Scripts\python.exe tools\static_hygiene.py --all
.\.venv\Scripts\python.exe tools\static_hygiene.py --fix-mojibake
.\.venv\Scripts\python.exe tools\static_hygiene.py --fix-line-endingstools\static_hygiene.py is the repo check for decorative HTML entities,
mojibake markers, trailing whitespace, LF line endings, and git diff --check.
By default it checks touched files only and does not modify files. Use --all
for a full tracked-file scan. --fix-mojibake, --fix-trailing-whitespace,
and --fix-line-endings may modify files in the scan set.
Pyright currently excludes tests in the checked-in configuration. For a
report-only type audit, use the local node_modules\.bin\pyright.cmd with a
temporary config that includes tests and adds the repository root to
extraPaths; do not treat this as a required gate until that setup is checked
in deliberately.
Test helpers:
- Prefer
tests/utils/books.py::create_file_backed_book()when a test needs a normal valid Book. The product invariant is that Books are file-backed.
Create a richer development/demo world for manual UI testing:
.\scripts\seed-dev-users.ps1The command:
- applies pending migrations before reading the database
- requires first-run setup to be complete; it does not create the Owner account
- ensures the configured Public group exists (
Common Roomon a default server) - ensures about 20 lorem-named demo users
- keeps Manager and Librarian demo accounts as broad-role users with their normal default Common Room membership
- never assigns curator flags to the Public/Common Room group
- creates one private
Reading Queueand one listedFavoritesshelf for each demo user, plus Common Room shelves - populates managed personal and Group shelves with small, varied selections of visible or exact-Group Books when Books are available
- when advanced library groups are enabled, additionally creates five non-Public demo groups by default, varied intentional memberships, curators distributed across Reader, Librarian, and Manager roles where possible, and 3-4 group-owned shelves per Group
- gives the named demo Groups simple thematic Catalog Tag profiles; a Book is assigned when a normalized profile keyword occurs anywhere in one of its normalized tag names, with deterministic fallback Books for small catalogs
The command preserves existing data by default. Existing users with matching usernames retain their names, email addresses, passwords, flags, and profile roles. Existing shelves are reused without overwriting their descriptions or other fields. Re-running with the same seed does not duplicate memberships, shelves, or shelf items. Existing demo groups are reused when advanced library groups are enabled.
Useful options:
.\scripts\seed-dev-users.ps1 --seed family-demo
.\scripts\seed-dev-users.ps1 --users 12 --groups 3
.\scripts\seed-dev-users.ps1 --skip-shelvesSafety:
- Newly created demo accounts use the predictable password
changeme123. - The command refuses to run unless
DEBUG=True(use--forceonly for local development). - This command is not the normal installation bootstrap path. Fresh installs should use the first-run setup page. It remains a development/demo helper only.
Recurring cleanup and repair commands belong to Operations. The explicit local source modes are:
python backend/manage.py import_library_folder_of_zip "path\to\book.epub"
python backend/manage.py import_library_folder_of_zip "path\to\per-book-archives"
python backend/manage.py import_library_aio_zip "path\to\large-library.zip"
python backend/manage.py import_library_tree "path\to\unpacked-library"All three commands share the Product/API per-candidate importer and do not create durable import history. See Imports for pairing, safety, large-archive, read-only bind-mount, interruption, and rerun behavior.
See also:
docs/api.md(shared conventions and external contracts)docs/imports.md(import workflow)