Skip to content

Latest commit

 

History

History
598 lines (473 loc) · 27.6 KB

development.md

File metadata and controls

598 lines (473 loc) · 27.6 KB

Development

This guide uses Windows and PowerShell for local development. Equivalent Linux commands are noted where they differ.

Setup

Second Pass Library targets Python 3.14.x for development and deployment. .python-version selects that minor version, and the local startup and verification scripts reject a different minor version. The React workspace uses the Node.js version in .node-version and npm version declared by frontend/package.json.

python -m venv .venv
.\.venv\Scripts\Activate.ps1
pip install -r requirements-dev.txt

requirements.txt contains runtime dependencies. requirements-dev.txt adds local test/type tooling such as Django/DRF typing packages. Pillow is included for cover image validation. Current EPUB, ZIP/OPF metadata, cover precedence, normalization, and safety rules are in Imports.

The React Product UI has its npm workspace under frontend/; see Frontend. There is no repository-root npm project.

Pre-release migration reset

The first-party migration history was flattened before release. Existing development databases created from the old migration history are intentionally unsupported. Recreate the database and run the new initial migrations; do not fake the new initial migrations onto an old database.

Existing files under userdata/media/ may be retained, but their old database rows are not reusable. Books must be re-imported into the fresh database before the retained media can be treated as library content again.

Create normal schema changes with migrations from the current initial state. Do not add compatibility migrations, fake initial state, or application-level shims to make databases from the deleted migration history appear compatible.

Code and fixture locations

Book stores its EPUB and cover fields directly. Do not add compatibility re-export modules or wrapper imports for deleted or renamed modules. Update callers to the current module boundary instead.

Runtime files belong under userdata/ or a test-isolated temporary root. Development and other non-Docker installations may set SECOND_PASS_USERDATA_DIR to a different runtime root. The standard Docker image always uses /app/userdata as its internal runtime root. Committed files under TestFiles/ or tests/fixtures/ are fixtures only, not runtime storage or a destination for generated artifacts.

Related docs

Application logging

Use the standard Python logging module with a module-level logger: logging.getLogger(__name__). Add a log only when it gives an operator or developer useful information about runtime behavior. A touched file does not need a new log merely because it was changed.

Choose the least severe useful level:

  • DEBUG: expected ignored edge cases, optional enrichment that was skipped, and fallback paths useful during development. These are normal conditions, not operator alerts.
  • INFO: successful operator-level actions and concise management-command, import, or cleanup summaries. Do not use it for routine request success.
  • WARNING: recoverable corruption, self-healing, ignored unsafe archive members, or failure of a best-effort feature after the requested primary action succeeded.
  • ERROR: an unexpected failure that blocks the requested action. Preserve useful exception context where it helps diagnose the failure, without exposing sensitive input.

Use structured or consistently bounded messages. Never log secrets, passwords, authentication or client tokens, email addresses, marginalia content, raw uploads or payloads, unsafe or absolute paths, file hashes, or storage keys. For unsafe archive input, log a safe summary or count rather than repeating attacker-controlled text. Avoid high-volume per-request success messages; Django and the serving stack already provide request-level diagnostics where configured.

When reviewing a runtime change, explicitly decide whether logging would help operate or diagnose it. If not, leave the code quiet and state in the implementation report that no useful logging was warranted. Deployment output continues to use standard stdout/stderr logging as described in Production startup; application code should not invent its own file-log directory.

Operator log-level changes and production log review are documented in Operations.

Media serving (dev)

Cover images (and other user media) are addressed under MEDIA_URL (default: /media/) and stored under MEDIA_ROOT (default: userdata/media).

In local development, Django serves only /media/covers/ so cover images render in the product UI.

Django owns cover cache headers in development and production-like direct serving. Canonical SHA-256 cover paths are public and immutable for one year, and non-content-addressed cover paths are not served. The route retains Last-Modified and conditional 304 handling for canonical covers.

Production deployments must handle durable media separately from WhiteNoise. WhiteNoise serves packaged Product UI assets under /static/ only. Django serves only the public cover namespace, /media/covers/; stored EPUB files, imports, exports, marginalia, and other protected user data are never served as raw media URLs. Production collectstatic output goes to backend/var/static/, which is generated and can be rebuilt. The canonical backup unit is documented in Operations.

Docker deployment options live in docker/compose.example.yml. The local PowerShell helpers define their own environment in the script files.

Run server

.\scripts\start-dev.ps1

The script sets DJANGO_DEBUG=1, disables WhiteNoise runtime caching for faster template/static iteration, runs python backend/manage.py migrate --noinput, and then starts Django on port 8000 and the Product UI Vite server on port 5174. Run npm.cmd --prefix frontend install before using it for the first time. Vite uses the proxy configuration documented in Frontend and is stopped when the Django process exits. The script's Python executable and application environment are defined in the script and do not inherit configuration choices from the calling shell. Edit the values near the top of the script when local settings need to change. Additional arguments are passed through to Django's runserver, for example:

.\scripts\start-dev.ps1 --noreload

Raw python backend/manage.py runserver uses the normal settings defaults. Because DEBUG defaults to false, local development behavior requires either the development startup script or an explicit DJANGO_DEBUG=1 in the shell before running raw runserver.

For production-likeness, use .\scripts\start-local-production.ps1; it keeps DJANGO_DEBUG=0, builds the React workspace, runs collectstatic, and uses WhiteNoise in manifest-backed mode. After collection it retains the Product UI index and removes the redundant source-asset copy. It requires npm.cmd install to have been run for frontend/, but it does not run a Vite or Node server. It runs one direct Uvicorn worker against secondpass.asgi:application, matching the Docker application target and runtime path. Access logs are disabled in both deployment-like paths. Static and media routing is unchanged: covers remain public display assets, while EPUB downloads remain authenticated application/API responses. Proxy-header trust remains deployment-owned and is not broadened by these startup helpers.

The development script sets DJANGO_ALLOWED_HOSTS to localhost,127.0.0.1,[::1] and enables the Django Admin service hatch. Edit the script if a LAN hostname or IP is needed during development.

For a clean local reset:

  1. Stop the server.
  2. Delete userdata/ if the database and all local runtime/user data may be discarded.
  3. Run the development startup script.
  4. Visit /.
  5. Complete the first-run setup wizard.

As an optional headless alternative to step 4-5, run python backend/manage.py setup_server --username owner from an interactive terminal after startup. Raw or Django-encoded credentials can instead be piped through stdin for automation; see Optional command-line setup for the secure input modes and Docker examples.

The setup page initializes:

  • Server Name: Second Pass Library
  • Server Description: blank
  • Public Group Name: Common Room
  • Public Group Description: Main Public Library Room for everyone
  • Enable Advanced Library Group Usage: off
  • The first active Owner account, its Manager UserProfile, and its reader membership in the Public group

Email and the display-name fields are optional account metadata. Common Room is the default display name for the protected shared public library space managed by librarians and managers. Enabling advanced library groups presents separate curator-managed rooms as a first-class UI feature and enables normal non-Public group mutation workflows. Product UI does not provide a disable control after enablement; the operator recovery flow is documented in Operations. Once an active Owner exists, /setup/ is disabled and normal Product UI login at /login/ is used.

Raw python backend/manage.py runserver remains available, but it does not create or migrate the database schema. If using raw runserver, set DJANGO_DEBUG=1 for local debug/static/media behavior and run python backend/manage.py migrate --noinput first. The setup wizard assumes migrations already exist; it does not create database tables during an HTTP request.

start-dev.ps1 and start-local-production.ps1 start and supervise the Huey maintenance consumer automatically. When using raw runserver or Uvicorn, start it in a second terminal:

python backend/manage.py run_huey

The consumer uses userdata/db/huey.sqlite3. Existing maintenance management commands remain synchronous and do not require Huey to be running.

The setup wizard is not development-only. The same migrated-database/no-active- Owner condition is used in production. See docs/deployment.md.

Standalone reader dev (React)

The first reader client is expected to be a standalone browser app (e.g. React) running on http://localhost:5173 during development.

The Django server enables open CORS for API + discovery endpoints only:

  • /.well-known/*
  • /api/*

This supports standalone reader clients from arbitrary origins. Cross-origin cookie credentials are not enabled; clients must use Authorization: Bearer ... tokens for protected API calls. The Product UI remains same-origin and uses session auth with CSRF.

Product UI

The Product UI is the Vite React workspace under frontend/. Django serves the authenticated React shell at /; ordinary development should use the Vite server on port 5174.

Django continues to render /setup/, /login/, /logout/, and the optional /admin/ service hatch. DRF browsable pages and /api-auth/ are disabled. React owns pairing approval at /profile/client-pairing; there are no alternate Product UI mounts or compatibility routes.

Error-handling checks

HTML and API missing-route behavior intentionally differs:

  • Non-API missing pages return styled HTML error pages.
  • /api/ missing routes return JSON 404 responses shaped as {"detail": "Not found."}.
  • API route-level 404 tests should assert JSON content type and response body.

Useful focused checks:

.\.venv\Scripts\python.exe -m pytest tests/core/test_api_route_errors.py -q
.\.venv\Scripts\python.exe tools\static_hygiene.py
.\.venv\Scripts\python.exe -m ruff check .

Authentication

Second Pass Library uses Django/DRF authentication:

  • Django session authentication for the React Product UI
  • Client API bearer token authentication on selected reader-client endpoints
  • Django admin authentication at /admin/ (a service hatch; not the product UI)

Practical notes:

  • Use /login/ for Product UI session authentication. APIs return JSON rather than DRF browsable pages.
  • Use /admin/ to access the Django admin only when SECOND_PASS_ENABLE_DJANGO_ADMIN=1 is set. The local production helper sets this for operator testing only when the variable is unset and respects an explicit 0; the settings default leaves the route unregistered.
  • Admin repair and recovery procedures are documented in Operations.
  • HTTP Basic authentication is not enabled. Non-browser reader clients should use the Client API bearer-token flow where supported.

Run Checks And Tests

Use the smallest workflow that matches the change:

Work Canonical command
Run the development stack .\scripts\start-dev.ps1
Focused backend test .\.venv\Scripts\python.exe -m pytest <test-path> -q
Focused frontend test npm.cmd --prefix frontend run test:vitest -- --run <test-path>
Normal cross-project check .\.venv\Scripts\python.exe tools\verify.py --lane fast
Backend release confidence .\.venv\Scripts\python.exe tools\verify.py --lane backend
Frontend release confidence .\.venv\Scripts\python.exe tools\verify.py --lane frontend
Complete release confidence .\.venv\Scripts\python.exe tools\verify.py --lane all

VS Code exposes these common workflows as Develop: Full Stack, the three Verify scope tasks, and Verify: All. Coverage remains available through the explicit command-line lanes below; it is intentionally absent from the short task list because it is not an every-change workflow.

Run the complete project-owned verification entry point with the active Python environment:

.\.venv\Scripts\python.exe tools\verify.py

On Linux, use .venv/bin/python tools/verify.py. Select a narrower supported lane with --lane:

  • fast: lint, Django checks, migration consistency, an explicitly owned backend push-confidence sample, complete Product UI/SDK tests with boundary enforcement, and static hygiene. This is the normal maintainer check before pushes and pull-request reviews and is designed to finish in a few minutes;
  • backend: the complete Python suite;
  • backend-coverage: the complete Python suite with branch coverage under test-results/coverage/python/;
  • frontend: Vitest, TypeScript, the boundary checker, and production build;
  • frontend-coverage: separate Product UI and SDK coverage under test-results/coverage/frontend/;
  • security-deps: opt-in live dependency advisory checks using pip-audit for pinned Python requirements and npm audit for the frontend lockfile;
  • all: the release-confidence aggregate and default.

The advisory lane requires network access and a developer-installed pip-audit (python -m pip install pip-audit). It is intentionally separate from fast and release-confidence verification so registry outages do not block offline development. pip check and npm ls remain dependency-integrity checks, not vulnerability scans.

CI and releases

Application verification belongs to local maintainers through the repository-owned tools/verify.py lanes. GitHub Actions only builds, smoke-tests, verifies, and publishes container artifacts; it does not run application tests or advisory scans.

The manual .github/workflows/build-image.yml workflow builds and smoke-tests the selected commit as a development image. Its publish input defaults to false. When enabled, it publishes the immutable dev-sha-<full-commit-sha> tag, verifies that artifact from the registry, and then moves dev to the same digest. The dev channel therefore identifies the latest successfully built, smoke-tested, and published development image. It does not create a release or publish latest.

Pushing a release tag matching alpha-rc<integer> or alpha-rc<integer>.<integer> starts .github/workflows/release.yml. That workflow validates the exact tag, then builds, smoke-tests, and publishes the production image. Before creating the tag, the maintainer must already have run appropriate local verification through tools/verify.py. It publishes both sha-<full-commit-sha> and the exact Git tag as image tags, then deletes its local image and pulls the SHA tag back from the registry to smoke-test runtime startup and embedded version/date metadata again. It also pulls the exact release tag and checks that both tags resolve to the same digest. The registry image is the release artifact; the workflow does not create a GitHub Release.

Both workflows share .github/actions/build-production-image for Buildx setup, image labels, embedded metadata, local loading, and the production smoke handoff. Publishing remains in each workflow because development channel promotion and release tags have different policies. Buildx layer caching through type=gha is deferred until build timings justify it; no application dependency cache is used.

Images publish to ghcr.io/orionshock/secondpasslibrary. Workflows authenticate with github.actor and the automatic GITHUB_TOKEN; no registry repository variables or custom token are required. Only publishing jobs receive packages: write, alongside contents: read. The manual workflow selects mutually exclusive build-only and publishing jobs sharing the same anchored steps; publish=false has no package-write permission, login, or push.

A new GHCR package defaults to private. Anonymous pulls require a later, intentional change of package visibility to public; these workflows do not change visibility. Until then, deployment hosts need authenticated GHCR access.

Tags should be intentional release identifiers. Existing releases establish the alpha-rc1 and alpha-rc1.1 forms. The tagged source tree contains the corresponding self-contained docker/compose.example.yml.

Individual checks remain available below for focused work.

python backend/manage.py check
python -m pytest
npm --prefix frontend run test:vitest -- tests/app/AppBootstrap.test.tsx
npm --prefix frontend run typecheck
npm --prefix frontend run build
node frontend/scripts/check-boundaries.mjs

Pyright uses the checked-in pyrightconfig.json with backend/ as the Python import root. The frontend production build runs the TypeScript project checks before Vite emits the Product UI artifact.

Prefer pytest for focused test runs. Run the smallest package, module, class, or test that covers the change. Broad suites are intentional, not the default. When tests change, report each test or coherent test group as an invariant, contract, regression, or implementation detail. Do not weaken an invariant or contract assertion without explicitly identifying and justifying the weaker guarantee. Use the smallest relevant path or node first. The supported broad backend lanes are explicit selections; unfiltered pytest still runs every test.

# Broad developer lane: everything except the measured/inherently slow slice.
.\.venv\Scripts\python.exe -m pytest -m "not slow" -q

# Purpose-based lanes. Markers overlap intentionally.
.\.venv\Scripts\python.exe -m pytest -m security -q
.\.venv\Scripts\python.exe -m pytest -m filesystem -q
.\.venv\Scripts\python.exe -m pytest -m integration -q
.\.venv\Scripts\python.exe -m pytest -m slow -q

# Full backend release/comprehensive run. No tests are excluded by default.
.\.venv\Scripts\python.exe -m pytest -q

Marker meanings:

  • push: the deliberately selected backend semantic sample used by fast. It covers representative authentication/token, password-policy, authority, API error/shape, immutable identity, canonical visibility, Group fallback, cache publication, maintenance registration, Marginalia lifecycle, and Shelf authorization/model contracts. It is allowlisted so newly added tests cannot silently make normal push feedback unbounded.
  • security: owning authentication, authorization, anti-enumeration, credential, archive-safety, destructive-operation, and immutable-policy boundaries. It is narrower than the complete set of permission assertions.
  • concurrency: real threads, executors, barriers, or competing transactions.
  • subprocess: a real server, subprocess, or separate interpreter.
  • filesystem: behavior whose contract depends on files, media, archive bytes, temporary directories, static output, or storage cleanup.
  • integration: several real application layers whose boundary is not replaced by a lower-level test.
  • slow: tests measured as expensive or inherently unsuitable for normal push feedback. Important coverage remains authoritative in backend and all.

The fast lane intentionally leaves exhaustive matrices and expensive families to backend and all: real concurrency and subprocess scheduling, filesystem/archive integration (including hostile-file cases), broad multi-layer integration, password-heavy role matrices, and complete import/export workflows. Those tests are not optional; they are excluded from routine push feedback because their boundary requires the slower execution model. Purpose markers continue to make those families directly runnable.

Run the concurrency lane with SQLite resource warnings promoted to errors:

.\.venv\Scripts\python.exe -X dev -m pytest -q -m concurrency `
  -W "error::ResourceWarning" `
  -W "error::pytest.PytestUnraisableExceptionWarning"

Coherent backend domain suites:

# Accounts and browser/client security
.\.venv\Scripts\python.exe -m pytest tests/accounts -q

# Core runtime, settings, server shell, static, and media behavior
.\.venv\Scripts\python.exe -m pytest tests/core -q

# Library catalog, Groups, bearer reads, models, queries, and Admin
.\.venv\Scripts\python.exe -m pytest tests/library/catalog tests/library/groups tests/library/bearer `
  tests/library/test_models.py tests/library/test_queries.py tests/library/test_roles.py `
  tests/library/test_admin_book_changelist.py tests/library/test_admin_book_group_assignment.py `
  tests/library/test_admin_book_layout.py tests/library/test_admin_catalog_tag.py -q

# Library imports and stored-file repair
.\.venv\Scripts\python.exe -m pytest tests/library/imports `
  tests/library/test_file_repair.py tests/library/test_admin_file_repair.py -q

.\.venv\Scripts\python.exe -m pytest tests/shelves -q
.\.venv\Scripts\python.exe -m pytest tests/marginalia -q
.\.venv\Scripts\python.exe -m pytest tests/testenv -q

Frontend selections use paths instead of a second marker system:

# Focused file, SDK tests, Product UI, then complete Vitest.
npm.cmd --prefix frontend run test:vitest -- --run tests/app/AppBootstrap.test.tsx
npm.cmd --prefix frontend run test:vitest -- --run tests/sdk
npm.cmd --prefix frontend run test:vitest -- --run tests/app tests/components tests/domain tests/features tests/shared
npm.cmd --prefix frontend run test:vitest -- --run

npm.cmd --prefix frontend run typecheck
node frontend/scripts/check-boundaries.mjs
npm.cmd --prefix frontend run build

These commands use the dedicated test root under frontend/tests/, with SDK wire-contract tests under frontend/tests/sdk/ and Product UI tests grouped by their runtime owner. Production SDK source remains under frontend/packages/spl-api/src/.

Use npm instead of npm.cmd on shells where the executable shim is not blocked by PowerShell execution policy.

Hygiene:

.\.venv\Scripts\python.exe -m ruff check tests
.\.venv\Scripts\python.exe tools\static_hygiene.py
.\.venv\Scripts\python.exe tools\static_hygiene.py --all
.\.venv\Scripts\python.exe tools\static_hygiene.py --fix-mojibake
.\.venv\Scripts\python.exe tools\static_hygiene.py --fix-line-endings

tools\static_hygiene.py is the repo check for decorative HTML entities, mojibake markers, trailing whitespace, LF line endings, and git diff --check. By default it checks touched files only and does not modify files. Use --all for a full tracked-file scan. --fix-mojibake, --fix-trailing-whitespace, and --fix-line-endings may modify files in the scan set.

Pyright currently excludes tests in the checked-in configuration. For a report-only type audit, use the local node_modules\.bin\pyright.cmd with a temporary config that includes tests and adds the repository root to extraPaths; do not treat this as a required gate until that setup is checked in deliberately.

Test helpers:

  • Prefer tests/utils/books.py::create_file_backed_book() when a test needs a normal valid Book. The product invariant is that Books are file-backed.

Dev/demo fixture world (local only)

Create a richer development/demo world for manual UI testing:

.\scripts\seed-dev-users.ps1

The command:

  • applies pending migrations before reading the database
  • requires first-run setup to be complete; it does not create the Owner account
  • ensures the configured Public group exists (Common Room on a default server)
  • ensures about 20 lorem-named demo users
  • keeps Manager and Librarian demo accounts as broad-role users with their normal default Common Room membership
  • never assigns curator flags to the Public/Common Room group
  • creates one private Reading Queue and one listed Favorites shelf for each demo user, plus Common Room shelves
  • populates managed personal and Group shelves with small, varied selections of visible or exact-Group Books when Books are available
  • when advanced library groups are enabled, additionally creates five non-Public demo groups by default, varied intentional memberships, curators distributed across Reader, Librarian, and Manager roles where possible, and 3-4 group-owned shelves per Group
  • gives the named demo Groups simple thematic Catalog Tag profiles; a Book is assigned when a normalized profile keyword occurs anywhere in one of its normalized tag names, with deterministic fallback Books for small catalogs

The command preserves existing data by default. Existing users with matching usernames retain their names, email addresses, passwords, flags, and profile roles. Existing shelves are reused without overwriting their descriptions or other fields. Re-running with the same seed does not duplicate memberships, shelves, or shelf items. Existing demo groups are reused when advanced library groups are enabled.

Useful options:

.\scripts\seed-dev-users.ps1 --seed family-demo
.\scripts\seed-dev-users.ps1 --users 12 --groups 3
.\scripts\seed-dev-users.ps1 --skip-shelves

Safety:

  • Newly created demo accounts use the predictable password changeme123.
  • The command refuses to run unless DEBUG=True (use --force only for local development).
  • This command is not the normal installation bootstrap path. Fresh installs should use the first-run setup page. It remains a development/demo helper only.

Local Library import commands

Recurring cleanup and repair commands belong to Operations. The explicit local source modes are:

python backend/manage.py import_library_folder_of_zip "path\to\book.epub"
python backend/manage.py import_library_folder_of_zip "path\to\per-book-archives"
python backend/manage.py import_library_aio_zip "path\to\large-library.zip"
python backend/manage.py import_library_tree "path\to\unpacked-library"

All three commands share the Product/API per-candidate importer and do not create durable import history. See Imports for pairing, safety, large-archive, read-only bind-mount, interruption, and rerun behavior.

See also:

  • docs/api.md (shared conventions and external contracts)
  • docs/imports.md (import workflow)