diff --git a/docs/operations/deployment-basics.md b/docs/operations/deployment-basics.md index 4ad7f36..de6f1f6 100644 --- a/docs/operations/deployment-basics.md +++ b/docs/operations/deployment-basics.md @@ -20,6 +20,8 @@ flowchart LR This page owns that path. The linked operations pages cover process topology, probes, metrics, backups, and security policy in more depth. +This page uses `forj` while preparing source and `./bin/` after an artifact has been built. Production supervisors and release checks should execute the exact binary being deployed, not source-aware development commands. + ## Before You Start Decide which Apps belong in the release and where they will run. Provision the production database, queue, cache, storage, and other external services selected by those Apps. The deployment platform must also own DNS, TLS termination, network policy, process supervision, and secret delivery. @@ -35,7 +37,7 @@ forj build && test -x ./bin/app ``` -Expected result: `bin/app` exists and is executable. The build refreshes generated Project files, runs Wire, prepares the API index, and compiles the default App. +Expected result: `bin/app` exists and is executable. The build refreshes Framework-managed Project files, runs Wire, prepares the API index, and compiles the default App. ### Apps with Web UI @@ -51,17 +53,31 @@ Expected result: `cmd/app/frontend/dist` contains current frontend output and `b ### Additional Apps -Build each independently deployable App: +Build each independently deployable App. If it has frontend source, build that App's frontend first using its path under `cmd//frontend/`: ```bash -forj admin build && +npm --prefix cmd/admin/frontend ci && + npm --prefix cmd/admin/frontend run build && + forj admin build && test -x ./bin/admin ``` -Expected result: `bin/admin` contains the staff-facing App selected by the command prefix. Repeat the build for every App included in the release. +Expected result: `cmd/admin/frontend/dist` contains the current staff frontend and `bin/admin` contains the staff-facing App selected by the command prefix. Omit the npm steps when that App has no frontend source. Repeat the applicable frontend build, App build, and artifact check for every App included in the release. Keep the artifact immutable after it has been tested. If a release is transferred to another host or registry, verify its checksum or image digest before activation. +### Hand Off the Artifact + +The build system should hand the deployment system one identified, immutable release. Record at least: + +- the binary or image digest; +- every App binary included in the release; +- the target operating system and architecture; +- the source revision and build time; and +- any non-secret defaults or overrides compiled with `forj build`. + +Frontend output is already embedded in an App binary after the frontend build and `forj build`; do not deploy an unrelated `dist` directory beside it. Promote the same tested bytes between environments. If configuration or frontend assets require a rebuild, assign the result a new release identity and repeat artifact checks. + ## Keep Configuration Outside the Artifact Supply production configuration through the process environment or the secret and configuration mechanism provided by the deployment platform. @@ -81,6 +97,8 @@ APP_DIAG_TOKEN= The rendered App determines the rest. Configure its selected database, queue, cache, storage, event, mail, and observability drivers with production values. Do not allow a missing production dependency to silently fall back to a process-local driver. +`forj build --env-defaults` and `--env-overrides` are explicit exceptions: they pin non-secret values into the binary. Defaults remain replaceable by deployment configuration; overrides do not. Use defaults only for artifact-level fallbacks and overrides only when every deployment of that artifact must use the same value. A rotated credential, environment endpoint, port, replica-specific identity, or retention setting belongs to the deployment system instead. See [Compiled Environment Values](/reference/configuration#compiled-environment-values) for exact precedence. + Bind to `127.0.0.1` when a reverse proxy on the same host owns public traffic. Bind to `0.0.0.0` only when a container network, firewall, or host network policy controls access. If the App intentionally uses SQLite, local storage, uploads, or another writable filesystem path, place that data outside the versioned release directory and configure an absolute path. Replacing an immutable release must not replace or orphan durable application data. @@ -110,6 +128,16 @@ Changing the command does not change application behavior or make in-memory driv Run one scheduler process unless the schedules use deliberate cross-process locking. See [Runtime Processes](/operations/runtime-processes) before introducing a split topology. +A concrete split deployment might use the same immutable artifact in these supervised process groups: + +| Process group | Replicas | Traffic or work handoff | Scaling signal | +| --- | ---: | --- | --- | +| `./bin/app api` | Two or more | The platform sends HTTP traffic only to ready instances. | Request load, latency, and resource use. | +| `./bin/app worker --queue emails` | One or more | A shared queue backend hands jobs to workers. | Queue depth, job age, failures, and resource use. | +| `./bin/app scheduler` | One | The supervisor maintains singleton ownership unless schedules use a shared lock. | Availability and due-run outcomes, not HTTP load. | + +This is a topology example, not a required replica count. Each group receives its own environment, resource limits, probe configuration, and restart policy. If an App has another binary such as `bin/admin`, model its HTTP, worker, and scheduler roles independently rather than assuming the default App process owns them. + ## Give the Process to a Supervisor The deployment platform should: @@ -125,7 +153,21 @@ The deployment platform should: This contract applies equally to systemd, a container runtime, Kubernetes, Nomad, or another supervisor. The exact service unit or workload manifest belongs to that platform. -Set the supervisor's stop grace period above the longest effective App shutdown path, with additional margin for the supervisor itself. In combined mode, runtime shutdown happens concurrently before the outer App lifecycle finishes. Test shutdown with real in-flight jobs instead of relying only on arithmetic. +### Budget Timeouts as a System + +Timeouts protect different boundaries and should be planned together: + +| Boundary | Example control | What it bounds | +| --- | --- | --- | +| One unit of work | A job timeout or `SCHEDULER_COMMAND_TIMEOUT` | The handler or scheduled command execution. | +| Runtime cleanup | `QUEUE_SHUTDOWN_TIMEOUT` | Queue drain and backend cleanup inside App shutdown. | +| App shutdown | `APP_SHUTDOWN_TIMEOUT` | The HTTP or scheduler graceful-stop path and outer App lifecycle. | +| Readiness request | Probe or `health --timeout-ms` | One operator or platform request, including sequential resource checks. | +| Process supervision | Platform stop grace period | The complete interval before the platform may force termination. | + +The App resolves this policy once at startup. A queue or scheduler subprocess value larger than `APP_SHUTDOWN_TIMEOUT` is capped to the App budget and emits one structured warning with the configured and effective values. Run `./bin/app about` to inspect the effective Runtime settings before changing supervisor limits. + +Do not add every configured duration and assume that sum is the required supervisor value. Some shutdown work is concurrent, some limits are nested, and a handler that ignores cancellation can outlive its intended budget. Set the supervisor's stop grace period above the longest observed graceful-stop path with margin for traffic removal and supervisor overhead. In combined mode, runtime shutdown happens concurrently before the outer App lifecycle finishes. Test `SIGTERM` with real in-flight requests, jobs, and scheduled commands; make interrupted work safe to retry. The long-running command must be the deployed binary: @@ -149,8 +191,30 @@ Expected result: the migration command exits successfully before processes from Run the command once per migration-owning App, not once per HTTP replica. During a rolling deployment, prefer additive schema changes that work with both the old and new binaries. +For an additional migration-owning App, run that App's staged binary independently: + +```bash +/srv/example/releases/2026-07-28/bin/admin migrate +``` + +Expected result: only the `admin` App's migration streams run. Repeat this once for each migration-owning App in the release; do not infer that the default App command migrated additional Apps. + Use stable paths for backup output rather than writing backup sets inside a versioned release directory. [Backup and Restore](/operations/backups) covers discovery, verification, retention, and restore safeguards. +## Hand Off Observability and Retention + +The App emits signals; the deployment platform and operators own their transport, access, and retention. Before activation, assign each signal to an operational destination: + +| Signal or data | App/process responsibility | Deployment responsibility | +| --- | --- | --- | +| Logs | Write structured runtime output to standard output and standard error. | Collect, index, redact, retain, and alert on it. | +| Metrics | Expose the endpoint appropriate to combined or split topology. | Scrape it with bounded labels, retain time series, and define alerts. | +| Health and readiness | Report process liveness and required dependency state. | Route probes correctly and remove unready HTTP instances from traffic. | +| Inspects and Lighthouse | Capture and present bounded recent execution detail when enabled. | Restrict operator access and choose capture, sampling, and recent-window limits. | +| Backups and durable data | Use the configured stable paths and backends. | Schedule backups, apply retention, verify restore, and keep data outside release directories. | + +Preserve the app name, runtime role, release identity, and instance identity in the surrounding platform metadata so an alert can be traced to the exact process and artifact. Set `APP_VERSION` and `APP_REVISION` while building framework-managed metrics discovery, and apply equivalent bounded labels in an external production scraper. Lighthouse's recent Inspect window is not a substitute for retained logs, metrics, or backups. + ## Activate the Release A useful filesystem layout separates immutable releases from mutable state: @@ -246,12 +310,14 @@ Queue payloads are another compatibility boundary. A previous worker binary must - Build every App for the deployment target. - Build frontend assets before `forj build` when the App has Web UI. - Store production configuration and secrets outside the artifact. +- Record any non-secret defaults or overrides intentionally compiled into it. - Use production drivers for state shared across processes or hosts. - Keep writable data and backup sets outside immutable release directories. - Run the staged release's migrations once before it receives traffic. - Supervise each required process with the deployed binary. - Keep the scheduler singleton unless locking makes overlap safe. - Set and test graceful shutdown budgets. +- Assign logs, metrics, Inspects, and backups explicit access and retention owners. - Verify liveness, readiness, metrics, and one application workflow. - Keep the previous artifact available for binary rollback. - Treat database migrations and queued payloads as separate rollback contracts. diff --git a/docs/operations/http-server.md b/docs/operations/http-server.md index ec821e2..3cb8d1d 100644 --- a/docs/operations/http-server.md +++ b/docs/operations/http-server.md @@ -77,6 +77,20 @@ For a staff operations App named `admin`, use its binary: Expected result: a complete, human-readable table of the staff-facing methods, paths, and handlers registered by `admin`. The command constructs the App route surface but does not start the HTTP listener. +A small route table looks like this (terminal output uses color when supported): + +```text ++-------------------+---------+-----------------------+------------+ +| API Routes › (1) ++-------------------+---------+-----------------------+------------+ +| Path | Methods | Handler | Middleware | ++-------------------+---------+-----------------------+------------+ +| /api/v1/users/:id | GET | users.Controller.Show | | ++-------------------+---------+-----------------------+------------+ +``` + +Long middleware names may be replaced by short codes with a `Middleware Legend` above the route title. The columns remain `Path`, `Methods`, `Handler`, and `Middleware`. + ## Health, Readiness, and Verification Use liveness to answer whether the process is responding and readiness to decide whether it should receive traffic: diff --git a/docs/operations/index.md b/docs/operations/index.md index 4da6218..564abe2 100644 --- a/docs/operations/index.md +++ b/docs/operations/index.md @@ -19,6 +19,7 @@ Use these guides when you need to run, split, observe, deploy, or recover an App | Configure probes | [Health and Readiness](/operations/health-readiness) | | Investigate runtime behavior | [Logging](/operations/logging), [Metrics](/operations/metrics), and [Inspects](/operations/inspects) | | Use the operator interface | [Lighthouse](/operations/lighthouse) | +| Compare HTTP and infrastructure performance safely | [Performance Benchmarks](/operations/performance-benchmarks) | | Protect and recover durable state | [Backup and Restore](/operations/backups) | | Build, roll out, verify, and roll back a release | [Deploy an App](/operations/deployment-basics) | diff --git a/docs/operations/logging.md b/docs/operations/logging.md index 5350812..9869d21 100644 --- a/docs/operations/logging.md +++ b/docs/operations/logging.md @@ -81,6 +81,22 @@ Each request event retains named fields for URI, method, status, latency, and cl Console output uses a compact value-oriented line with status-aware color. JSON output and registered log sinks retain the structured field names, so machine processing does not depend on console formatting. +An App logger can route one structured entry to several destinations. `AddSink` preserves the simple synchronous callback API. Use `AddSinkWithOptions` when a destination needs error isolation or a bounded asynchronous queue: + + +```go +if err := appLogger.AddSinkWithOptions(auditSink, logger.SinkOptions{ + Name: "audit", + Async: true, + QueueCapacity: 256, + Overflow: logger.SinkOverflowDropOldest, +}); err != nil { + return err +} +``` + +Registrations append; they do not replace process output or an earlier sink. Synchronous sinks run in the logging path, so keep them fast. Asynchronous sinks preserve accepted entries in FIFO order and make overload behavior explicit with `block`, `drop-newest`, or `drop-oldest`. Flush managed sinks from an App shutdown hook with `appLogger.FlushSinks(ctx)` so accepted entries get the remaining shutdown budget. + Disable access logs for a runtime where request volume would hide higher-signal events, then rely on metrics and inspects for the intended visibility. Keep access logs enabled during a new deployment until the request path, status, and latency are understood. If you disable them for steady-state volume, retain an explicit route-level metric and a safe Inspect sampling policy; otherwise a 5xx increase has no request-level path back to an operator. @@ -97,7 +113,16 @@ APP_LOG_TIME Use structured fields that answer an operational question: App identity, Runtime source, route pattern, queue or schedule name, status, and latency. The request context can carry the `trace_id` correlation field used by Inspects, but the product surface is still called an Inspect. -Never log authorization headers, cookies, credentials, raw queue payloads, or unredacted request bodies by default. Local HTTP error-response capture is intentionally local-environment behavior; do not rely on it as a production payload-dump mechanism. +Never log authorization headers, cookies, credentials, raw queue payloads, or unredacted request bodies by default. Before deduplication, process output, or sink delivery, the App logger redacts common secret-bearing field names and high-confidence message forms such as bearer tokens and `password=...` assignments. + +Extend that mandatory policy for application-specific data: + +```dotenv +APP_LOG_REDACT_KEYS=session_id,customer_reference +APP_LOG_REDACT_MESSAGE_PATTERNS=["client_secret=[^ ]+","session=[^ ]+"] +``` + +The first value is a comma-separated list of additional case-insensitive field names. The second is a JSON array of regular expressions; invalid JSON or expressions fail during logger construction instead of silently weakening the policy. These controls supplement the framework defaults rather than disabling them. Redaction is a safety net, not permission to log whole payloads. Local HTTP error-response capture remains intentionally local-environment behavior. ## Failure Modes diff --git a/docs/operations/metrics.md b/docs/operations/metrics.md index 7a14e7e..205d637 100644 --- a/docs/operations/metrics.md +++ b/docs/operations/metrics.md @@ -113,6 +113,25 @@ environment=local This label set identifies a background staff-operations job from `admin`. Use `source` for logical runtime attribution and `process` for scrape topology. +## Deployment Markers + +Set release identity in the Project-owned environment used to generate the deployment artifact: + +```dotenv +APP_VERSION=v1.8.0 +APP_REVISION=8d5c20f +``` + +Then build the artifact: + +```bash +forj build +``` + +When present, `APP_VERSION` becomes the bounded `release` target label and `APP_REVISION` becomes `revision`. Blank values are omitted. Keep both values low-cardinality: use one release version and one immutable commit or artifact revision, not a request ID or build timestamp. This lets a dashboard annotation or deployment comparison separate a real regression from ordinary traffic movement. + +Generation intentionally reads `.env.example` and `.env`, not unrelated ambient shell variables, so the discovery file and the artifact are derived from one reviewable Project snapshot. These labels are written into framework-managed local discovery files during generation. In an externally managed production scraper, apply the same release and revision labels in that platform's service-discovery configuration. + Avoid user IDs, emails, raw URLs, raw SQL, cache keys, filenames, request IDs, and arbitrary error strings. ## Proving Path diff --git a/docs/operations/performance-benchmarks.md b/docs/operations/performance-benchmarks.md new file mode 100644 index 0000000..5c5b48d --- /dev/null +++ b/docs/operations/performance-benchmarks.md @@ -0,0 +1,98 @@ +--- +title: Performance Benchmarks +description: Run and interpret Lighthouse browser benchmarks for HTTP, cache, queues, storage, and databases. +--- + +# Performance Benchmarks + +Lighthouse can send controlled workload through a connected App and compare HTTP or configured infrastructure resources from the browser. + +Use this page to establish a reproducible baseline, compare an intentional driver or configuration change, or find the point where higher concurrency stops helping. These benchmarks measure the selected suite under the selected conditions; they do not predict whole-application capacity or define a service-level objective. + +## What Lighthouse Can Measure + +The Benchmarks view asks a connected benchmark-capable agent for its catalog. The available rows follow the components compiled into that App and the resource instances its Runtime discovers. + +| Suite | Target | Work performed | +| --- | --- | --- | +| HTTP | Configured benchmark URL and path | Repeated `GET` requests over the configured duration. | +| Cache | One configured cache instance | Benchmark-owned keys and payloads against that Driver. | +| Queue | One configured queue instance | Benchmark jobs dispatched and drained through the selected queue. | +| Storage | One configured storage disk | Benchmark-owned paths and payloads against that Driver. | +| Database | One configured connection | Operations against a benchmark table created when needed. | + +Lighthouse does not show a cache, storage, or database suite merely because the UI knows its name. The App must compile the component and expose a usable instance. Queue is part of the benchmark runner, but a meaningful queue run also needs workers consuming the selected benchmark queue. + +## Before You Run + +Treat a benchmark as an operator change, not a read-only diagnostic. + +1. Prefer an isolated or representative environment with the same topology you want to compare. +2. Select the exact App, Runtime agent, resource instance, and Driver. +3. Confirm the HTTP URL and path or queue name before sending work. +4. Record the release, agent instance, duration, concurrency, payload size, sweep limit, and surrounding load. +5. Watch normal CPU, memory, network, connection-pool, broker, database, storage, error, and saturation signals during the run. + +::: warning Active workload +The HTTP suite repeatedly sends `GET` requests. Do not point it at a route that mutates data or triggers expensive side effects. Resource suites use run-scoped benchmark keys, paths, jobs, and records and attempt cleanup, but a crash, cancellation, backend error, or unavailable cleanup operation can leave data or queued work behind. The database suite can create its benchmark table. +::: + +## Run a Baseline in Lighthouse + +Open **Benchmarks** in Lighthouse, then: + +1. Confirm the selected agent. Lighthouse prefers a connected jobs Runtime when one is available because the generated benchmark runner lives with App job infrastructure. +2. Select one target first. Each configured cache, queue, storage, or database instance appears as its own target. +3. Keep the initial duration, concurrency, and payload settings unchanged so the first result is a baseline rather than a tuning experiment. +4. Select **Run Baseline** and keep the page attached until the target finishes. +5. Save the target identity, system baseline, result details, and any errors with your change record. + +Expected result: the page reports the suite and Driver, configured duration and concurrency, actual elapsed time, operation count, operations per second, errors, and p50, p95, and p99 latency. Suite-specific detail and system information appear with the report. + +If the target does not appear, confirm that the component and resource instance belong to the selected App. If a queue run stalls, confirm workers are consuming the selected queue before increasing the drain timeout or concurrency. + +## Repeat the Baseline from the Artifact + +The built App exposes the same generated runner for repeatable release checks. For an HTTP-only baseline: + +```bash +./bin/app benchmark:run \ + --suites http \ + --duration-ms 15000 \ + --concurrency 8 \ + --payload-size 512 +``` + +Expected result: the command prints the system baseline, one HTTP result row, and suite details with throughput, operation count, errors, p50, p95, p99, and elapsed time. Add `--json` when a comparison pipeline needs structured output. The command runs the same App-owned suite as Lighthouse; it does not make a production target safer. + +## Compare One Change + +Change one material variable at a time: release, Driver, backend location, pool size, payload, or concurrency. Keep the remaining conditions stable and repeat enough runs to distinguish a durable change from environmental noise. + +Use a concurrency sweep to explore where throughput stops improving or errors and tail latency begin to rise. A sweep is still a synthetic comparison: it does not model production request mixes, think time, cache warmth, or contention from unrelated workloads. + +## Interpret the Report + +Read the result as one evidence bundle: + +- `ops/sec` is achieved throughput for that suite, target, and configuration—not an App-wide total, capacity guarantee, or SLO. +- p50, p95, and p99 describe the observed successful timing distribution. Read them with operation count and errors; a low percentile does not excuse failed work. +- elapsed time can differ from configured duration because setup, drain, cleanup, or backend behavior belongs to the run. +- Driver and instance identity matter. Results from two differently located backends are not interchangeable even when their logical resource name matches. +- the system baseline matters. CPU topology, memory, virtualization, co-located work, and network distance can dominate a small code change. + +Do not add operations-per-second values from different targets and call the sum application throughput. The suites run distinct operations against distinct resources. + +## Confirm with Production-Shaped Evidence + +Use Lighthouse benchmarks to form or reject a focused hypothesis. Before changing capacity, timeouts, or service objectives, confirm the conclusion with production-shaped load testing and the App's ordinary metrics, logs, readiness, and Inspects. + +After a production-authorized run, inspect the selected backend for benchmark records or queued work that cleanup could not remove. + +## Related + +- [Lighthouse](/operations/lighthouse) +- [Metrics](/operations/metrics) +- [Inspects](/operations/inspects) +- [Queue Workers](/operations/queue-workers) +- [Environment Reference](/reference/env-vars#demo-app) diff --git a/docs/operations/runtime-processes.md b/docs/operations/runtime-processes.md index ae1e256..1449d33 100644 --- a/docs/operations/runtime-processes.md +++ b/docs/operations/runtime-processes.md @@ -69,7 +69,7 @@ Common variables: ```text APP_SHUTDOWN_TIMEOUT=30s QUEUE_SHUTDOWN_TIMEOUT=10s -SCHEDULER_SUBPROCESS_SHUTDOWN_TIMEOUT=90s +SCHEDULER_SUBPROCESS_SHUTDOWN_TIMEOUT=30s ``` Constructors should build dependencies, not start long-running work. Keep business behavior independent of process topology, allow workers time to finish in-flight jobs, and use locking or singleton control before running multiple schedulers. Splitting processes does not provide shared state or job correctness by itself.