Feat/restore from sqlite backup

README.md

@@ -94,7 +94,7 @@ The notification bell reports a drop of at least five positions from a previous
 
 Open **Listing history → Track a listing**. Collection is off by default. Select an app, store, country, and frequency; Google Play also supports a language choice. The credit estimate appears before you enable tracking. Each scheduled check uses one SerpApi product request, with an initial baseline check when tracking starts. Manual checks and retries can consume additional credits.
 
-The timeline highlights changes to the fields returned by the store, including titles, descriptions, versions, pricing, and images. Text comparisons highlight additions and removals. Screenshots can be compared in order, with added and moved images labeled. Supported images are archived locally in the database and included in backups. If an image cannot be archived, the comparison shows an unavailable-image placeholder. AppTrail never substitutes the live image for a missing historical image.
+The timeline highlights changes to the fields returned by the store, including titles, descriptions, versions, pricing, and images. Text comparisons highlight additions and removals. Screenshots can be compared in order, with added and moved images labeled. Supported images are archived locally in the database. Downloaded backups omit archived images and raw SerpApi responses to save space, while keeping saved results, matched evidence, listing text, and image-change records. After restoring, AppTrail explains why omitted images and raw responses are unavailable. It never substitutes a live image for a missing historical image.
 
 Use **Manage** to change frequency, pause or resume collection, or request a manual check. Pausing preserves history. Unchanged checks are hidden until you select **Show unchanged checks**. Collection requires the AppTrail process to remain running, and begins when you enable it; earlier listing versions cannot be reconstructed.
 

docs/AUTHENTICATION.md

@@ -12,6 +12,8 @@ The server generates a random one-time setup code, stores it in a permission-pro
 
 Choose a username of 3–64 letters, numbers, dots, underscores, or hyphens, starting with a letter or number. Usernames are case-insensitive. Passwords require 8–128 characters, including at least one number (0–9) and one special character, such as `!` or `@`. Spaces are preserved but do not count as special characters.
 
+If you already have a SQLite backup, select **Have a backup? Restore your data** below **Create account**. Enter this server's setup code and upload the backup. After restoration, sign in with the username and password saved in the backup. This option is available only before an owner account exists.
+
 ## Sessions and requests
 
 A successful login issues a random cookie with `HttpOnly`, `SameSite=Strict`, and a host-only scope. Requests that AppTrail sees as HTTPS also use `Secure` and the `__Host-` cookie prefix. A proxy can forward the original scheme through the optional [forwarded-header settings](SELF_HOSTING.md#public-https-hosting). Sessions are bound to the scheme, host, and port seen by AppTrail when signing in. SQLite stores a hash of the session token, not the token itself. Tokens are not stored in browser local storage.
@@ -20,10 +22,12 @@ Sessions last 24 hours from sign-in, including time spent away from the app. The
 
 Every workspace route requires a valid session, including reads, searches, discovery, key changes, exports, backups, and the dashboard HTML. Only the login/setup page, its static assets, authentication status and entry endpoints, and the minimal health probe are public. Data responses use `Cache-Control: no-store`.
 
-Writes require JSON, a custom request header, and a CSRF token tied to the authenticated session. Login and setup require JSON and the custom header before a session exists. Requests marked `cross-site` by the browser's `Sec-Fetch-Site` header are rejected, and no cross-origin access is enabled through CORS. AppTrail does not compare the browser's `Origin` header with the internal server address, so an HTTPS proxy can forward HTTP without blocking account setup or login. Login, setup, and password changes share a persisted limit of ten attempts per client address per five minutes and a global limit of 100 per minute. See the proxy instructions before forwarding client addresses.
+Writes require JSON, a custom request header, and a CSRF token tied to the authenticated session. SQLite restoration accepts a binary upload with `Content-Type: application/vnd.sqlite3` and `X-AppTrail-Confirm-Restore: overwrite`, with the same session, custom header, and CSRF checks. Login and setup require JSON and the custom header before a session exists. Requests marked `cross-site` by the browser's `Sec-Fetch-Site` header are rejected, and no cross-origin access is enabled through CORS. AppTrail does not compare the browser's `Origin` header with the internal server address, so an HTTPS proxy can forward HTTP without blocking account setup or login. Login, setup, and password changes share a persisted limit of ten attempts per client address per five minutes and a global limit of 100 per minute. See the proxy instructions before forwarding client addresses.
 
 These controls follow the [OWASP password storage](https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html), [session management](https://cheatsheetseries.owasp.org/cheatsheets/Session_Management_Cheat_Sheet.html), and [CSRF prevention](https://cheatsheetseries.owasp.org/cheatsheets/Cross-Site_Request_Forgery_Prevention_Cheat_Sheet.html) guidance. They are covered by automated tests and browser checks, not an independent security assessment. There is no MFA or SSO in this version.
 
+Before account creation, `POST /api/auth/restore` accepts the same SQLite upload and overwrite confirmation with `X-AppTrail-Setup-Token` containing the server's setup code, in place of a session and CSRF token. It requires the custom request header and rejects cross-site requests. Attempts share the setup and login rate limits. AppTrail checks that no owner exists both before accepting the upload and immediately before restoring, then removes the setup code after success.
+
 ## API clients
 
 Use an HTTP client with a cookie jar. Submit JSON to `POST /api/auth/login` with `username` and `password`, plus `X-AppTrail-Request: 1` and `Content-Type: application/json`. Retain the returned cookie and `csrf_token`. Send the cookie on subsequent requests; writes also need `X-CSRF-Token` with that value and the same JSON/custom headers. `GET /api/auth/status` returns the current session's CSRF token. Never place passwords or session tokens in URLs.
@@ -32,6 +36,8 @@ HTTP 401 means authentication is required. HTTP 403 indicates a failed setup cod
 
 ## Recovery and background work
 
+Restoring a SQLite backup in Settings replaces the owner account and password with those saved in the upload. It revokes all current and uploaded sessions. Sign in again with the restored credentials. Invalid uploads leave the current account and sessions intact. See [backups and restoration](SELF_HOSTING.md#backups-and-restoration) for compatibility limits and recovery copies.
+
 Change a known password in Settings. Recover a forgotten password with `apptrail --reset-password` on the server, using the same data directory and operating-system user. Stop the AppTrail service first. Server filesystem access is required; there is no unauthenticated web password-reset endpoint.
 
 The worker waits for an owner account before processing searches. After setup, scheduled monitoring continues while you are signed out. Logging out stops browser access, not scheduled work. Pause queries in the dashboard to stop monitoring.

docs/DEVELOPMENT.md

@@ -28,6 +28,8 @@ Charts use the bundled Chart.js 4.4.9 distribution in `static/vendor/chart.umd.m
 | `src/apptrail/cli.py` | Console command, free-port binding, browser launch |
 | `src/apptrail/api.py` | FastAPI routes and input validation |
 | `src/apptrail/auth.py` | Owner setup, password hashing, session validation, CSRF, and login throttling |
+| `src/apptrail/backups.py` | Restore validation, schema compatibility, and exclusion of concurrent database users during restore |
+| `src/apptrail/storage.py` | Storage measurements, age-based response and image cleanup, and database compaction |
 | `src/apptrail/config.py` | Data directory and credentials |
 | `src/apptrail/db.py` | SQLAlchemy models and numbered SQLite schema migrations |
 | `src/apptrail/engines.py` | Official SerpApi SDK requests and source normalization |
@@ -75,8 +77,8 @@ Never commit keys, credentials, `.env` files, databases, or unredacted provider
 
 ```bash
 uv build
-uvx --from ./dist/apptrail-0.4.1-py3-none-any.whl apptrail --version
-uvx --from ./dist/apptrail-0.4.1-py3-none-any.whl apptrail --no-browser
+uvx --from ./dist/apptrail-1.0.0-py3-none-any.whl apptrail --version
+uvx --from ./dist/apptrail-1.0.0-py3-none-any.whl apptrail --no-browser
 ```
 
 The wheel includes the static UI. Verify it from outside the checkout. The command's data directory is independent of the installed package and uv tool cache.
@@ -93,6 +95,10 @@ The separate live workflow runs on the same events and can be started manually.
 
 SQLite uses WAL mode, foreign keys, and a busy timeout. Partial unique indexes prevent duplicate queued/running jobs for the same query or listing. A process lock allows one active AppTrail worker per data directory. Do not run multiple Uvicorn workers against the same workspace.
 
+Downloaded backups call `Database.backup(compact=True, include_sessions=False)` to remove raw response JSON, archived image blobs, and sessions from the copy, then run `VACUUM` on that copy. Normalized results and snapshot image hashes remain. The `backup_omissions` setting records the last affected run and snapshot IDs so restored views can explain missing content without labelling new checks as incomplete. Internal recovery and migration snapshots use full backups. Restore supports the current schema and schema 6, whose image comparison upgrade does not change the table layout.
+
+Storage cleanup uses the same request gate as restore and stops the worker before deleting content and running `VACUUM` plus a WAL checkpoint. It only clears responses for completed runs before the selected cutoff. Image age comes from the latest snapshot reference across every media field, so deduplicated assets used by newer snapshots survive. Unreferenced assets are also removed. The `storage_cleanup` setting records each category's cutoff for missing-content notices. Measurements read blob and JSON byte lengths without loading their contents into Python; snapshot media references are indexed in a temporary table for shared-image checks.
+
 Schema 4 stores normalized results and provider responses in `run_payloads`, separate from the small `runs` records used for polling and scheduling. Load payloads only for evidence, matching, or reanalysis. State and dashboard queries must not fetch them. The upgrade preserves a snapshot in `backups/before-schema-3.sqlite3` before moving existing payloads.
 
 Schema 5 adds notifications, per-query alert state, regional listing watches, listing snapshots, and archived image blobs. Existing workspaces receive no enabled listing watches. The migration backs up a schema-4 database to `backups/before-schema-4.sqlite3`. Listing-history jobs use `watch_id` and a separate partial unique index so different countries can be collected independently. Images share content hashes to avoid storing identical bytes repeatedly, and authenticated asset routes serve them from the database.
@@ -134,7 +140,7 @@ The workflow name is the filename, without `.github/workflows/`. If the project
 
 1. Update `pyproject.toml` and `src/apptrail/__init__.py` to the same version, then run `uv lock` to update `uv.lock`.
 2. Merge those changes and the workflows into `main`, and wait for the CI and live test workflows to pass.
-3. Create and publish a GitHub Release with a tag of `v<version>` targeting `main`, for example `v0.4.1` for package version `0.4.1`. Use a version that has not already been published to PyPI.
+3. Create and publish a GitHub Release with a tag of `v<version>` targeting `main`, for example `v1.0.0` for package version `1.0.0`. Use a version that has not already been published to PyPI.
 
 The [Publish to PyPI workflow](https://github.com/serpapi/apptrail/actions/workflows/publish.yml) starts when the release is published, including published prereleases. Saving a draft or pushing a tag alone does not publish a package. Tags without a `v` prefix are ignored; mismatched versions and commits outside `main` fail validation. The workflow reruns Python, JavaScript, and Chromium tests, builds with `uv build`, and smoke-tests both the wheel and source distribution before uploading those artifacts to PyPI. Live tests run separately and are not a publishing-job dependency.