diff --git a/README.md b/README.md index a683958..bf13af4 100644 --- a/README.md +++ b/README.md @@ -94,7 +94,7 @@ The notification bell reports a drop of at least five positions from a previous Open **Listing history → Track a listing**. Collection is off by default. Select an app, store, country, and frequency; Google Play also supports a language choice. The credit estimate appears before you enable tracking. Each scheduled check uses one SerpApi product request, with an initial baseline check when tracking starts. Manual checks and retries can consume additional credits. -The timeline highlights changes to the fields returned by the store, including titles, descriptions, versions, pricing, and images. Text comparisons highlight additions and removals. Screenshots can be compared in order, with added and moved images labeled. Supported images are archived locally in the database and included in backups. If an image cannot be archived, the comparison shows an unavailable-image placeholder. AppTrail never substitutes the live image for a missing historical image. +The timeline highlights changes to the fields returned by the store, including titles, descriptions, versions, pricing, and images. Text comparisons highlight additions and removals. Screenshots can be compared in order, with added and moved images labeled. Supported images are archived locally in the database. Downloaded backups omit archived images and raw SerpApi responses to save space, while keeping saved results, matched evidence, listing text, and image-change records. After restoring, AppTrail explains why omitted images and raw responses are unavailable. It never substitutes a live image for a missing historical image. Use **Manage** to change frequency, pause or resume collection, or request a manual check. Pausing preserves history. Unchanged checks are hidden until you select **Show unchanged checks**. Collection requires the AppTrail process to remain running, and begins when you enable it; earlier listing versions cannot be reconstructed. diff --git a/docs/AUTHENTICATION.md b/docs/AUTHENTICATION.md index 6a95d51..f2043ef 100644 --- a/docs/AUTHENTICATION.md +++ b/docs/AUTHENTICATION.md @@ -12,6 +12,8 @@ The server generates a random one-time setup code, stores it in a permission-pro Choose a username of 3–64 letters, numbers, dots, underscores, or hyphens, starting with a letter or number. Usernames are case-insensitive. Passwords require 8–128 characters, including at least one number (0–9) and one special character, such as `!` or `@`. Spaces are preserved but do not count as special characters. +If you already have a SQLite backup, select **Have a backup? Restore your data** below **Create account**. Enter this server's setup code and upload the backup. After restoration, sign in with the username and password saved in the backup. This option is available only before an owner account exists. + ## Sessions and requests A successful login issues a random cookie with `HttpOnly`, `SameSite=Strict`, and a host-only scope. Requests that AppTrail sees as HTTPS also use `Secure` and the `__Host-` cookie prefix. A proxy can forward the original scheme through the optional [forwarded-header settings](SELF_HOSTING.md#public-https-hosting). Sessions are bound to the scheme, host, and port seen by AppTrail when signing in. SQLite stores a hash of the session token, not the token itself. Tokens are not stored in browser local storage. @@ -20,10 +22,12 @@ Sessions last 24 hours from sign-in, including time spent away from the app. The Every workspace route requires a valid session, including reads, searches, discovery, key changes, exports, backups, and the dashboard HTML. Only the login/setup page, its static assets, authentication status and entry endpoints, and the minimal health probe are public. Data responses use `Cache-Control: no-store`. -Writes require JSON, a custom request header, and a CSRF token tied to the authenticated session. Login and setup require JSON and the custom header before a session exists. Requests marked `cross-site` by the browser's `Sec-Fetch-Site` header are rejected, and no cross-origin access is enabled through CORS. AppTrail does not compare the browser's `Origin` header with the internal server address, so an HTTPS proxy can forward HTTP without blocking account setup or login. Login, setup, and password changes share a persisted limit of ten attempts per client address per five minutes and a global limit of 100 per minute. See the proxy instructions before forwarding client addresses. +Writes require JSON, a custom request header, and a CSRF token tied to the authenticated session. SQLite restoration accepts a binary upload with `Content-Type: application/vnd.sqlite3` and `X-AppTrail-Confirm-Restore: overwrite`, with the same session, custom header, and CSRF checks. Login and setup require JSON and the custom header before a session exists. Requests marked `cross-site` by the browser's `Sec-Fetch-Site` header are rejected, and no cross-origin access is enabled through CORS. AppTrail does not compare the browser's `Origin` header with the internal server address, so an HTTPS proxy can forward HTTP without blocking account setup or login. Login, setup, and password changes share a persisted limit of ten attempts per client address per five minutes and a global limit of 100 per minute. See the proxy instructions before forwarding client addresses. These controls follow the [OWASP password storage](https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html), [session management](https://cheatsheetseries.owasp.org/cheatsheets/Session_Management_Cheat_Sheet.html), and [CSRF prevention](https://cheatsheetseries.owasp.org/cheatsheets/Cross-Site_Request_Forgery_Prevention_Cheat_Sheet.html) guidance. They are covered by automated tests and browser checks, not an independent security assessment. There is no MFA or SSO in this version. +Before account creation, `POST /api/auth/restore` accepts the same SQLite upload and overwrite confirmation with `X-AppTrail-Setup-Token` containing the server's setup code, in place of a session and CSRF token. It requires the custom request header and rejects cross-site requests. Attempts share the setup and login rate limits. AppTrail checks that no owner exists both before accepting the upload and immediately before restoring, then removes the setup code after success. + ## API clients Use an HTTP client with a cookie jar. Submit JSON to `POST /api/auth/login` with `username` and `password`, plus `X-AppTrail-Request: 1` and `Content-Type: application/json`. Retain the returned cookie and `csrf_token`. Send the cookie on subsequent requests; writes also need `X-CSRF-Token` with that value and the same JSON/custom headers. `GET /api/auth/status` returns the current session's CSRF token. Never place passwords or session tokens in URLs. @@ -32,6 +36,8 @@ HTTP 401 means authentication is required. HTTP 403 indicates a failed setup cod ## Recovery and background work +Restoring a SQLite backup in Settings replaces the owner account and password with those saved in the upload. It revokes all current and uploaded sessions. Sign in again with the restored credentials. Invalid uploads leave the current account and sessions intact. See [backups and restoration](SELF_HOSTING.md#backups-and-restoration) for compatibility limits and recovery copies. + Change a known password in Settings. Recover a forgotten password with `apptrail --reset-password` on the server, using the same data directory and operating-system user. Stop the AppTrail service first. Server filesystem access is required; there is no unauthenticated web password-reset endpoint. The worker waits for an owner account before processing searches. After setup, scheduled monitoring continues while you are signed out. Logging out stops browser access, not scheduled work. Pause queries in the dashboard to stop monitoring. diff --git a/docs/DEVELOPMENT.md b/docs/DEVELOPMENT.md index 1656d05..f3a4c12 100644 --- a/docs/DEVELOPMENT.md +++ b/docs/DEVELOPMENT.md @@ -28,6 +28,8 @@ Charts use the bundled Chart.js 4.4.9 distribution in `static/vendor/chart.umd.m | `src/apptrail/cli.py` | Console command, free-port binding, browser launch | | `src/apptrail/api.py` | FastAPI routes and input validation | | `src/apptrail/auth.py` | Owner setup, password hashing, session validation, CSRF, and login throttling | +| `src/apptrail/backups.py` | Restore validation, schema compatibility, and exclusion of concurrent database users during restore | +| `src/apptrail/storage.py` | Storage measurements, age-based response and image cleanup, and database compaction | | `src/apptrail/config.py` | Data directory and credentials | | `src/apptrail/db.py` | SQLAlchemy models and numbered SQLite schema migrations | | `src/apptrail/engines.py` | Official SerpApi SDK requests and source normalization | @@ -75,8 +77,8 @@ Never commit keys, credentials, `.env` files, databases, or unredacted provider ```bash uv build -uvx --from ./dist/apptrail-0.4.1-py3-none-any.whl apptrail --version -uvx --from ./dist/apptrail-0.4.1-py3-none-any.whl apptrail --no-browser +uvx --from ./dist/apptrail-1.0.0-py3-none-any.whl apptrail --version +uvx --from ./dist/apptrail-1.0.0-py3-none-any.whl apptrail --no-browser ``` The wheel includes the static UI. Verify it from outside the checkout. The command's data directory is independent of the installed package and uv tool cache. @@ -93,6 +95,10 @@ The separate live workflow runs on the same events and can be started manually. SQLite uses WAL mode, foreign keys, and a busy timeout. Partial unique indexes prevent duplicate queued/running jobs for the same query or listing. A process lock allows one active AppTrail worker per data directory. Do not run multiple Uvicorn workers against the same workspace. +Downloaded backups call `Database.backup(compact=True, include_sessions=False)` to remove raw response JSON, archived image blobs, and sessions from the copy, then run `VACUUM` on that copy. Normalized results and snapshot image hashes remain. The `backup_omissions` setting records the last affected run and snapshot IDs so restored views can explain missing content without labelling new checks as incomplete. Internal recovery and migration snapshots use full backups. Restore supports the current schema and schema 6, whose image comparison upgrade does not change the table layout. + +Storage cleanup uses the same request gate as restore and stops the worker before deleting content and running `VACUUM` plus a WAL checkpoint. It only clears responses for completed runs before the selected cutoff. Image age comes from the latest snapshot reference across every media field, so deduplicated assets used by newer snapshots survive. Unreferenced assets are also removed. The `storage_cleanup` setting records each category's cutoff for missing-content notices. Measurements read blob and JSON byte lengths without loading their contents into Python; snapshot media references are indexed in a temporary table for shared-image checks. + Schema 4 stores normalized results and provider responses in `run_payloads`, separate from the small `runs` records used for polling and scheduling. Load payloads only for evidence, matching, or reanalysis. State and dashboard queries must not fetch them. The upgrade preserves a snapshot in `backups/before-schema-3.sqlite3` before moving existing payloads. Schema 5 adds notifications, per-query alert state, regional listing watches, listing snapshots, and archived image blobs. Existing workspaces receive no enabled listing watches. The migration backs up a schema-4 database to `backups/before-schema-4.sqlite3`. Listing-history jobs use `watch_id` and a separate partial unique index so different countries can be collected independently. Images share content hashes to avoid storing identical bytes repeatedly, and authenticated asset routes serve them from the database. @@ -134,7 +140,7 @@ The workflow name is the filename, without `.github/workflows/`. If the project 1. Update `pyproject.toml` and `src/apptrail/__init__.py` to the same version, then run `uv lock` to update `uv.lock`. 2. Merge those changes and the workflows into `main`, and wait for the CI and live test workflows to pass. -3. Create and publish a GitHub Release with a tag of `v` targeting `main`, for example `v0.4.1` for package version `0.4.1`. Use a version that has not already been published to PyPI. +3. Create and publish a GitHub Release with a tag of `v` targeting `main`, for example `v1.0.0` for package version `1.0.0`. Use a version that has not already been published to PyPI. The [Publish to PyPI workflow](https://github.com/serpapi/apptrail/actions/workflows/publish.yml) starts when the release is published, including published prereleases. Saving a draft or pushing a tag alone does not publish a package. Tags without a `v` prefix are ignored; mismatched versions and commits outside `main` fail validation. The workflow reruns Python, JavaScript, and Chromium tests, builds with `uv build`, and smoke-tests both the wheel and source distribution before uploading those artifacts to PyPI. Live tests run separately and are not a publishing-job dependency. diff --git a/docs/SELF_HOSTING.md b/docs/SELF_HOSTING.md index 2ad0b05..b14547a 100644 --- a/docs/SELF_HOSTING.md +++ b/docs/SELF_HOSTING.md @@ -24,9 +24,48 @@ The image supports Intel/AMD and ARM Linux and uses port `80`. If that port is a ### CapRover and Coolify -Deploy `serpapi/apptrail:latest`. Mount persistent storage at `/data` and run one instance. Use the setup code in the application logs to create your owner account. If configuring a platform health check, use HTTP `/healthz`. +Use the Docker Hub image `serpapi/apptrail:latest` and configure persistent storage at `/data` before the first deployment. AppTrail stores your account, history, and saved credentials there. Reusing this storage keeps your data when an update replaces the container. Run one instance. -Account setup and login work behind the platform's HTTPS proxy without an origin setting or proxy IP configuration. Read [Public HTTPS hosting](#public-https-hosting) for optional forwarded-header settings. +#### CapRover + +1. In **Apps**, create a new app named `apptrail` with **Has Persistent Data** checked. +2. Open the app's **App Configs** and add a directory under **Persistent Directories**: + + | Setting | Value | + |---|---| + | Path in App | `/data` | + | Label / Volume Name | `apptrail-data` | + | Set specific host path | Leave unchecked | + +3. Keep **Instance Count** at `1` and click **Save & Update** to save the storage configuration. +4. Open **Deployment**, find **Deploy via ImageName**, enter `serpapi/apptrail:latest`, and click **Deploy**. + +CapRover manages the volume's location on the server. See [CapRover's persistent apps guide](https://caprover.com/docs/persistent-apps) for storage details. + +#### Coolify + +1. Open your project and environment, select **+ New**, then choose **Docker Image**. +2. Enter `serpapi/apptrail` as **Image Name** and `latest` as **Tag**, then save to create the application. +3. Before clicking **Deploy**, open **Configuration > Persistent Storage**, select **Add > Volume Mount**, and enter: + + | Setting | Value | + |---|---| + | Name | `apptrail-data` | + | Source Path | Leave empty | + | Destination Path | `/data` | + +4. Click **Add** to save the mount. Leaving **Source Path** empty lets Docker manage a named volume. +5. Configure your domain, keep the application on one server with one instance, then click **Deploy**. + +See [Coolify's persistent storage guide](https://coolify.io/docs/core/persistent-storage/storage-mounts/overview) and [volume mount instructions](https://coolify.io/docs/core/persistent-storage/storage-mounts/volume-mounts) for details. + +#### First setup and updates + +After deploying on either platform, use the one-time setup code in the application logs to create your owner account, then connect SerpApi. If configuring a platform health check, use HTTP `/healthz`. Account setup and login work behind the platform's HTTPS proxy without an origin setting or proxy IP configuration. Read [Public HTTPS hosting](#public-https-hosting) for optional forwarded-header settings. + +Before updating, open **Settings → Download SQLite backup** in AppTrail and save the SQLite database backup to your computer. Keep it so you can [restore your data](#backups-and-restoration) if the update causes issues. + +Then deploy `serpapi/apptrail:latest` again in the existing CapRover app or click **Redeploy** in the existing Coolify application. Keep the same volume mounted at `/data`; do not delete or recreate it. Your saved data persists across container replacement, though the app may briefly be unavailable while restarting. ## Build from source with Docker Compose @@ -140,11 +179,25 @@ The worker retries transient failures up to three attempts with a delay. Final e The browser can close while tracking continues. If the process stops or the computer sleeps, checks pause. On restart, AppTrail queues current checks for overdue queries. It cannot recover past results from the time it was offline. Graceful shutdown waits for the current provider operation to finish; individual requests have a 90-second timeout. Compose allows six minutes for shutdown. Use `--stop-timeout 360` with `docker run` as well, since a Google Play check can fetch three pages. The CLI stops waiting for stalled HTTP responses after ten seconds before shutting down the worker. +## Storage usage and cleanup + +In **Settings → Storage usage**, check the total database size and the space used by saved SerpApi responses and listing-history images. Each category has a one-time cleanup option for content older than **7 days** or **30 days**, with an estimate of the content eligible for deletion. Review the confirmation before deleting. Rankings, saved answers, matched evidence, listing text and image-change records are kept. Images shared with snapshots inside the selected period are also kept. + +Cleanup waits for active checks, deletes the selected older content, then compacts the database to return space to disk. Other requests may briefly show that cleanup is in progress. The history views explain when original responses or archived images were deleted. New checks continue saving both; cleanup does not set an automatic retention policy. The total includes SQLite's temporary journal but excludes separate backup files, which cleanup leaves untouched. Compaction needs temporary free disk space; if it cannot finish, AppTrail reports that the deleted pages remain available for SQLite to reuse. + ## Backups and restoration -Use **Settings → Download backup** to create a consistent SQLite snapshot while the app is running. The backup includes the owner account and password hash, apps, queries, jobs, account usage snapshots, and saved search history. Keep it private. It excludes login sessions, `credentials.json`, and environment secrets. Sign in again after restoring a downloaded backup. +Use **Settings → Download SQLite backup** to create a consistent snapshot while the app is running. The backup includes the owner account and password hash, apps, competitors, queries, settings, jobs, account usage snapshots, saved rankings and answers, matched evidence, and listing text history. Keep it private. It excludes raw SerpApi responses, archived listing images, login sessions, `credentials.json`, and environment secrets. AppTrail compacts the copy after removing those records so the downloaded file uses less space. The running database is unchanged. **Export history CSV** downloads search results for spreadsheet analysis; CSV files cannot restore a workspace. -To restore: +Restored search details explain that original response data was omitted to save space. Listing comparisons show placeholders for omitted images and keep their hashes and change records, so future checks can still detect changes. New checks save responses and images normally. AppTrail does not fetch live images as substitutes for missing historical images. Automatic recovery and schema-upgrade backups preserve all available response data and images. + +Use **Settings → Restore from SQLite** to upload a compatible AppTrail backup of up to 2 GB. Schema versions 6 and 7 are supported; schema 6 is validated and upgraded in a temporary copy before restoration. Confirm the overwrite, then select **Overwrite and restore**. Invalid uploads leave your workspace and login intact. + +On a fresh installation without an owner account, select **Have a backup? Restore your data** below **Create account**. Use the setup code from the new server's terminal or container logs, then upload your backup. You can restore before creating another account and sign in with the credentials saved in the backup. + +Restoring overwrites all current server data, including the owner account, password, apps, settings, and history. AppTrail waits for active requests and background checks to finish, saves the current database in `backups/before-restore-*.sqlite3`, then replaces it. All sessions are revoked, including sessions in manually copied backups. Sign in with the username and password saved in the uploaded backup. The current server's SerpApi key is kept because it is stored outside SQLite. Restored schedules resume with that key. + +To restore a larger backup, upgrade an older backup through the startup migrations, or restore without signing in: 1. Stop every AppTrail process using the destination directory. 2. Move the existing data directory aside as a rollback copy. @@ -156,7 +209,9 @@ For Docker, perform the same operation inside the named volume with the service ## Updates -Back up first. For an installation from Docker Hub, pull the new image and replace the container, reusing its data volume: +Before any update, use **Settings → Download SQLite backup** in AppTrail to save a SQLite database backup to your computer. Keep it so you can [restore your data](#backups-and-restoration) if the update causes issues. + +For an installation from Docker Hub, pull the new image and replace the container, reusing its data volume: ```bash docker pull serpapi/apptrail:latest diff --git a/pyproject.toml b/pyproject.toml index 7a04d07..0c07154 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "uv_build" [project] name = "apptrail" -version = "0.4.1" +version = "1.0.0" description = "Open-source app visibility tracking across the App Store, Google Play, and AI search." readme = "README.md" requires-python = ">=3.11" diff --git a/src/apptrail/__init__.py b/src/apptrail/__init__.py index 6958d75..e2dedbd 100644 --- a/src/apptrail/__init__.py +++ b/src/apptrail/__init__.py @@ -1,3 +1,3 @@ """App visibility tracking, on your own machine.""" -__version__ = "0.4.1" +__version__ = "1.0.0" diff --git a/src/apptrail/api.py b/src/apptrail/api.py index d9c9c82..6c8bcd7 100644 --- a/src/apptrail/api.py +++ b/src/apptrail/api.py @@ -16,9 +16,11 @@ from sqlalchemy import select, text from sqlalchemy.exc import IntegrityError from starlette.background import BackgroundTask +from starlette.concurrency import run_in_threadpool -from . import __version__ +from . import __version__, storage from .auth import Auth +from .backups import MAX_RESTORE_BYTES, RestoreGate, RestoreMiddleware, prepare_restore from .config import Config from .db import ( App, @@ -32,6 +34,7 @@ Run, Setting, Target, + backup_omits, now, ) from .engines import Gateway, ProviderError, apple_language, http_url @@ -54,6 +57,12 @@ class KeyInput(Payload): api_key: SecretStr +class StorageCleanupInput(Payload): + kind: Literal["responses", "images"] + days: Literal[7, 30] + confirm: Literal[True] + + class ReplaceQueryInput(Payload): query: str = Field(min_length=1, max_length=500) @@ -192,6 +201,7 @@ def create_app(directory=None, *, start_worker=True, gateway_factory=Gateway): raise service = Service(db, config, gateway_factory) worker = Worker(service) + restore_gate = RestoreGate() @asynccontextmanager async def lifespan(app): @@ -229,6 +239,8 @@ async def security(request: Request, call_next): response.headers["Cache-Control"] = "no-store" return response + app.add_middleware(RestoreMiddleware, gate=restore_gate) + @app.exception_handler(RequestValidationError) async def invalid(request, exc): # Pydantic's default errors include submitted input, including API keys. @@ -560,6 +572,13 @@ def run_details(run_id: int): raise HTTPException(404, "Run not found.") return { **record(item), + "responses_omitted": backup_omits(session, "responses_through_run", item.id) + and not item.responses, + "responses_cleaned": not item.responses + and item.status not in {"queued", "running"} + and storage.removed_by_cleanup( + session, "responses", item.finished_at or item.started_at or item.created_at + ), **( { "listing_snapshot_id": session.scalar( @@ -648,11 +667,31 @@ def export(app_id: int | None = None, country: str | None = None, source: str | background=BackgroundTask(path.unlink, missing_ok=True), ) + @app.get("/api/storage") + def storage_usage(): + return storage.usage(db) + + @app.post("/api/storage/cleanup") + def storage_cleanup(payload: StorageCleanupInput): + with restore_gate.exclusive(operation="Storage cleanup"): + running = worker.thread is not None and worker.thread.is_alive() + if running: + worker.stop() + try: + return storage.cleanup(db, payload.kind, payload.days) + finally: + if running: + worker.start() + @app.get("/api/backup") def backup(): with tempfile.NamedTemporaryFile(suffix=".sqlite3", delete=False) as handle: path = Path(handle.name) - db.backup(path, include_sessions=False) + try: + db.backup(path, include_sessions=False, compact=True) + except BaseException: + path.unlink(missing_ok=True) + raise return FileResponse( path, filename="apptrail-backup.sqlite3", @@ -660,6 +699,67 @@ def backup(): background=BackgroundTask(path.unlink, missing_ok=True), ) + def replace_database(request, path): + with restore_gate.exclusive(): + if request.url.path == "/api/auth/restore": + auth.authorize_setup_restore(request, throttle=False) + elif not auth.identify(request): + raise HTTPException(401, "Sign in to AppTrail again before restoring.") + running = worker.thread is not None and worker.thread.is_alive() + if running: + worker.stop() + try: + backups = config.directory / "backups" + backups.mkdir(exist_ok=True, mode=0o700) + with tempfile.NamedTemporaryFile( + dir=backups, prefix="before-restore-", suffix=".sqlite3", delete=False + ) as handle: + recovery = Path(handle.name) + try: + db.backup(recovery, include_sessions=False) + except BaseException: + recovery.unlink(missing_ok=True) + raise + db.restore(path) + auth.setup_path.unlink(missing_ok=True) + finally: + if running: + worker.start() + + @app.post("/api/restore") + @app.post("/api/auth/restore") + async def restore(request: Request): + if request.url.path == "/api/auth/restore": + await run_in_threadpool(auth.authorize_setup_restore, request) + if request.headers.get("x-apptrail-confirm-restore") != "overwrite": + raise HTTPException( + 422, "Confirm that the backup will overwrite all current server data." + ) + if int(request.headers.get("content-length", "0")) > MAX_RESTORE_BYTES: + raise HTTPException(413, "SQLite backups must be 2 GB or smaller.") + with tempfile.TemporaryDirectory(prefix=".restore-", dir=config.directory) as directory: + staging = Path(directory) + upload = staging / "upload.sqlite3" + size = 0 + with upload.open("wb") as handle: + upload.chmod(0o600) + async for chunk in request.stream(): + size += len(chunk) + if size > MAX_RESTORE_BYTES: + raise HTTPException(413, "SQLite backups must be 2 GB or smaller.") + handle.write(chunk) + path = await run_in_threadpool(prepare_restore, upload, staging) + await run_in_threadpool(replace_database, request, path) + response = JSONResponse({"ok": True}) + response.delete_cookie( + auth.cookie_name(request), + path="/", + secure=request.url.scheme == "https", + httponly=True, + samesite="strict", + ) + return response + static = Path(__file__).parent / "static" from .insights_api import register_insights diff --git a/src/apptrail/auth.py b/src/apptrail/auth.py index c1cc05e..7ff44d3 100644 --- a/src/apptrail/auth.py +++ b/src/apptrail/auth.py @@ -21,7 +21,7 @@ SESSION_SECONDS = 24 * 3600 SAFE_METHODS = {"GET", "HEAD", "OPTIONS"} -PUBLIC_API = {"/api/auth/status", "/api/auth/setup", "/api/auth/login"} +PUBLIC_API = {"/api/auth/status", "/api/auth/setup", "/api/auth/login", "/api/auth/restore"} PUBLIC_FILES = { "/login", "/healthz", @@ -104,6 +104,22 @@ def has_owner(self): def setup_token(self): return self.setup_path.read_text().strip() if self.setup_path.exists() else "" + def require_setup_token(self, supplied): + expected = self.setup_token() + if not expected or not secrets.compare_digest(expected.encode(), supplied.encode()): + raise HTTPException( + 403, "The setup code is incorrect. Use the code from the server terminal." + ) + + def authorize_setup_restore(self, request, *, throttle=True): + if throttle: + self.limit(request) + if self.has_owner(): + raise HTTPException( + 409, "An account already exists. Sign in and restore from Settings." + ) + self.require_setup_token(request.headers.get("x-apptrail-setup-token", "")) + def limit(self, request): ip = request.client.host if request.client else "unknown" try: @@ -222,15 +238,17 @@ async def guard(self, request, call_next): return RedirectResponse("/login", status_code=303) return JSONResponse({"detail": "Sign in to AppTrail."}, status_code=401) if request.method not in SAFE_METHODS: - if ( - request.headers.get("x-apptrail-request") != "1" - or request.headers.get("content-type", "").split(";")[0] != "application/json" - ): + content_type = request.headers.get("content-type", "").split(";")[0] + allowed_type = content_type == "application/json" or ( + path in {"/api/restore", "/api/auth/restore"} + and content_type == "application/vnd.sqlite3" + ) + if request.headers.get("x-apptrail-request") != "1" or not allowed_type: return JSONResponse( - {"detail": "Use the AppTrail interface or authenticated JSON requests."}, + {"detail": "Use the AppTrail interface or authenticated API requests."}, status_code=403, ) - if path not in {"/api/auth/setup", "/api/auth/login"}: + if path not in {"/api/auth/setup", "/api/auth/login", "/api/auth/restore"}: supplied = request.headers.get("x-csrf-token", "") expected = (request.state.auth or {}).get("csrf_token", "") if not expected or not secrets.compare_digest(supplied.encode(), expected.encode()): @@ -272,13 +290,7 @@ def setup(payload: SetupInput, request: Request): session.execute(text("BEGIN IMMEDIATE")) if session.get(Owner, 1): raise HTTPException(409, "An account already exists. Sign in instead.") - expected = self.setup_token() - if not expected or not secrets.compare_digest( - expected.encode(), payload.setup_token.get_secret_value().encode() - ): - raise HTTPException( - 403, "The setup code is incorrect. Use the code from the server terminal." - ) + self.require_setup_token(payload.setup_token.get_secret_value()) owner = Owner( id=1, username=payload.username.lower(), diff --git a/src/apptrail/backups.py b/src/apptrail/backups.py new file mode 100644 index 0000000..39e1a58 --- /dev/null +++ b/src/apptrail/backups.py @@ -0,0 +1,165 @@ +from __future__ import annotations + +import sqlite3 +import threading +from contextlib import closing, contextmanager +from pathlib import Path + +from argon2 import extract_parameters +from argon2.exceptions import InvalidHashError +from fastapi import HTTPException +from starlette.responses import JSONResponse + +from .db import SCHEMA_VERSION, Database, now + +MAX_RESTORE_BYTES = 2 * 1024 * 1024 * 1024 + + +class RestoreGate: + def __init__(self): + self.condition = threading.Condition() + self.active = 0 + self.restoring = False + self.operation = "A restore" + + @contextmanager + def exclusive(self, operation="A restore"): + with self.condition: + if self.restoring: + raise HTTPException( + 503, f"{self.operation} is already in progress. Try again shortly." + ) + self.restoring = True + self.operation = operation + if not self.condition.wait_for(lambda: self.active == 1, timeout=30): + self.restoring = False + raise HTTPException(503, "The server is busy. Try again shortly.") + try: + yield + finally: + with self.condition: + self.restoring = False + + +class RestoreMiddleware: + def __init__(self, app, gate): + self.app, self.gate = app, gate + + async def __call__(self, scope, receive, send): + if scope["type"] != "http" or scope["path"] == "/healthz": + return await self.app(scope, receive, send) + with self.gate.condition: + blocked = self.gate.restoring + if not blocked: + self.gate.active += 1 + if blocked: + response = JSONResponse( + {"detail": f"{self.gate.operation} is in progress. Try again shortly."}, + status_code=503, + headers={"Retry-After": "5", "Cache-Control": "no-store"}, + ) + return await response(scope, receive, send) + try: + await self.app(scope, receive, send) + finally: + with self.gate.condition: + self.gate.active -= 1 + self.gate.condition.notify_all() + + +def schema(connection): + # Ignore SQL formatting differences between SQLite and SQLAlchemy versions. + return { + (kind, name): " ".join(sql.split()).lower() if sql else None + for kind, name, sql in connection.execute( + "SELECT type, name, sql FROM sqlite_schema WHERE name NOT LIKE 'sqlite_%'" + ) + } + + +def prepare_restore(upload: Path, directory: Path) -> Path: + with upload.open("rb") as handle: + if handle.read(16) != b"SQLite format 3\0": + raise ValueError("Upload an AppTrail SQLite backup (.sqlite3).") + + # Build a trusted schema so uploaded triggers or constraints never reach the server. + staged = Database(directory) + staged.close() + try: + with ( + closing(sqlite3.connect(upload.as_uri() + "?mode=ro", uri=True)) as source, + closing(sqlite3.connect(staged.path)) as target, + ): + source.execute("PRAGMA trusted_schema=OFF") + version = source.execute("PRAGMA user_version").fetchone()[0] + # Schema 7 updates image comparisons without changing the table layout. + if version not in {6, SCHEMA_VERSION}: + raise ValueError( + "This backup uses an incompatible database format. " + "Use a backup from a compatible AppTrail version." + ) + if schema(source) != schema(target): + raise ValueError("This file is not a compatible AppTrail backup.") + if source.execute("PRAGMA quick_check(1)").fetchone() != ("ok",): + raise ValueError("The backup is damaged. Upload a different SQLite backup.") + + target.execute("BEGIN") + for (table,) in target.execute( + "SELECT name FROM sqlite_schema WHERE type='table' AND name NOT LIKE 'sqlite_%'" + ).fetchall(): + columns = source.execute(f'PRAGMA table_info("{table}")').fetchall() + for _, column, kind, *_ in columns: + if kind != "JSON": + continue + if source.execute( + f'SELECT 1 FROM "{table}" WHERE NOT json_valid("{column}") LIMIT 1' + ).fetchone(): + raise ValueError("The backup contains invalid app data.") + if table != "settings": + expected = ( + "array" if column in {"aliases", "responses", "changes"} else "object" + ) + if source.execute( + f'SELECT 1 FROM "{table}" WHERE json_type("{column}") != ? LIMIT 1', + (expected,), + ).fetchone(): + raise ValueError("The backup contains incompatible app data.") + # Sessions must never be imported, even from a manually copied database. + if table in {"login_sessions", "auth_limits"}: + continue + rows = source.execute(f'SELECT * FROM "{table}"') + placeholders = ",".join("?" for _ in columns) + while batch := rows.fetchmany(1 if table == "listing_assets" else 100): + target.executemany(f'INSERT INTO "{table}" VALUES ({placeholders})', batch) + + if target.execute("PRAGMA foreign_key_check").fetchone(): + raise ValueError("The backup contains broken app or history references.") + owners = target.execute("SELECT id, username, password_hash FROM owner").fetchall() + if len(owners) != 1 or owners[0][0] != 1 or not owners[0][1]: + raise ValueError("The backup must contain an AppTrail owner account.") + try: + parameters = extract_parameters(owners[0][2]) + if not ( + 1 <= parameters.time_cost <= 10 + and 8 <= parameters.memory_cost <= 262144 + and 1 <= parameters.parallelism <= 8 + ): + raise InvalidHashError + except (InvalidHashError, TypeError): + raise ValueError("The backup contains an invalid account password.") from None + target.execute( + "UPDATE runs SET status='queued', available_at=?, " + "error='Interrupted; resuming after restore.' WHERE status='running'", + (now(),), + ) + target.commit() + if version == 6: + from .listing_history import upgrade_image_comparisons + + with staged.engine.begin() as connection: + upgrade_image_comparisons(connection) + except sqlite3.DatabaseError: + raise ValueError("The backup is damaged or contains incompatible app data.") from None + finally: + staged.close() + return staged.path diff --git a/src/apptrail/db.py b/src/apptrail/db.py index 22fe572..00381da 100644 --- a/src/apptrail/db.py +++ b/src/apptrail/db.py @@ -1,5 +1,6 @@ from __future__ import annotations +import json import os import sqlite3 import tempfile @@ -24,6 +25,8 @@ ) from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column, relationship, sessionmaker +SCHEMA_VERSION = 7 + def now() -> float: return datetime.now(UTC).timestamp() @@ -197,6 +200,13 @@ class Setting(Base): value: Mapped[dict] = mapped_column(JSON) +def backup_omits(session, key, item_id): + setting = session.get(Setting, "backup_omissions") + metadata = setting.value if setting and isinstance(setting.value, dict) else {} + cutoff = metadata.get(key) + return isinstance(cutoff, int) and item_id <= cutoff + + class Notification(Base): __tablename__ = "notifications" id: Mapped[int] = mapped_column(primary_key=True) @@ -297,11 +307,11 @@ def configure(connection, _): with self.engine.begin() as connection: connection.exec_driver_sql("BEGIN IMMEDIATE") version = connection.execute(text("PRAGMA user_version")).scalar() - if version > 7: + if version > SCHEMA_VERSION: raise RuntimeError( "This database needs a newer AppTrail version. Upgrade AppTrail." ) - if version and version < 7: + if version and version < SCHEMA_VERSION: backups = directory / "backups" backups.mkdir(exist_ok=True, mode=0o700) with tempfile.NamedTemporaryFile( @@ -408,7 +418,7 @@ def configure(connection, _): self.path.chmod(0o600) self.session = sessionmaker(self.engine, expire_on_commit=False) - def backup(self, destination: Path, *, include_sessions=True): + def backup(self, destination: Path, *, include_sessions=True, compact=False): private_sqlite_file(destination) with ( closing(sqlite3.connect(self.path)) as source, @@ -417,8 +427,37 @@ def backup(self, destination: Path, *, include_sessions=True): source.backup(target) if not include_sessions: target.execute("DELETE FROM login_sessions") - target.commit() + if compact: + target.execute("DELETE FROM listing_assets") + target.execute("UPDATE run_payloads SET responses='[]'") + target.execute("UPDATE runs SET responses='[]'") + omissions = { + "responses_through_run": target.execute( + "SELECT COALESCE(MAX(id), 0) FROM runs" + ).fetchone()[0], + "images_through_snapshot": target.execute( + "SELECT COALESCE(MAX(id), 0) FROM listing_snapshots" + ).fetchone()[0], + } + target.execute( + "INSERT INTO settings (key, value) VALUES ('backup_omissions', ?) " + "ON CONFLICT(key) DO UPDATE SET value=excluded.value", + (json.dumps(omissions),), + ) + target.commit() + if compact: + # Deleting rows alone leaves their pages in the downloaded file. + target.execute("VACUUM") destination.chmod(0o600) def close(self): self.engine.dispose() + + def restore(self, source_path: Path): + # Call only after HTTP requests and the worker have finished using the database. + self.engine.dispose() + with ( + closing(sqlite3.connect(source_path)) as source, + closing(sqlite3.connect(self.path)) as target, + ): + source.backup(target) diff --git a/src/apptrail/listing_history.py b/src/apptrail/listing_history.py index e71cde6..adce4ac 100644 --- a/src/apptrail/listing_history.py +++ b/src/apptrail/listing_history.py @@ -10,7 +10,8 @@ from PIL import Image, ImageOps from sqlalchemy import select -from .db import App, Listing, ListingAsset, ListingSnapshot, ListingWatch, Run, now +from .db import App, Listing, ListingAsset, ListingSnapshot, ListingWatch, Run, backup_omits, now +from .storage import removed_by_cleanup MEDIA_FIELDS = { "icon", @@ -362,7 +363,39 @@ def comparison(self, snapshot_id): .order_by(ListingSnapshot.id.desc()) .limit(1) ) + + def with_image_availability(item): + if item is None: + return None + asset_ids = set() + for field in MEDIA_FIELDS: + value = item.data.get(field) + refs = [value] if field == "icon" else value or [] + asset_ids.update( + ref["asset_id"] + for ref in refs + if isinstance(ref, dict) and ref.get("asset_id") + ) + available = ( + set( + session.scalars( + select(ListingAsset.id).where(ListingAsset.id.in_(asset_ids)) + ) + ) + if asset_ids + else set() + ) + missing = sorted(asset_ids - available) + return { + **record(item), + "missing_assets": missing, + "images_omitted": bool(missing) + and backup_omits(session, "images_through_snapshot", item.id), + "images_cleaned": bool(missing) + and removed_by_cleanup(session, "images", item.checked_at), + } + return { - "snapshot": record(snapshot), - "previous": record(previous) if previous else None, + "snapshot": with_image_availability(snapshot), + "previous": with_image_availability(previous), } diff --git a/src/apptrail/static/app.js b/src/apptrail/static/app.js index b30e4dc..e2ce73f 100644 --- a/src/apptrail/static/app.js +++ b/src/apptrail/static/app.js @@ -70,7 +70,11 @@ const sourceColors = { }; let csrfToken = "", accountName = "", + restoring = false, + cleaningStorage = false, leaving = false; +let storageUsage = null, storageError = "", storagePending = false; +const storageDays = { responses: 7, images: 7 }; let state, dashboard, charts = [], @@ -216,7 +220,9 @@ async function api(path, options = {}) { "X-CSRF-Token": csrfToken, ...options.headers, }, - body: options.body === undefined ? undefined : JSON.stringify(options.body), + body: options.body instanceof Blob + ? options.body + : options.body === undefined ? undefined : JSON.stringify(options.body), }); if (!response.ok) { if (response.status === 401 && path !== "auth/password") { @@ -855,6 +861,132 @@ function activityPage() { }` ); } +function storageBytes(value) { + if (value < 1024) return `${value} B`; + const unit = Math.min(Math.floor(Math.log(value) / Math.log(1024)), 3); + return `${(value / 1024 ** unit).toFixed(1)} ${["B", "KiB", "MiB", "GiB"][unit]}`; +} +function storagePanel() { + const rows = storageUsage ? [ + ["responses", "Saved SerpApi responses", "Original response data used for search evidence. Saved results and matched evidence are kept."], + ["images", "Listing-history images", "Archived icons and screenshots. Images still used by newer snapshots are kept."], + ].map(([kind, title, description]) => { + const data = storageUsage[kind], days = storageDays[kind], eligible = data.older_than[days]; + return `

${title}

${storageBytes(data.bytes)}

${description}

${button("Delete older " + (kind === "images" ? "images" : "responses"), "cleanup-storage", "danger small", `data-kind="${kind}" ${eligible.count ? "" : "disabled"}`)}
${storageBytes(eligible.bytes)} eligible for deletion.
`; + }).join("") : `

${storageError ? esc(storageError) : "Calculating storage usage…"}

`; + return `

Storage usage

${button(icon("refresh") + " Refresh", "refresh-storage", "small", storagePending ? "disabled" : "")}
${storageUsage ? `
Total database size
${storageBytes(storageUsage.total_bytes)}

Includes saved data, indexes, free pages and ${storageBytes(storageUsage.journal_bytes)} of temporary database journal data. Separate backup files are excluded.

` : ""}${rows}

Cleanup runs once when you confirm. It permanently deletes the selected older content and compacts the database to free disk space. New checks continue saving responses and images.

`; +} +function renderStorage() { + const panel = $("#storage-panel"); + if (panel) panel.outerHTML = storagePanel(); +} +async function loadStorage() { + if (storagePending || cleaningStorage || restoring) return; + storagePending = true; + storageError = ""; + renderStorage(); + try { + storageUsage = await api("storage"); + } catch (error) { + storageUsage = null; + storageError = error.message; + } finally { + storagePending = false; + renderStorage(); + } +} +function cleanupStorageDialog(kind) { + const days = storageDays[kind], label = kind === "images" ? "listing-history images" : "saved SerpApi responses"; + showModal("Delete older " + (kind === "images" ? "images" : "responses"), "Free space in this workspace.", + `

Permanently delete ${label} older than ${days} days? About ${storageBytes(storageUsage[kind].older_than[days].bytes)} of content is eligible.

Saved rankings, answers, matched evidence and listing text will remain available.${kind === "images" ? " Images shared with newer snapshots will be kept." : ""}

This cannot be undone. AppTrail waits for active checks, then compacts the database. Keep this page open until cleanup finishes.

${button("Cancel", "close-modal")}
`); +} +async function cleanupStorage(form) { + cleaningStorage = true; + syncVersion++; + const controls = $$("button", modal).filter(el => !el.disabled); + controls.forEach(el => { el.disabled = true; }); + $("#cleanup-status").textContent = "Waiting for active checks, deleting older content and compacting the database…"; + try { + const result = await api("storage/cleanup", { method: "POST", body: { + kind: form.dataset.kind, days: Number(form.dataset.days), confirm: true, + } }); + storageUsage = result.usage; + modal.close(); + renderStorage(); + toast(result.compacted ? `Deleted ${storageBytes(result.removed_bytes)} of older content. Database compacted.` : "Older content was deleted, but the database could not be compacted. Its freed pages can still be reused by new data.", !result.compacted); + } finally { + cleaningStorage = false; + controls.forEach(el => { el.disabled = false; }); + if ($("#cleanup-status")) $("#cleanup-status").textContent = ""; + } +} +document.addEventListener("change", event => { + const kind = event.target.dataset.storageDays; + if (!kind) return; + storageDays[kind] = Number(event.target.value); + renderStorage(); + $(`[data-storage-days="${kind}"]`)?.focus(); +}); +function backupsPanel() { + return `

Data & backups

+

History CSV

Export saved search results for spreadsheets and analysis. CSV files cannot restore your workspace and do not include account details or settings.

${icon("download")} Export history CSV
+

SQLite backup

Save your apps, competitors, queries, settings, rankings, saved answers and listing text history. Includes your account and saved password. Keep this file private.

${icon("download")} Download SQLite backup

Raw SerpApi responses and listing-history images are omitted to keep backups small. Saved results, matched evidence and image-change records are kept. After restoring, omitted content is marked as unavailable.

Login sessions and your SerpApi key are not included. Your key is stored separately on the server.

+

Restore from SQLite

Upload a compatible AppTrail backup to replace the current server data. Your account, password and app data will be reset to the uploaded backup. Everyone will be signed out.

${button("Restore from SQLite", "restore-backup", "danger")}

AppTrail validates the file before restoring and saves a recovery backup on the server. The server’s current SerpApi key is kept.

+
`; +} +function restoreBackupDialog() { + showModal( + "Restore from SQLite", + "Replace this server’s workspace with an AppTrail backup.", + `
+
This will overwrite all current server data.

Your account, password, apps, settings and saved history will be reset to the uploaded backup. Everyone will be signed out. You will need the username and password saved in that backup to sign in again.

+ +

Up to 2 GB. We check compatibility before changing any data and save a recovery backup on the server. The current SerpApi key is kept. Downloaded backups omit raw SerpApi responses and listing-history images to save space; saved results and text history are kept.

+ +

+
+
`, + ); +} +async function restoreBackup(form) { + const file = form.elements.backup.files[0]; + if (!file || !file.size) throw new Error("Choose a SQLite backup file."); + if (file.size > 2 * 1024 * 1024 * 1024) + throw new Error("SQLite backups must be 2 GB or smaller."); + if (!form.elements.confirm.checked) + throw new Error("Confirm that the backup will overwrite all current server data."); + restoring = true; + syncVersion++; + const controls = $$("input, button", modal).filter((el) => !el.disabled); + controls.forEach((el) => { el.disabled = true; }); + $("#restore-status").textContent = "Uploading, validating and restoring your backup. Waiting for active checks may take a few minutes. Keep this page open."; + try { + await api("restore", { + method: "POST", + headers: { + "Content-Type": "application/vnd.sqlite3", + "X-AppTrail-Confirm-Restore": "overwrite", + }, + body: file, + }); + leaving = true; + localStorage.removeItem("apptrail-app"); + document.body.replaceChildren(); + location.replace("/login?restored=1"); + } finally { + restoring = false; + controls.forEach((el) => { el.disabled = false; }); + if ($("#restore-status")) $("#restore-status").textContent = ""; + } +} +function missingKeyBanner() { + if ( + state.configured || + (!state.apps.length && state.onboarding?.visible !== false) + ) + return ""; + return `
SerpApi API key missingA SerpApi key is required to run checks. Get your key from Manage API Key, then add it in Settings.
`; +} function settingsPage() { const a = state.account?.data || {}, estimate = state.estimated_monthly; @@ -869,7 +1001,7 @@ function settingsPage() { (state.onboarding?.completed.length === 3 ? "" : `

Setup checklist

Reopen the shortcuts for connecting an app and adding tracking queries.

${button("Show checklist", "restore-checklist", "small")}
`) + - `

Your account

Signed in as ${esc(accountName)}.

Use 8 to 128 characters, including a number and a special character. Changing your password signs out other sessions.

SerpApi connection

${state.configured ? "Connected" : "Not connected"}

Connect SerpApi to check your app’s visibility.

${state.key_from_environment ? "" : `
`}Find your SerpApi key ${icon("external")}

Data & backups

Download your search history or save a backup of your apps and settings.

${icon("download")} Export history CSV${icon("download")} Download backup

Your SerpApi key is not included in backups.

Your search credits

${a.total_searches_left != null ? num(a.total_searches_left) : "—"}
total searches remaining
Plan
${esc(a.plan_name || "—")}
Used this month
${a.this_month_usage != null ? num(a.this_month_usage) : "—"}
Monthly allowance
${a.searches_per_month != null ? num(a.searches_per_month) : "—"}
Extra credits
${a.extra_credits != null ? num(a.extra_credits) : "—"}
Next renewal
${esc(a.plan_renewal_date || "Not applicable")}
Hourly limit
${a.account_rate_limit_per_hour != null ? num(a.account_rate_limit_per_hour) : "—"}
Estimated AppTrail usage
${num(estimate.min)}${estimate.min !== estimate.max ? "–" + num(estimate.max) : ""}/mo

${state.sync_paused ? "Estimate for when workspace sync resumes. " : ""}Based on active queries and enabled listing histories. Discovery, retries, and manual checks are additional.

Account-wide balance · Updated ${ago(state.account?.checked_at)}
` + `

Your account

Signed in as ${esc(accountName)}.

Use 8 to 128 characters, including a number and a special character. Changing your password signs out other sessions.

SerpApi connection

${state.configured ? "Connected" : "Not connected"}

Connect SerpApi to check your app’s visibility.

${state.key_from_environment ? "" : `
`}Find your SerpApi key ${icon("external")}
${storagePanel()}${backupsPanel()}

Your search credits

${a.total_searches_left != null ? num(a.total_searches_left) : "—"}
total searches remaining
Plan
${esc(a.plan_name || "—")}
Used this month
${a.this_month_usage != null ? num(a.this_month_usage) : "—"}
Monthly allowance
${a.searches_per_month != null ? num(a.searches_per_month) : "—"}
Extra credits
${a.extra_credits != null ? num(a.extra_credits) : "—"}
Next renewal
${esc(a.plan_renewal_date || "Not applicable")}
Hourly limit
${a.account_rate_limit_per_hour != null ? num(a.account_rate_limit_per_hour) : "—"}
Estimated AppTrail usage
${num(estimate.min)}${estimate.min !== estimate.max ? "–" + num(estimate.max) : ""}/mo

${state.sync_paused ? "Estimate for when workspace sync resumes. " : ""}Based on active queries and enabled listing histories. Discovery, retries, and manual checks are additional.

Account-wide balance · Updated ${ago(state.account?.checked_at)}
` ); } function welcome() { @@ -1161,7 +1293,8 @@ function render() { $("#app-count").textContent = state.apps.filter((a) => !a.archived).length; $("#version-label").textContent = `OPEN SOURCE · v${state.version}`; $("#main").innerHTML = - !state.apps.length && route() !== "settings" + missingKeyBanner() + + (!state.apps.length && route() !== "settings" ? welcome() : { overview, @@ -1171,7 +1304,7 @@ function render() { activity: activityPage, "listing-history": insights.listingPage, settings: settingsPage, - }[route()](); + }[route()]()); $$("[data-width]").forEach( (el) => (el.style.width = `${Number(el.dataset.width)}%`), ); @@ -1191,6 +1324,7 @@ function render() { ); }); competitors.updateSummary(); + if (route() === "settings" && !storageUsage && !storageError) void loadStorage(); } function drawChart() { const canvas = $("#trend-chart"); @@ -1317,7 +1451,7 @@ document.addEventListener("input", (event) => { }); document.addEventListener("reset", (event) => dirtyForms.delete(event.target)); async function sync(silent = false) { - if (leaving || (silent && syncPending)) return; + if (leaving || restoring || cleaningStorage || (silent && syncPending)) return; const finishLoading = silent ? null : beginLoading(); syncPending++; if (!silent) renderRequested = true; @@ -1767,6 +1901,10 @@ async function showRun(id) { return; } let body = `
${statusTag(r.status)}${r.kind === "listing_history" ? "Listing history" : sourceLabels[r.params.source] || "Listing refresh"}${esc(countryName(r.params.country || "global"))}${when(r.started_at || r.created_at)}${r.requests_count} requests
${r.error ? `
${esc(r.error)}
` : ""}${r.observations.map((o) => `

${esc(appById(o.app_id)?.name || "Tracked app")}${o.retrospective ? " · Reanalyzed" : ""}

${o.data.evidence?.map((e) => `
${esc(e.type.replaceAll("_", " "))} ${esc(e.text)}${e.url ? `
Open matching source ${icon("external")}` : ""}
`).join("") || `

No matching evidence in this check.${result.kind === "ai" ? " If the answer uses another name, add it as an alias in My apps → Manage, then reanalyze history." : ""}

`}`).join("")}`; + if (r.responses_omitted) + body = '

Original SerpApi responses were omitted from this backup to save space. Saved results and matched evidence are still available.

' + body; + else if (r.responses_cleaned) + body = '

Original SerpApi responses were deleted during storage cleanup. Saved results and matched evidence are still available.

' + body; if (result.kind === "ai") body += `

Saved answer

${esc(result.text || "No answer returned.")}
${result.references.map((ref) => `${esc(ref.title)} ${icon("external")}${esc(ref.link)}`).join("")}
`; if (result.kind === "store") @@ -1781,6 +1919,9 @@ async function showRun(id) { } } let profileChart; +modal.addEventListener("cancel", (event) => { + if (restoring || cleaningStorage) event.preventDefault(); +}); modal.addEventListener("close", () => { placeLoadingIndicator(); profileChart?.destroy(); @@ -1900,6 +2041,14 @@ document.addEventListener("submit", async (event) => { const errorBox = $("#modal-error"); if (errorBox) errorBox.textContent = ""; try { + if (form.id === "restore-backup-form") { + await restoreBackup(form); + return; + } + if (form.id === "storage-cleanup-form") { + await cleanupStorage(form); + return; + } if (await competitors.submit(form)) return; if (await insights.submit(form)) return; if (form.id === "query-group-form") { @@ -2023,10 +2172,22 @@ document.addEventListener("submit", async (event) => { }); document.addEventListener("click", async (event) => { const el = event.target.closest("[data-action]"); - if (!el || el.disabled || isLoading(el)) return; + if (!el || el.disabled || isLoading(el) || restoring || cleaningStorage) return; const action = el.dataset.action; const finishLoading = beginLoading(el); try { + if (action === "refresh-storage") { + await loadStorage(); + return; + } + if (action === "cleanup-storage") { + cleanupStorageDialog(el.dataset.kind); + return; + } + if (action === "restore-backup") { + restoreBackupDialog(); + return; + } if (competitors.click(el)) return; if (await insights.click(el)) return; if ( @@ -2587,6 +2748,7 @@ document.addEventListener("keydown", (event) => { window.addEventListener("scroll", hideControlHint, true); window.addEventListener("resize", hideControlHint); window.addEventListener("hashchange", () => { + if (route() === "settings") { storageUsage = null; storageError = ""; } setNavigation(false); sync(); }); diff --git a/src/apptrail/static/auth.css b/src/apptrail/static/auth.css index a27eb57..ffcaf66 100644 --- a/src/apptrail/static/auth.css +++ b/src/apptrail/static/auth.css @@ -36,11 +36,14 @@ color: var(--muted); margin: -4px 0 20px; } -#auth-submit { +#auth-submit, +#setup-restore-submit { width: 100%; min-height: 44px; } -#auth-error:empty { +#auth-error:empty, +#setup-restore-error:empty, +#setup-restore-status:empty { display: none; } #auth-error { @@ -49,6 +52,33 @@ #auth-recovery { margin: 20px 0 0; } +.auth-secondary { + display: block; + margin: 20px auto 0; + padding: 0; + background: none; + border: 0; + font: inherit; + font-size: 12px; + color: var(--muted); + cursor: pointer; + text-align: center; +} +.auth-secondary:hover { + color: var(--ink); + text-decoration: underline; +} +.auth-secondary:focus-visible { + outline: 2px solid var(--primary); + outline-offset: 4px; +} +#setup-restore-form .restore-warning, +#setup-restore-form .restore-confirm { + font-size: 12px; +} +#setup-restore-error { + margin-bottom: 16px; +} .auth-help code { overflow-wrap: anywhere; } diff --git a/src/apptrail/static/auth.html b/src/apptrail/static/auth.html index 9628843..32e83c1 100644 --- a/src/apptrail/static/auth.html +++ b/src/apptrail/static/auth.html @@ -83,6 +83,41 @@

Sign in to AppTrail

apptrail --reset-password.

+ + diff --git a/src/apptrail/static/auth.js b/src/apptrail/static/auth.js index 8de1ab8..291ab9d 100644 --- a/src/apptrail/static/auth.js +++ b/src/apptrail/static/auth.js @@ -3,7 +3,31 @@ import { beginLoading, isLoading } from "./loading.js"; const form = document.querySelector("#auth-form"); const error = document.querySelector("#auth-error"); const submit = document.querySelector("#auth-submit"); +const restoreForm = document.querySelector("#setup-restore-form"); +const restoreOption = document.querySelector("#setup-restore-option"); +const restoreError = document.querySelector("#setup-restore-error"); +const restoreSubmit = document.querySelector("#setup-restore-submit"); let setup = false; +function showRestore(open) { + if (!setup || isLoading(form) || isLoading(restoreForm)) return; + const from = open ? form : restoreForm; + const to = open ? restoreForm : form; + to.elements.setup_token.value = from.elements.setup_token.value; + form.hidden = open; + restoreOption.hidden = open; + restoreForm.hidden = !open; + document.querySelector("#auth-title").textContent = open + ? "Restore your data" + : "Create your account"; + document.querySelector("#auth-description").textContent = open + ? "Recover your AppTrail workspace from a backup after an update or move." + : "Set up the owner account for this AppTrail workspace."; + document.title = (open ? "Restore your data" : "Create your account") + " · AppTrail"; + if (open) restoreForm.elements.setup_token.focus(); + else document.querySelector("#show-restore").focus(); +} +document.querySelector("#show-restore").addEventListener("click", () => showRestore(true)); +document.querySelector("#cancel-restore").addEventListener("click", () => showRestore(false)); async function load() { try { const response = await fetch("/api/auth/status", { @@ -20,12 +44,16 @@ async function load() { return; } setup = state.setup_required; + restoreOption.hidden = !setup; + restoreForm.hidden = true; document.querySelector("#auth-title").textContent = setup ? "Create your account" : "Sign in to AppTrail"; document.querySelector("#auth-description").textContent = setup ? "Set up the owner account for this AppTrail workspace." - : "Access your apps and search history."; + : new URLSearchParams(location.search).get("restored") === "1" + ? "Backup restored. Sign in with the username and password saved in that backup." + : "Access your apps and search history."; document.title = (setup ? "Create your account" : "Sign in") + " · AppTrail"; for (const id of ["setup-fields", "confirm-field", "password-help"]) @@ -54,6 +82,8 @@ async function load() { submit.textContent = setup ? "Create account" : "Sign in"; form.hidden = false; } catch (e) { + restoreOption.hidden = true; + restoreForm.hidden = true; error.textContent = e.message; form.hidden = false; submit.disabled = true; @@ -100,4 +130,56 @@ form.addEventListener("submit", async (event) => { submit.textContent = setup ? "Create account" : "Sign in"; } }); +restoreForm.addEventListener("submit", async (event) => { + event.preventDefault(); + if (!setup || isLoading(restoreForm) || restoreSubmit.disabled) return; + restoreError.textContent = ""; + const file = restoreForm.elements.backup.files[0]; + if (!file || !file.size || file.size > 2 * 1024 * 1024 * 1024) { + restoreError.textContent = "Choose a SQLite backup file of up to 2 GB."; + return; + } + if (!restoreForm.elements.confirm.checked) { + restoreError.textContent = "Confirm that the backup will overwrite current server data."; + return; + } + const setupToken = restoreForm.elements.setup_token.value.trim(); + const finishLoading = beginLoading(restoreSubmit, restoreForm); + const controls = [...restoreForm.querySelectorAll("input, button")].filter((el) => !el.disabled); + controls.forEach((el) => { el.disabled = true; }); + const status = document.querySelector("#setup-restore-status"); + status.textContent = "Uploading, validating and restoring your backup. Keep this page open."; + restoreSubmit.textContent = "Restoring…"; + try { + const response = await fetch("/api/auth/restore", { + method: "POST", + credentials: "same-origin", + headers: { + "Content-Type": "application/vnd.sqlite3", + "X-AppTrail-Request": "1", + "X-AppTrail-Setup-Token": setupToken, + "X-AppTrail-Confirm-Restore": "overwrite", + }, + body: file, + }); + const result = await response.json().catch(() => ({})); + if (!response.ok) { + if (response.status === 409) { + await load(); + error.textContent = result.detail; + return; + } + throw new Error(result.detail || "Unable to restore the backup. Try again."); + } + localStorage.removeItem("apptrail-app"); + location.replace("/login?restored=1"); + } catch (e) { + restoreError.textContent = e.message; + } finally { + status.textContent = ""; + controls.forEach((el) => { el.disabled = false; }); + finishLoading(); + restoreSubmit.textContent = "Restore backup"; + } +}); await load(); diff --git a/src/apptrail/static/insights-ui.js b/src/apptrail/static/insights-ui.js index 6074b88..5bff41c 100644 --- a/src/apptrail/static/insights-ui.js +++ b/src/apptrail/static/insights-ui.js @@ -365,10 +365,15 @@ export function createInsightsUI(ctx) { ); } - function mediaImage(item, alt) { - return item?.asset_id - ? `${esc(alt)}` - : '
Image could not be archived
'; + function mediaImage(item, alt, snapshot) { + if (item?.asset_id && !snapshot?.missing_assets?.includes(item.asset_id)) + return `${esc(alt)}`; + const reason = item?.asset_id && snapshot?.images_omitted + ? "Image omitted from backup to save space" + : item?.asset_id && snapshot?.images_cleaned + ? "Image deleted during storage cleanup" + : "Archived image unavailable"; + return `
${reason}
`; } async function showSnapshot(id) { const request = ++requestVersion; @@ -400,17 +405,17 @@ export function createInsightsUI(ctx) { let left, right; if (field === "icon") { left = a - ? mediaImage(a, "Previous app icon") + ? mediaImage(a, "Previous app icon", before) : 'No previous icon'; right = b - ? mediaImage(b, "Current app icon") + ? mediaImage(b, "Current app icon", after) : 'No icon'; } else if (field.endsWith("screenshots")) { const old = a || [], next = b || [], annotated = mediaChanges(old, next); - left = `
${old.map((item, i) => `
${mediaImage(item, `Previous screenshot ${i + 1}`)}
${i + 1} · ${next.some((n) => sameMedia(n, item)) ? "Previous" : "Removed"}
`).join("") || 'No screenshots'}
`; - right = `
${annotated.map((item, i) => `
${mediaImage(item, `Current screenshot ${i + 1}`)}
${i + 1} · ${item.label}
`).join("") || 'No screenshots'}
`; + left = `
${old.map((item, i) => `
${mediaImage(item, `Previous screenshot ${i + 1}`, before)}
${i + 1} · ${next.some((n) => sameMedia(n, item)) ? "Previous" : "Removed"}
`).join("") || 'No screenshots'}
`; + right = `
${annotated.map((item, i) => `
${mediaImage(item, `Current screenshot ${i + 1}`, after)}
${i + 1} · ${item.label}
`).join("") || 'No screenshots'}
`; } else { const diff = textDiff(a ?? "", b ?? ""); const text = (parts, tag) => @@ -429,7 +434,7 @@ export function createInsightsUI(ctx) { before ? `${when(before.checked_at)} → ${when(after.checked_at)}` : when(after.checked_at), - `
${before ? 'RemovedAdded' : "Baseline saved. Future changes will be compared with this listing."}Archived images · saved text
${before ? `
BEFORE ${when(before.checked_at)}AFTER ${when(after.checked_at)}
` : ""}${fields.map(fieldView).join("") || '
No listing changes in this check.
'}`, + `${before?.images_omitted || after.images_omitted ? '

Listing-history images were omitted from this backup to save space. Saved text and image-change records are still available.

' : before?.images_cleaned || after.images_cleaned ? '

Older listing-history images were deleted during storage cleanup. Saved text and image-change records are still available.

' : ""}
${before ? 'RemovedAdded' : "Baseline saved. Future changes will be compared with this listing."}Saved listing history
${before ? `
BEFORE ${when(before.checked_at)}AFTER ${when(after.checked_at)}
` : ""}${fields.map(fieldView).join("") || '
No listing changes in this check.
'}`, ); modal.dataset.view = "listing-diff"; } diff --git a/src/apptrail/static/style.css b/src/apptrail/static/style.css index b9e5f5b..a325f9c 100644 --- a/src/apptrail/static/style.css +++ b/src/apptrail/static/style.css @@ -2017,6 +2017,18 @@ tbody tr:hover { padding: 24px; margin-bottom: 20px; } +.key-missing-banner { + border-left: 4px solid var(--red); +} +.key-missing-banner strong { + display: block; + margin-bottom: 4px; +} +.key-missing-banner a { + color: inherit; + text-decoration: underline; + text-underline-offset: 2px; +} .settings-panel h2 { margin-bottom: 7px; } @@ -2026,6 +2038,51 @@ tbody tr:hover { .settings-panel > .actions h2 { margin-bottom: 0; } +.storage-total, +.storage-category-heading { + display: flex; + justify-content: space-between; + align-items: baseline; + gap: 12px; +} +.storage-total { + margin: 24px 0 8px; +} +.storage-total dd { + margin: 0; + font-size: 23px; + font-weight: 650; + white-space: nowrap; +} +.storage-category { + margin: 20px 0; + padding-top: 20px; + border-top: 1px solid var(--line); +} +.storage-category-heading h3 { + margin: 0; + font-size: 14px; +} +.storage-category-heading strong { + white-space: nowrap; +} +.storage-category p { + margin: 8px 0 14px; + color: var(--muted); + font-size: 12px; + line-height: 1.6; +} +.storage-controls { + display: flex; + align-items: flex-end; + flex-wrap: wrap; + gap: 12px; + margin-bottom: 10px; +} +.storage-controls .field { + flex: 1 1 180px; + margin: 0; +} .manage-listing-refresh { display: flex; align-items: center; @@ -2095,6 +2152,45 @@ tbody tr:hover { font-size: 10px; margin: 12px 0 0; } +.backup-option + .backup-option { + border-top: 1px solid var(--line); + margin-top: 20px; + padding-top: 20px; +} +.backup-option h3 { + font-size: 13px; + margin-top: 16px; +} +.backup-option p { + font-size: 12px; + line-height: 1.6; + color: var(--muted); + margin: 8px 0 14px; +} +.restore-warning { + border: 1px solid var(--red); + border-radius: 8px; + padding: 16px; + margin-bottom: 20px; + line-height: 1.6; +} +.restore-warning strong { + color: var(--red); +} +.restore-warning p { + margin-top: 8px; +} +.restore-confirm { + display: flex; + align-items: flex-start; + gap: 10px; + margin: 20px 0; + line-height: 1.6; +} +.restore-confirm input { + flex-shrink: 0; + margin-top: 4px; +} .credit-number { font-family: Manrope, sans-serif; font-size: 42px; diff --git a/src/apptrail/storage.py b/src/apptrail/storage.py new file mode 100644 index 0000000..7a361ca --- /dev/null +++ b/src/apptrail/storage.py @@ -0,0 +1,147 @@ +from __future__ import annotations + +import json +import sqlite3 +from contextlib import closing + +from .db import Setting, now + +RUN_TIME = "COALESCE(finished_at, started_at, created_at)" +FINISHED = "status NOT IN ('queued', 'running')" +RESPONSE_BYTES = "CASE WHEN responses != '[]' THEN length(CAST(responses AS BLOB)) ELSE 0 END" + + +def removed_by_cleanup(session, kind, checked_at): + setting = session.get(Setting, "storage_cleanup") + metadata = setting.value if setting and isinstance(setting.value, dict) else {} + cutoff = metadata.get(kind) + return isinstance(cutoff, (int, float)) and checked_at < cutoff + + +def image_references(connection): + from .listing_history import MEDIA_FIELDS + + connection.execute("CREATE TEMP TABLE image_references (id TEXT PRIMARY KEY, seen REAL)") + fields = ", ".join(f"json_extract(data, '$.{field}')" for field in sorted(MEDIA_FIELDS)) + for checked_at, *values in connection.execute( + f"SELECT checked_at, {fields} FROM listing_snapshots" + ): + refs = [] + for value in values: + parsed = json.loads(value) if value else None + refs.extend(parsed if isinstance(parsed, list) else [parsed]) + connection.executemany( + "INSERT INTO image_references VALUES (?, ?) " + "ON CONFLICT(id) DO UPDATE SET seen=MAX(seen, excluded.seen)", + [ + (ref["asset_id"], checked_at) + for ref in refs + if isinstance(ref, dict) and ref.get("asset_id") + ], + ) + + +def response_usage(connection, cutoff=None): + condition = f"{FINISHED} AND {RUN_TIME} < ?" if cutoff is not None else "1" + params = (cutoff,) if cutoff is not None else () + return connection.execute( + f"SELECT COALESCE(SUM(size), 0), COUNT(*) FROM (" + f"SELECT id, {RESPONSE_BYTES} AS size FROM runs WHERE {condition} " + f"UNION ALL SELECT run_id, {RESPONSE_BYTES} AS size FROM run_payloads " + f"WHERE run_id IN (SELECT id FROM runs WHERE {condition})" + ") WHERE size > 0", + params + params, + ).fetchone() + + +def image_usage(connection, cutoff=None): + condition = "WHERE refs.seen IS NULL OR refs.seen < ?" if cutoff is not None else "" + return connection.execute( + "SELECT COALESCE(SUM(length(content)), 0), COUNT(*) FROM listing_assets " + "LEFT JOIN image_references refs ON refs.id=listing_assets.id " + condition, + (cutoff,) if cutoff is not None else (), + ).fetchone() + + +def usage(db): + timestamp = now() + with closing(sqlite3.connect(db.path, timeout=30)) as connection: + connection.execute("BEGIN") + image_references(connection) + categories = {} + for kind, measure in (("responses", response_usage), ("images", image_usage)): + size, count = measure(connection) + categories[kind] = { + "bytes": size, + "count": count, + "older_than": { + str(days): dict( + zip(("bytes", "count"), measure(connection, timestamp - days * 86400)) + ) + for days in (7, 30) + }, + } + database_bytes = db.path.stat().st_size + wal = db.path.with_name(db.path.name + "-wal") + try: + journal_bytes = wal.stat().st_size + except FileNotFoundError: + journal_bytes = 0 + return { + **categories, + "database_bytes": database_bytes, + "journal_bytes": journal_bytes, + "total_bytes": database_bytes + journal_bytes, + } + + +def cleanup(db, kind, days): + # The caller drains requests and stops the worker before changing stored history. + cutoff = now() - days * 86400 + db.close() + with closing(sqlite3.connect(db.path, timeout=30)) as connection: + connection.execute("BEGIN IMMEDIATE") + if kind == "responses": + size, count = response_usage(connection, cutoff) + condition = f"{FINISHED} AND {RUN_TIME} < ?" + connection.execute( + f"UPDATE run_payloads SET responses='[]' WHERE run_id IN " + f"(SELECT id FROM runs WHERE {condition})", + (cutoff,), + ) + connection.execute(f"UPDATE runs SET responses='[]' WHERE {condition}", (cutoff,)) + else: + image_references(connection) + size, count = image_usage(connection, cutoff) + connection.execute( + "DELETE FROM listing_assets WHERE id NOT IN " + "(SELECT id FROM image_references WHERE seen >= ?)", + (cutoff,), + ) + row = connection.execute( + "SELECT value FROM settings WHERE key='storage_cleanup'" + ).fetchone() + metadata = json.loads(row[0]) if row else {} + if not isinstance(metadata, dict): + metadata = {} + previous = metadata.get(kind, 0) + metadata[kind] = max(cutoff, previous if isinstance(previous, (int, float)) else 0) + connection.execute( + "INSERT INTO settings VALUES ('storage_cleanup', ?) " + "ON CONFLICT(key) DO UPDATE SET value=excluded.value", + (json.dumps(metadata),), + ) + connection.commit() + compacted = True + try: + connection.execute("VACUUM") + compacted = connection.execute("PRAGMA wal_checkpoint(TRUNCATE)").fetchone()[0] == 0 + except sqlite3.DatabaseError: + # Deletion is already committed; report that disk reclamation needs another attempt. + compacted = False + return { + "removed_bytes": size, + "removed_count": count, + "compacted": compacted, + "usage": usage(db), + } diff --git a/src/apptrail/worker.py b/src/apptrail/worker.py index 66463bf..d8dc613 100644 --- a/src/apptrail/worker.py +++ b/src/apptrail/worker.py @@ -22,6 +22,7 @@ def __init__(self, service): self.thread = None def start(self): + self.stop_event.clear() with self.db.session.begin() as session: for run in session.scalars(select(Run).where(Run.status.in_(["queued", "running"]))): if reason := self.cancellation_reason(session, run): diff --git a/tests/test_backups.py b/tests/test_backups.py new file mode 100644 index 0000000..7d56e76 --- /dev/null +++ b/tests/test_backups.py @@ -0,0 +1,459 @@ +import json +import sqlite3 +import threading +from concurrent.futures import ThreadPoolExecutor +from contextlib import closing + +import pytest +from conftest import TEST_PASSWORD, FakeGateway, add_monitor, sign_in +from fastapi.testclient import TestClient +from sqlalchemy import select +from test_insights import watch +from test_listing_images import collect +from test_listing_images import listing_images as listing_images + +from apptrail import backups +from apptrail.api import create_app +from apptrail.auth import hash_password +from apptrail.db import App, ListingAsset, ListingSnapshot, LoginSession, Owner, Run, Setting + +HEADERS = { + "Content-Type": "application/vnd.sqlite3", + "X-AppTrail-Confirm-Restore": "overwrite", +} + + +@pytest.fixture +def fresh_workspace(tmp_path): + with TestClient( + create_app(tmp_path / "fresh", start_worker=False, gateway_factory=FakeGateway) + ) as client: + client.headers.update(HEADERS | {"X-AppTrail-Request": "1"}) + client.headers["X-AppTrail-Setup-Token"] = client.app.state.auth.setup_token() + yield client + + +def test_setup_restore_uses_backup_account_and_closes_setup(client, tracked, fresh_workspace): + backup = client.get("/api/backup").content + fresh = fresh_workspace + token = fresh.app.state.auth.setup_token() + response = fresh.post("/api/auth/restore", content=backup) + assert response.status_code == 200, response.text + assert fresh.get("/api/auth/status").json() == { + "setup_required": False, + "authenticated": False, + } + assert not fresh.app.state.auth.setup_path.exists() + assert fresh.get("/api/state").status_code == 401 + assert fresh.post("/api/auth/restore", content=backup).status_code == 409 + sign_in(fresh) + assert fresh.get("/api/state").json()["apps"][0]["name"] == "Todo Example" + assert fresh.app.state.service.config.api_key == "" + assert ( + fresh.post( + "/api/auth/setup", + json={ + "username": "intruder", + "password": TEST_PASSWORD, + "setup_token": token, + }, + ).status_code + == 409 + ) + + +@pytest.mark.parametrize( + "headers,status", + [ + ({"X-AppTrail-Setup-Token": ""}, 403), + ({"X-AppTrail-Setup-Token": "wrong"}, 403), + ({"X-AppTrail-Request": ""}, 403), + ({"Sec-Fetch-Site": "cross-site"}, 403), + ({"Content-Type": "text/plain"}, 403), + ({"X-AppTrail-Confirm-Restore": ""}, 422), + ], +) +def test_setup_restore_checks_access_before_reading_upload(fresh_workspace, headers, status): + fresh = fresh_workspace + response = fresh.post("/api/auth/restore", content=b"bad", headers=headers) + assert response.status_code == status + assert fresh.get("/api/auth/status").json()["setup_required"] + assert fresh.app.state.auth.setup_token() + assert not list(fresh.app.state.service.config.directory.glob(".restore-*")) + + +def test_setup_restore_rejects_invalid_files_and_limits_attempts(fresh_workspace): + fresh = fresh_workspace + assert fresh.post("/api/auth/restore", content=b"bad").status_code == 422 + for _ in range(9): + assert ( + fresh.post( + "/api/auth/restore", + content=b"bad", + headers={ + "X-AppTrail-Setup-Token": "wrong", + }, + ).status_code + == 403 + ) + assert fresh.post("/api/auth/restore", content=b"bad").status_code == 429 + assert fresh.get("/api/auth/status").json()["setup_required"] + + +def test_setup_restore_cannot_replace_an_existing_account(client, tracked): + response = client.post( + "/api/auth/restore", + content=b"bad", + headers=HEADERS + | { + "X-AppTrail-Setup-Token": "anything", + }, + ) + assert response.status_code == 409 + assert client.get("/api/state").json()["apps"][0]["id"] == tracked + + +def test_setup_restore_rechecks_account_after_validation( + client, + tracked, + fresh_workspace, + monkeypatch, +): + content = client.get("/api/backup").content + entered, release = threading.Event(), threading.Event() + original = backups.prepare_restore + + def prepare(*args): + path = original(*args) + entered.set() + assert release.wait(10) + return path + + monkeypatch.setattr("apptrail.api.prepare_restore", prepare) + fresh = fresh_workspace + with ThreadPoolExecutor(max_workers=1) as pool: + pending = pool.submit(fresh.post, "/api/auth/restore", content=content) + try: + assert entered.wait(5) + sign_in(fresh, username="new-owner") + finally: + release.set() + assert pending.result(timeout=5).status_code == 409 + assert fresh.get("/api/auth/status").json()["username"] == "new-owner" + assert fresh.get("/api/state").json()["apps"] == [] + + +def restore(client, content, **headers): + return client.post("/api/restore", content=content, headers=HEADERS | headers) + + +def changed_backup(client, tmp_path, sql, parameters=()): + path = tmp_path / "edited.sqlite3" + path.write_bytes(client.get("/api/backup").content) + with closing(sqlite3.connect(path)) as connection: + connection.execute(sql, parameters) + connection.commit() + return path.read_bytes() + + +def test_restore_replaces_data_and_password_revokes_every_session(client, tracked, tmp_path): + add_monitor(client, tracked) + assert client.app.state.worker.process_one() + db = client.app.state.service.db + with db.session.begin() as session: + session.add(ListingAsset(id="image", mime="image/png", content=b"saved-image")) + session.merge(Setting(key="sync_paused", value=True)) + # Include live sessions to exercise restoration from a manually copied database too. + raw = tmp_path / "raw.sqlite3" + db.backup(raw) + with closing(sqlite3.connect(raw)) as connection: + connection.execute( + "UPDATE owner SET username=?, password_hash=?", + ("restored-owner", hash_password("restored password 2!")), + ) + connection.commit() + with TestClient(client.app) as other: + sign_in(other) + old_cookie = client.cookies.get(client.app.state.auth.cookie) + assert ( + client.patch(f"/api/apps/{tracked}", json={"name": "Current name"}).status_code == 200 + ) + with db.session.begin() as session: + session.add(App(name="Only in current database")) + response = restore(client, raw.read_bytes()) + assert response.status_code == 200, response.text + assert "Max-Age=0" in response.headers["set-cookie"] + assert client.get("/api/state").status_code == 401 + assert other.get("/api/state").status_code == 401 + client.cookies.set(client.app.state.auth.cookie, old_cookie) + assert client.get("/api/state").status_code == 401 + client.cookies.clear() + assert ( + client.post( + "/api/auth/login", json={"username": "owner", "password": TEST_PASSWORD} + ).status_code + == 401 + ) + sign_in(client, username="restored-owner", password="restored password 2!") + state = client.get("/api/state").json() + assert [app["name"] for app in state["apps"]] == ["Todo Example"] + assert state["sync_paused"] is True + assert client.app.state.service.config.api_key == "test-credential-not-a-real-key" + with db.session() as session: + assert session.get(ListingAsset, "image").content == b"saved-image" + assert session.scalar(select(Run)).status == "success" + assert len(list(session.scalars(select(LoginSession)))) == 1 + (recovery,) = (tmp_path / "backups").glob("before-restore-*.sqlite3") + with closing(sqlite3.connect(recovery)) as connection: + assert connection.execute("SELECT username FROM owner").fetchone() == ("owner",) + assert connection.execute("SELECT count(*) FROM apps").fetchone() == (2,) + assert connection.execute("SELECT count(*) FROM login_sessions").fetchone() == (0,) + assert recovery.stat().st_mode & 0o777 == 0o600 + assert not list(tmp_path.glob(".restore-*")) + + +@pytest.mark.parametrize("body", [b"", b"not a database", b"SQLite format 3\0" + b"\0" * 100]) +def test_invalid_upload_preserves_workspace_and_login(client, tracked, tmp_path, body): + response = restore(client, body) + assert response.status_code == 422, response.text + assert client.get("/api/state").json()["apps"][0]["id"] == tracked + assert not list(tmp_path.glob(".restore-*")) + assert not list(tmp_path.glob("backups/before-restore-*")) + + +@pytest.mark.parametrize( + "sql", + [ + "PRAGMA user_version=999", + "PRAGMA user_version=5", + "DROP TABLE observations", + "ALTER TABLE apps DROP COLUMN name", + "CREATE TRIGGER surprise AFTER INSERT ON apps BEGIN DELETE FROM owner; END", + "DELETE FROM owner", + "UPDATE owner SET password_hash='bad-hash'", + "UPDATE listings SET app_id=98765", + "UPDATE apps SET aliases='invalid-json'", + "UPDATE apps SET aliases='null'", + "UPDATE listings SET metadata_json='[]'", + ], +) +def test_incompatible_backup_is_rejected_before_changes(client, tracked, tmp_path, sql): + content = changed_backup(client, tmp_path, sql) + response = restore(client, content) + assert response.status_code == 422, response.text + assert client.get("/api/state").json()["apps"][0]["name"] == "Todo Example" + assert not list(tmp_path.glob("backups/before-restore-*")) + + +@pytest.mark.parametrize( + "headers,status", + [ + ({"X-CSRF-Token": "wrong"}, 403), + ({"X-AppTrail-Request": ""}, 403), + ({"Sec-Fetch-Site": "cross-site"}, 403), + ({"Content-Type": "multipart/form-data"}, 403), + ({"X-AppTrail-Confirm-Restore": ""}, 422), + ], +) +def test_restore_requires_csrf_and_explicit_overwrite(client, headers, status): + assert restore(client, b"bad", **headers).status_code == status + assert client.get("/api/state").status_code == 200 + + +def test_upload_limit_checks_declared_and_streamed_size(client, monkeypatch, tmp_path): + monkeypatch.setattr("apptrail.api.MAX_RESTORE_BYTES", 16) + assert restore(client, b"x" * 17).status_code == 413 + assert restore(client, iter([b"x" * 10, b"x" * 10])).status_code == 413 + assert not list(tmp_path.glob(".restore-*")) + assert client.get("/api/state").status_code == 200 + + +def test_compact_backup_shrinks_file_and_restores_history_with_omission_notices( + client, + tracked, + listing_images, + tmp_path, +): + watch_id = watch(client, platform="android") + baseline = collect(client, watch_id, first=True) + before = client.get(f"/api/listing-snapshots/{baseline['id']}").json()["snapshot"] + monitor_id = add_monitor(client, tracked) + assert client.app.state.worker.process_one() + db = client.app.state.service.db + with db.session.begin() as session: + run = session.scalar(select(Run).where(Run.monitor_id == monitor_id)) + run.responses = [{"data": {"large": "RAW_RESPONSE_TO_OMIT" * 100_000}}] + run_id, result = run.id, run.result + session.add(ListingAsset(id="f" * 64, mime="image/png", content=b"IMAGE_TO_OMIT" * 100_000)) + full = tmp_path / "full.sqlite3" + db.backup(full) + content = client.get("/api/backup").content + compact = tmp_path / "compact.sqlite3" + compact.write_bytes(content) + assert compact.stat().st_size < full.stat().st_size / 4 + assert b"RAW_RESPONSE_TO_OMIT" not in content and b"IMAGE_TO_OMIT" not in content + with closing(sqlite3.connect(compact)) as saved: + assert saved.execute("SELECT count(*) FROM listing_assets").fetchone() == (0,) + assert saved.execute( + "SELECT count(*) FROM run_payloads WHERE responses != '[]'" + ).fetchone() == (0,) + assert saved.execute("PRAGMA freelist_count").fetchone() == (0,) + assert saved.execute("PRAGMA quick_check").fetchone() == ("ok",) + original = client.get(f"/api/runs/{run_id}").json() + assert original["responses"] and not original["responses_omitted"] + asset_id = before["data"]["icon"]["asset_id"] + assert client.get(f"/api/listing-assets/{asset_id}").status_code == 200 + + assert restore(client, content).status_code == 200 + sign_in(client) + restored = client.get(f"/api/runs/{run_id}").json() + assert restored["result"] == result + assert restored["observations"] == original["observations"] + assert restored["responses"] == [] and restored["responses_omitted"] + snapshot = client.get(f"/api/listing-snapshots/{baseline['id']}").json()["snapshot"] + assert snapshot["data"] == before["data"] + assert snapshot["images_omitted"] and asset_id in snapshot["missing_assets"] + assert client.get(f"/api/listing-assets/{asset_id}").status_code == 404 + assert client.post(f"/api/apps/{tracked}/reanalyze").json()["updated"] == 1 + + # New checks archive images again and compare against the retained hashes. + after = collect(client, watch_id) + assert after["changes"] == [] + comparison = client.get(f"/api/listing-snapshots/{after['id']}").json() + assert comparison["snapshot"]["missing_assets"] == [] + assert not comparison["snapshot"]["images_omitted"] + assert not comparison["previous"]["images_omitted"] + assert client.get(f"/api/listing-assets/{asset_id}").status_code == 200 + new_run = client.post(f"/api/monitors/{monitor_id}/check").json()["run_id"] + assert client.app.state.worker.process_one() + checked = client.get(f"/api/runs/{new_run}").json() + assert checked["responses"] and not checked["responses_omitted"] + assert client.get(f"/api/runs/{run_id}").json()["responses_omitted"] + + again = tmp_path / "again.sqlite3" + again.write_bytes(client.get("/api/backup").content) + with closing(sqlite3.connect(again)) as saved: + omitted = json.loads( + saved.execute("SELECT value FROM settings WHERE key='backup_omissions'").fetchone()[0] + ) + assert omitted["responses_through_run"] == new_run + assert omitted["images_through_snapshot"] == after["id"] + + +def test_missing_image_without_backup_marker_is_not_mislabelled(client, tracked, listing_images): + baseline = collect(client, watch(client, platform="android"), first=True) + db = client.app.state.service.db + with db.session.begin() as session: + asset_id = session.get(ListingSnapshot, baseline["id"]).data["icon"]["asset_id"] + session.delete(session.get(ListingAsset, asset_id)) + saved = client.get(f"/api/listing-snapshots/{baseline['id']}").json()["snapshot"] + assert asset_id in saved["missing_assets"] + assert not saved["images_omitted"] + + +def test_schema_six_backup_upgrades_in_staging(client, tracked, listing_images, tmp_path): + baseline = collect(client, watch(client, platform="android"), first=True) + db = client.app.state.service.db + legacy = tmp_path / "legacy.sqlite3" + db.backup(legacy) + with closing(sqlite3.connect(legacy)) as saved: + saved.execute("PRAGMA user_version=6") + data = json.loads(saved.execute("SELECT data FROM listing_snapshots").fetchone()[0]) + data["icon"].pop("pixel_hash") + saved.execute("UPDATE listing_snapshots SET data=?", (json.dumps(data),)) + saved.commit() + assert restore(client, legacy.read_bytes()).status_code == 200 + sign_in(client) + snapshot = client.get(f"/api/listing-snapshots/{baseline['id']}").json()["snapshot"] + assert snapshot["data"]["icon"]["pixel_hash"] + assert not snapshot["images_omitted"] + with closing(sqlite3.connect(db.path)) as saved: + assert saved.execute("PRAGMA user_version").fetchone() == (7,) + + +def test_failed_replacement_keeps_data_and_login(client, tracked, monkeypatch): + content = client.get("/api/backup").content + + def fail(_): + raise OSError("Disk unavailable") + + monkeypatch.setattr(client.app.state.service.db, "restore", fail) + with pytest.raises(OSError, match="Disk unavailable"): + restore(client, content) + assert client.get("/api/state").json()["apps"][0]["id"] == tracked + + +def test_restore_blocks_requests_and_waits_for_active_work(client, tracked, monkeypatch): + content = client.get("/api/backup").content + service = client.app.state.service + worker = client.app.state.worker + entered, release = threading.Event(), threading.Event() + original = service.state + + def slow_state(): + entered.set() + assert release.wait(10) + return original() + + monkeypatch.setattr(service, "state", slow_state) + exclusive = backups.RestoreGate.exclusive + waiting = threading.Event() + + def tracked_exclusive(gate): + waiting.set() + return exclusive(gate) + + monkeypatch.setattr(backups.RestoreGate, "exclusive", tracked_exclusive) + with ThreadPoolExecutor(max_workers=2) as pool: + reading = pool.submit(client.get, "/api/state") + assert entered.wait(5) + restoring = pool.submit(restore, client, content) + try: + assert waiting.wait(5) + assert not restoring.done() + assert client.get("/healthz").status_code == 200 + assert client.get("/api/auth/status").status_code == 503 + finally: + release.set() + assert reading.result(timeout=5).status_code == 200 + assert restoring.result(timeout=5).status_code == 200 + assert worker.thread is None + + +def test_restore_waits_for_worker_and_restarts_it(client, tracked, monkeypatch): + db = client.app.state.service.db + worker = client.app.state.worker + content = client.get("/api/backup").content + entered, release = threading.Event(), threading.Event() + calls = [] + + def loop(): + calls.append("start") + entered.set() + assert release.wait(10) + with db.session.begin() as session: + session.add(App(name="Old worker result")) + calls.append("finish") + worker.stop_event.wait(10) + + monkeypatch.setattr(worker, "loop", loop) + worker.start() + assert entered.wait(5) + with ThreadPoolExecutor(max_workers=1) as pool: + pending = pool.submit(restore, client, content) + try: + assert worker.stop_event.wait(5) + assert not pending.done() + # The restarted worker waits instead of writing another old result. + monkeypatch.setattr(worker, "loop", lambda: worker.stop_event.wait(10)) + release.set() + assert pending.result(timeout=5).status_code == 200 + assert not worker.stop_event.is_set() + assert worker.thread.is_alive() + with db.session() as session: + assert list(session.scalars(select(App.name))) == ["Todo Example"] + assert session.scalar(select(Owner)).username == "owner" + assert calls == ["start", "finish"] + finally: + release.set() + worker.stop() diff --git a/tests/test_backups_browser.py b/tests/test_backups_browser.py new file mode 100644 index 0000000..24d24f3 --- /dev/null +++ b/tests/test_backups_browser.py @@ -0,0 +1,215 @@ +import os + +import pytest +from conftest import TEST_PASSWORD, add_monitor, sign_in +from playwright.sync_api import expect +from sqlalchemy import delete +from test_backups import restore +from test_insights import watch +from test_insights_browser import browser as browser # noqa: F401 +from test_insights_browser import browser_page as browser_page # noqa: F401 +from test_insights_browser import login +from test_listing_images import collect +from test_listing_images import listing_images as listing_images + +from apptrail.auth import Auth +from apptrail.db import App, LoginSession, Owner + +pytestmark = [ + pytest.mark.browser, + pytest.mark.skipif( + os.getenv("APPTRAIL_BROWSER_TESTS") != "1", reason="Opt-in local browser check" + ), +] + + +@pytest.mark.parametrize("width", [1440, 390]) +def test_settings_restore_warning_validation_and_logout( + client, tracked, browser_page, tmp_path, width +): + backup = tmp_path / "restore.sqlite3" + backup.write_bytes(client.get("/api/backup").content) + new_password = "current password after backup 3!" + response = client.post( + "/api/auth/password", + json={"current_password": TEST_PASSWORD, "new_password": new_password}, + ) + assert response.status_code == 200 + client.headers["X-CSRF-Token"] = response.json()["csrf_token"] + assert client.patch(f"/api/apps/{tracked}", json={"name": "New name"}).status_code == 200 + page, url = browser_page + page.set_viewport_size({"width": width, "height": 1000}) + page.goto(url + "/login") + page.get_by_role("textbox", name="Username").fill("owner") + page.get_by_label("Password", exact=True).fill(new_password) + page.locator("#auth-submit").click() + page.wait_for_url(url + "/") + page.goto(url + "/#settings") + backup_panel = page.locator(".settings-panel").filter( + has=page.get_by_role("heading", name="Data & backups") + ) + expect(backup_panel).to_contain_text("CSV files cannot restore your workspace") + expect(backup_panel).to_contain_text( + "Raw SerpApi responses and listing-history images are omitted" + ) + expect(backup_panel).to_contain_text("Login sessions and your SerpApi key are not included") + expect(backup_panel.get_by_role("link", name="Download SQLite backup")).to_have_attribute( + "href", "/api/backup" + ) + page.get_by_role("button", name="Restore from SQLite", exact=True).click() + expect(page.locator("#modal")).to_contain_text("This will overwrite all current server data") + expect(page.locator("#modal")).to_contain_text("Everyone will be signed out") + page.screenshot(path=str(tmp_path / f"restore-dialog-{width}.png")) + assert page.locator("#modal").evaluate("el => el.scrollWidth <= el.clientWidth") + page.get_by_role("button", name="Cancel", exact=True).click() + expect(page.locator("#modal")).not_to_be_visible() + assert client.get("/api/state").json()["apps"][0]["name"] == "New name" + + page.get_by_role("button", name="Restore from SQLite", exact=True).click() + page.get_by_label("SQLite backup file").set_input_files( + {"name": "bad.sqlite3", "mimeType": "application/vnd.sqlite3", "buffer": b"bad"} + ) + page.get_by_role("button", name="Overwrite and restore").click() + assert page.locator("#restore-backup-form").evaluate("el => !el.checkValidity()") + page.get_by_role("checkbox").check() + page.get_by_role("button", name="Overwrite and restore").click() + expect(page.locator("#modal-error")).to_contain_text("Upload an AppTrail SQLite backup") + expect(page.get_by_role("button", name="Overwrite and restore")).to_be_enabled() + assert client.get("/api/state").json()["apps"][0]["name"] == "New name" + + page.get_by_label("SQLite backup file").set_input_files(backup) + page.get_by_role("button", name="Overwrite and restore").click() + page.wait_for_url(url + "/login?restored=1") + expect(page.locator("#auth-description")).to_contain_text("Backup restored") + expect(page.locator("#auth-description")).to_contain_text("password saved in that backup") + assert client.get("/api/state").status_code == 401 + login(page, url) + expect(page.locator("#main")).to_contain_text("Todo Example") + + +@pytest.mark.parametrize("width", [1440, 390]) +def test_setup_restore_is_secondary_and_restores_without_creating_account( + client, + tracked, + browser_page, + tmp_path, + width, +): + backup = tmp_path / "setup-restore.sqlite3" + backup.write_bytes(client.get("/api/backup").content) + service = client.app.state.service + with service.db.session.begin() as session: + session.execute(delete(LoginSession)) + session.execute(delete(Owner)) + session.execute(delete(App)) + Auth(service.db, service.config) + token = client.app.state.auth.setup_token() + page, url = browser_page + page.set_viewport_size({"width": width, "height": 1000}) + page.goto(url + "/login") + expect(page.get_by_role("heading", name="Create your account")).to_be_visible() + expect(page.locator("#setup-restore-form")).to_be_hidden() + expect(page.get_by_role("button", name="Create account", exact=True)).to_be_visible() + entry = page.get_by_role("button", name="Have a backup? Restore your data") + expect(entry).to_be_visible() + assert entry.evaluate("el => getComputedStyle(el).fontSize") == "12px" + page.get_by_role("textbox", name="Setup code", exact=True).fill(token) + page.screenshot(path=str(tmp_path / f"setup-{width}.png"), full_page=True) + entry.click() + expect(page.get_by_role("heading", name="Restore your data")).to_be_visible() + expect(page.locator("#auth-description")).to_contain_text("after an update or move") + expect(page.get_by_role("textbox", name="Setup code", exact=True)).to_have_value(token) + page.get_by_role("button", name="Back to account setup").click() + expect(page.get_by_role("button", name="Create account", exact=True)).to_be_visible() + entry.click() + page.get_by_label("SQLite backup file").set_input_files(backup) + page.get_by_role("button", name="Restore backup", exact=True).click() + assert page.locator("#setup-restore-form").evaluate("el => !el.checkValidity()") + page.get_by_role("checkbox").check() + page.get_by_role("textbox", name="Setup code", exact=True).fill("wrong") + page.get_by_role("button", name="Restore backup", exact=True).click() + expect(page.locator("#setup-restore-error")).to_contain_text("setup code is incorrect") + page.get_by_role("textbox", name="Setup code", exact=True).fill(token) + page.get_by_label("SQLite backup file").set_input_files( + { + "name": "bad.sqlite3", + "mimeType": "application/vnd.sqlite3", + "buffer": b"bad", + } + ) + page.get_by_role("button", name="Restore backup", exact=True).click() + expect(page.locator("#setup-restore-error")).to_contain_text("Upload an AppTrail SQLite backup") + assert client.get("/api/auth/status").json()["setup_required"] + page.get_by_label("SQLite backup file").set_input_files(backup) + page.screenshot(path=str(tmp_path / f"setup-restore-{width}.png"), full_page=True) + page.get_by_role("button", name="Restore backup", exact=True).click() + page.wait_for_url(url + "/login?restored=1") + expect(page.locator("#auth-description")).to_contain_text("Backup restored") + expect(entry).to_be_hidden() + assert client.get("/api/auth/status").json()["setup_required"] is False + login(page, url) + expect(page.locator("#main")).to_contain_text("Todo Example") + sign_in(client) + assert client.get("/api/state").json()["apps"][0]["name"] == "Todo Example" + + +@pytest.mark.parametrize("width", [1440, 390]) +def test_restored_backup_explains_omissions_without_broken_images( + client, + tracked, + listing_images, + browser_page, + tmp_path, + width, +): + watch_id = watch(client, platform="android") + collect(client, watch_id, first=True) + monitor_id = add_monitor(client, tracked) + assert client.app.state.worker.process_one() + run_id = next( + run["id"] + for run in client.get("/api/dashboard").json()["runs"] + if run["monitor_id"] == monitor_id + ) + assert restore(client, client.get("/api/backup").content).status_code == 200 + sign_in(client) + page, url = browser_page + page.set_viewport_size({"width": width, "height": 1000}) + requested_images = [] + page.on( + "request", + lambda request: ( + requested_images.append(request.url) if "/api/listing-assets/" in request.url else None + ), + ) + login(page, url) + page.goto(url + "/#listing-history") + page.locator(".timeline-event").first.click() + expect(page.locator("#modal .backup-omission")).to_contain_text( + "images were omitted from this backup" + ) + expect(page.locator("#modal .unavailable-image").first).to_have_text( + "Image omitted from backup to save space" + ) + expect(page.locator("#modal img")).to_have_count(0) + assert requested_images == [] + page.screenshot(path=str(tmp_path / f"omitted-images-{width}.png")) + page.get_by_role("button", name="Close dialog").click() + page.goto(url + "/#activity") + page.locator(f'[data-action="run"][data-id="{run_id}"]').click() + expect(page.locator("#modal .backup-omission")).to_contain_text( + "Original SerpApi responses were omitted" + ) + expect(page.locator("#modal")).to_contain_text("Store results") + expect(page.locator("#modal .evidence")).not_to_have_count(0) + page.screenshot(path=str(tmp_path / f"omitted-responses-{width}.png")) + page.get_by_role("button", name="Close dialog").click() + + # Subsequent collection can display images again and should not claim they were omitted. + collect(client, watch_id) + page.goto(url + "/#listing-history") + page.locator(".timeline-event").first.click() + expect(page.locator("#modal .backup-omission")).to_have_count(0) + expect(page.locator("#modal .unavailable-image")).to_have_count(0) + page.wait_for_function("""() => [...document.querySelectorAll('#modal img')] + .some(img => img.complete && img.naturalWidth > 0)""") diff --git a/tests/test_storage.py b/tests/test_storage.py new file mode 100644 index 0000000..4be6b3c --- /dev/null +++ b/tests/test_storage.py @@ -0,0 +1,231 @@ +import hashlib +import sqlite3 +import threading +from concurrent.futures import ThreadPoolExecutor +from contextlib import closing + +import pytest +from sqlalchemy import select +from test_insights import watch + +from apptrail import storage +from apptrail.db import ListingAsset, ListingSnapshot, Observation, Run, now + + +def asset_id(name): + return hashlib.sha256(name.encode()).hexdigest() + + +@pytest.fixture +def stored_history(client, tracked, monkeypatch): + timestamp = now() + monkeypatch.setattr(storage, "now", lambda: timestamp) + watch_id = watch(client, platform="android") + runs, snapshots = {}, {} + db = client.app.state.service.db + with db.session.begin() as session: + for age in (40, 10, 7, 3): + run = Run( + kind="search", + status="success", + created_at=timestamp - age * 86400, + started_at=timestamp - age * 86400, + finished_at=timestamp - age * 86400, + params={"source": "google_play", "query": f"Search {age}", "country": "us"}, + result={"kind": "store", "items": [], "results_checked": 1}, + responses=[{"raw": "response" * 150_000 + "é"}], + ) + session.add(run) + session.flush() + session.add( + Observation( + run_id=run.id, + app_id=tracked, + data={ + "position": 1, + "found": True, + "evidence": [{"type": "app_id", "text": "Matched app"}], + }, + ) + ) + session.add( + ListingAsset( + id=asset_id(f"image-{age}"), mime="image/webp", content=b"image" * 100_000 + ) + ) + snapshot = ListingSnapshot( + watch_id=watch_id, + run_id=run.id, + checked_at=run.finished_at, + data={ + "title": f"Listing {age}", + "description": "Saved listing text", + "icon": {"asset_id": asset_id(f"image-{age}"), "pixel_hash": f"pixels-{age}"}, + }, + changes=["title", "icon"], + baseline=age == 40, + ) + session.add(snapshot) + session.flush() + runs[age], snapshots[age] = run.id, snapshot.id + session.add( + ListingAsset(id=asset_id("shared"), mime="image/webp", content=b"shared" * 100_000) + ) + for age in (40, 3): + snapshot = session.get(ListingSnapshot, snapshots[age]) + snapshot.data = { + **snapshot.data, + "ipad_screenshots": [ + {"asset_id": asset_id("shared"), "pixel_hash": "shared-pixels"} + ], + } + session.add(ListingAsset(id=asset_id("orphan"), mime="image/webp", content=b"unused")) + session.add( + Run( + kind="profile", + status="running", + created_at=timestamp - 60 * 86400, + responses=[{"active": True}], + ) + ) + return {"runs": runs, "snapshots": snapshots, "timestamp": timestamp} + + +def clean(client, kind="responses", days=7): + response = client.post( + "/api/storage/cleanup", json={"kind": kind, "days": days, "confirm": True} + ) + assert response.status_code == 200, response.text + return response.json() + + +def test_storage_counts_payload_bytes_shared_images_and_cutoffs(client, stored_history): + usage = client.get("/api/storage").json() + db = client.app.state.service.db + with closing(sqlite3.connect(db.path)) as connection: + expected = connection.execute( + "SELECT SUM(length(CAST(responses AS BLOB))) FROM run_payloads WHERE responses != '[]'" + ).fetchone()[0] + assert usage["responses"]["bytes"] == expected + assert usage["responses"]["older_than"]["7"]["count"] == 2 + assert usage["responses"]["older_than"]["30"]["count"] == 1 + assert usage["images"]["bytes"] == 2_600_006 + assert usage["images"]["count"] == 6 + assert usage["images"]["older_than"]["7"] == {"bytes": 1_000_006, "count": 3} + assert usage["images"]["older_than"]["30"] == {"bytes": 500_006, "count": 2} + assert usage["total_bytes"] == usage["database_bytes"] + usage["journal_bytes"] + assert usage["total_bytes"] > usage["images"]["bytes"] + expected + + +@pytest.mark.parametrize("days, removed", [(7, {40, 10}), (30, {40})]) +def test_response_cleanup_preserves_results_images_and_active_runs( + client, stored_history, days, removed +): + before = client.get("/api/storage").json() + originals = { + age: client.get(f"/api/runs/{run_id}").json() + for age, run_id in stored_history["runs"].items() + } + result = clean(client, days=days) + assert result["removed_count"] == len(removed) + assert result["removed_bytes"] == before["responses"]["older_than"][str(days)]["bytes"] + assert result["compacted"] + assert result["usage"]["total_bytes"] < before["total_bytes"] - result["removed_bytes"] // 2 + assert result["usage"]["images"] == before["images"] + for age, run_id in stored_history["runs"].items(): + run = client.get(f"/api/runs/{run_id}").json() + assert bool(run["responses"]) == (age not in removed) + assert run["responses_cleaned"] == (age in removed) + assert not run["responses_omitted"] + assert run["result"] == originals[age]["result"] + assert run["observations"] == originals[age]["observations"] + with client.app.state.service.db.session() as session: + active = session.scalar(select(Run).where(Run.status == "running")) + assert active.responses == [{"active": True}] + assert clean(client, days=days)["removed_bytes"] == 0 + + +@pytest.mark.parametrize("days, removed", [(7, {40, 10}), (30, {40})]) +def test_image_cleanup_protects_recent_references_and_keeps_hashes( + client, stored_history, days, removed +): + original = client.get("/api/storage").json() + result = clean(client, "images", days) + assert result["removed_count"] == len(removed) + 1 + assert result["usage"]["responses"] == original["responses"] + for age, snapshot_id in stored_history["snapshots"].items(): + snapshot = client.get(f"/api/listing-snapshots/{snapshot_id}").json()["snapshot"] + assert snapshot["images_cleaned"] == (age in removed) + assert not snapshot["images_omitted"] + assert snapshot["missing_assets"] == ([asset_id(f"image-{age}")] if age in removed else []) + assert snapshot["data"]["icon"]["pixel_hash"] == f"pixels-{age}" + assert snapshot["data"]["description"] == "Saved listing text" + assert snapshot["changes"] == ["title", "icon"] + assert client.get(f"/api/listing-assets/{asset_id('shared')}").status_code == 200 + assert client.get(f"/api/listing-assets/{asset_id('orphan')}").status_code == 404 + assert client.get("/api/state").status_code == 200 + + +def test_legacy_responses_and_cleanup_marker_survive_different_cutoffs(client, stored_history): + db = client.app.state.service.db + with db.session.begin() as session: + session.get(Run, stored_history["runs"][10]).legacy_responses = [{"legacy": True}] + result = clean(client) + assert result["removed_count"] == 3 + with db.session() as session: + assert session.get(Run, stored_history["runs"][10]).legacy_responses == [] + clean(client, days=30) + assert client.get(f"/api/runs/{stored_history['runs'][10]}").json()["responses_cleaned"] + + +@pytest.mark.parametrize( + "payload", + [ + {"kind": "responses", "days": 1, "confirm": True}, + {"kind": "everything", "days": 7, "confirm": True}, + {"kind": "images", "days": 7}, + {"kind": "images", "days": 7, "confirm": False}, + ], +) +def test_cleanup_requires_explicit_scope_and_confirmation(client, stored_history, payload): + assert client.post("/api/storage/cleanup", json=payload).status_code == 422 + assert client.get("/api/storage").json()["images"]["count"] == 6 + + +def test_storage_requires_authentication_and_csrf(client): + payload = {"kind": "images", "days": 7, "confirm": True} + assert ( + client.post( + "/api/storage/cleanup", json=payload, headers={"Sec-Fetch-Site": "cross-site"} + ).status_code + == 403 + ) + del client.headers["X-CSRF-Token"] + assert client.post("/api/storage/cleanup", json=payload).status_code == 403 + client.cookies.clear() + assert client.get("/api/storage").status_code == 401 + assert client.post("/api/storage/cleanup", json=payload).status_code == 401 + + +def test_cleanup_blocks_concurrent_requests_and_recovers_after_failure(client, monkeypatch): + entered, release = threading.Event(), threading.Event() + + def failing_cleanup(*args): + entered.set() + assert release.wait(5) + raise ValueError("Cleanup failed") + + monkeypatch.setattr(storage, "cleanup", failing_cleanup) + with ThreadPoolExecutor() as executor: + future = executor.submit( + client.post, "/api/storage/cleanup", json={"kind": "images", "days": 7, "confirm": True} + ) + assert entered.wait(5) + try: + response = client.get("/api/state") + assert response.status_code == 503 + assert "Storage cleanup" in response.json()["detail"] + finally: + release.set() + assert future.result().status_code == 422 + assert client.get("/api/state").status_code == 200 diff --git a/tests/test_storage_browser.py b/tests/test_storage_browser.py new file mode 100644 index 0000000..dbe0f42 --- /dev/null +++ b/tests/test_storage_browser.py @@ -0,0 +1,88 @@ +import os + +import pytest +from playwright.sync_api import expect +from test_insights_browser import browser as browser # noqa: F401 +from test_insights_browser import browser_page as browser_page # noqa: F401 +from test_insights_browser import login +from test_listing_images import image_bytes +from test_storage import asset_id +from test_storage import stored_history as stored_history + +from apptrail.db import ListingAsset + +pytestmark = [ + pytest.mark.browser, + pytest.mark.skipif( + os.getenv("APPTRAIL_BROWSER_TESTS") != "1", reason="Opt-in local browser check" + ), +] + + +@pytest.mark.parametrize("width", [1440, 390]) +def test_storage_controls_cleanup_and_history_notices( + client, stored_history, browser_page, tmp_path, width +): + with client.app.state.service.db.session.begin() as session: + session.get(ListingAsset, asset_id("shared")).content = image_bytes() + page, url = browser_page + page.set_viewport_size({"width": width, "height": 1000}) + login(page, url) + page.goto(url + "/#settings") + panel = page.locator("#storage-panel") + expect(panel).to_contain_text("Total database size") + expect(panel).to_contain_text("Saved SerpApi responses") + expect(panel).to_contain_text("Listing-history images") + assert panel.evaluate("el => el.nextElementSibling.textContent.includes('Data & backups')") + assert panel.evaluate("el => el.scrollWidth <= el.clientWidth") + panel.screenshot(path=str(tmp_path / f"storage-{width}.png")) + responses = page.locator('[data-storage-days="responses"]') + responses.select_option("30") + panel.get_by_role("button", name="Delete older responses", exact=True).click() + expect(page.locator("#modal")).to_contain_text("older than 30 days") + page.get_by_role("button", name="Cancel", exact=True).click() + assert client.get(f"/api/runs/{stored_history['runs'][40]}").json()["responses"] + responses.select_option("7") + panel.get_by_role("button", name="Delete older responses", exact=True).click() + expect(page.locator("#modal")).to_contain_text("older than 7 days") + expect(page.locator("#modal")).to_contain_text("cannot be undone") + page.get_by_role("button", name="Delete older content", exact=True).click() + expect(page.locator("#modal")).not_to_be_visible() + expect(panel.get_by_role("button", name="Delete older responses", exact=True)).to_be_disabled() + assert client.get(f"/api/runs/{stored_history['runs'][3]}").json()["responses"] + + page.locator('[data-storage-days="images"]').select_option("30") + panel.get_by_role("button", name="Delete older images", exact=True).click() + expect(page.locator("#modal")).to_contain_text( + "Images shared with newer snapshots will be kept" + ) + page.get_by_role("button", name="Delete older content", exact=True).click() + expect(page.locator("#modal")).not_to_be_visible() + assert client.get(f"/api/listing-assets/{asset_id('image-10')}").status_code == 200 + page.locator('[data-storage-days="images"]').select_option("7") + panel.get_by_role("button", name="Delete older images", exact=True).click() + page.get_by_role("button", name="Delete older content", exact=True).click() + expect(page.locator("#modal")).not_to_be_visible() + expect(panel.get_by_role("button", name="Delete older images", exact=True)).to_be_disabled() + expect(panel).to_contain_text("0 B eligible for deletion") + panel.screenshot(path=str(tmp_path / f"storage-cleaned-{width}.png")) + + page.goto(url + "/#activity") + page.locator(f'[data-action="run"][data-id="{stored_history["runs"][10]}"]').click() + expect(page.locator("#modal .storage-omission")).to_contain_text( + "deleted during storage cleanup" + ) + expect(page.locator("#modal .evidence")).to_contain_text("Matched app") + page.get_by_role("button", name="Close dialog").click() + page.goto(url + "/#listing-history") + page.locator( + f'[data-action="listing-snapshot"][data-id="{stored_history["snapshots"][10]}"]' + ).click() + expect(page.locator("#modal .storage-omission")).to_contain_text( + "deleted during storage cleanup" + ) + expect(page.locator("#modal .unavailable-image").first).to_have_text( + "Image deleted during storage cleanup" + ) + expect(page.locator("#modal .diff-text")).to_have_text(["Listing 40", "Listing 10"]) + page.screenshot(path=str(tmp_path / f"cleaned-history-{width}.png")) diff --git a/uv.lock b/uv.lock index a74a681..88b5193 100644 --- a/uv.lock +++ b/uv.lock @@ -39,7 +39,7 @@ wheels = [ [[package]] name = "apptrail" -version = "0.4.1" +version = "1.0.0" source = { editable = "." } dependencies = [ { name = "argon2-cffi" },