Skip to content

Latest commit

 

History

History
1145 lines (1075 loc) · 81.9 KB

SKELETON.md

File metadata and controls

1145 lines (1075 loc) · 81.9 KB

Server skeleton: how this crate is put together

Read PORT.md first; this file is the map of what already exists, how to add to it, and where the port deliberately differs from upstream.

1. Run, test, lint

server/scripts/dev.sh            # start the dev DB (127.0.0.1:55421), migrate, serve on 127.0.0.1:8421
server/scripts/dev.sh migrate    # only migrate + seed
cd server
cargo test                       # unit + integration tests (no database needed)
cargo clippy --all-targets -- -D warnings
cargo fmt --check                # rustfmt.toml: max_width 120

dev.sh generates server/docker/.env.devdb (database password, gitignored) on first use and never prints it. The file lives in the MAIN checkout (/path/to/modlight/server/docker/.env.devdb, found through the shared git directory): every worktree uses the same database volume, so they must share one password. Extra variables for the server can go into server/.env (gitignored, loaded like dotenv/config upstream; variables already in the shell win). Logs go to stderr, filtered by RUST_LOG.

Database tests (tests/db_compat.rs) run only when a server they may create databases on is given; each run creates and drops its own database:

set -a; source "$(dirname "$(git rev-parse --path-format=absolute --git-common-dir)")/server/docker/.env.devdb"; set +a
MODLIGHT_TEST_DATABASE_URL="postgresql://modlight:${POSTGRES_PASSWORD}@127.0.0.1:55421/modlight" \
  cargo test --test db_compat

CLI: modlight-server [serve] (migrate, then serve; the upstream container ran migrate.js before server.js) and modlight-server migrate (migrate.ts: raw DATABASE_URL, PG* variables when unset, no production checks).

Environment: everything lib/config.ts reads (see src/config.rs, ENV_VARS), plus BIND_ADDR (default 127.0.0.1; the Docker image sets 0.0.0.0 inside the container), MODLIGHT_ROOT (where migrations/, messages/, public/ and web/dist/ live; default: the working directory if it contains migrations/, else the crate directory), TZ (the timeZone handed to the SPA, default UTC like the upstream container) and RUST_LOG. NODE_ENV=production still means production (refusals in config.rs, production CSP, sanitised action errors), so upstream .env files keep their meaning.

2. Module map

Rust Upstream Status
main.rs server.ts, migrate.ts done: CLI, runtime dirs, migrate-then-serve, jobs, graceful shutdown
http.rs server.ts request handling done: raw upload bypass, path normalisation, router assembly, hyper loop without timeouts
http/proxy.rs proxy.ts done
http/headers.rs next.config.ts headers() done (CSP computed at startup)
http/cookies.rs, http/set_cookie.rs Next / better-call / better-auth cookie parsing and serialisation done
http/static_files.rs public/, app/icon.svg, app/apple-icon.png, SPA assets/ done
config.rs lib/config.ts done
paths.rs (working directory conventions) done
error.rs (typed error at the edge) done
api_json.rs lib/api-json.ts + the per-route json() helpers done
db/mod.rs, db/migrate.rs lib/db/index.ts, migrate.ts + drizzle 0.45.2 migrator done
db/models.rs, db/types.rs lib/db/schema/*.ts + drizzle/*.sql done (every table and column, checked against information_schema)
rpc/ Next server actions done (framework)
actions/tokens.rs app/actions/tokens.ts done
actions/{uploads,folders,duplicates,shortlinks,shortlink_unlock,pastes}.rs app/actions/ same names done (parity: tests/parity/library_actions.py)
actions/admin_library.rs app/actions/admin.ts (loadOrphans, deleteOrphan, purgeAllOrphans, loadDeletionImpact, deleteUserAccount, recalculateEveryQuota, runStorageScan), components/admin/confirm.ts done; registered as module admin beside the accounts port's own actions/admin.rs
js_value.rs (JavaScript operators on raw arguments) done
actions/avatar.rs app/actions/avatar.ts done (myAvatar is in CONTRACT 7.3 but unreachable on the reference build: no client component imports it)
actions/convert.rs app/actions/convert.ts done (parity: tests/parity_media/convert_actions.py)
actions/unlock.rs app/actions/unlock.ts done (verifyMediaPassword; parity media_serve.py --only gates)
pages/ page data, SPA shell, metadata done (framework + table of every page and route handler)
pages/root_layout.rs app/layout.tsx done
pages/layout_app.rs, pages/layout_auth.rs, account pages app/(app)/layout.tsx, app/(auth)/layout.tsx, /, /welcome, /download, auth pages, /dashboard/{profile,tokens,admin} done (accounts port; the (app) layout reads the library's data functions)
pages/dashboard*.rs, pages/shortlink_unlock.rs /dashboard, /dashboard/{folders,trash,duplicates,links,paste,upload}, /dashboard/admin/users/[id]/files, /s/[code]/unlock, /s/[code]/[name]/unlock done (parity: tests/parity/library_pages.py)
pages/media.rs, pages/public_profile.rs, pages/public_album.rs app/[userCode]/[mediaCode]/page.tsx, app/[userCode]/page.tsx, app/u/[userCode]/page.tsx, app/a/[userCode]/[slug]/page.tsx done: props, generateMetadata (Open Graph, Twitter, oEmbed and link-info discovery), the owner's skin and wallpaper on <html>, the tagless action redirect (parity media_serve.py)
i18n/ i18n/request.ts, next-intl formatting done (locale, messages, ICU subset used by the dictionaries)
themes.rs, theme_cookies.rs lib/themes.ts, lib/theme-cookies.ts done
auth/cookies.rs, auth/session.rs better-auth 1.6.13 cookies + getSession done
auth_helpers.rs lib/auth-helpers.ts done (functions + axum extractors)
auth/api.rs, auth/password.rs, auth/two_factor.rs, auth/admin.rs, auth/email.rs /api/auth/*, better-auth plugins layout + docs only (auth agent)
api_scopes.rs, api_token.rs, api_caller.rs lib/api-scopes.ts, lib/api-token.ts, lib/api-caller.ts done
password.rs lib/password.ts done: client_ip, rate limiter, file-password hashing (accounts port); unlock tokens and cookies, owner-scoped consent (library port)
ids.rs, audit.rs lib/ids.ts, lib/audit.ts done
storage/ lib/storage/* done: local driver; S3 driver on aws-sdk-s3 with ports of lib-storage Upload (s3/lib_storage.rs) and s3-request-presigner (s3/presigner.rs, URLs byte-identical to the JS SDK's); is_not_found
upload/tus_server/s3_store.rs @tus/s3-store 2.0.4 done (S3Store::from_env = makeDatastore() for S3); keys, .info JSON, metadata and tags as the TS store, uploads resume across the two
routes/api/health.rs app/api/health/route.ts done
jobs.rs timers in server.ts done: tus sweep (upload/tus.rs), trash purge (maintenance/trash.rs), mail settings (accounts port)
upload/sharex.rs, upload/busboy.rs lib/upload/sharex.ts + busboy 1.6's part rules done (security fix 2; parity library.py --only sharex)
upload/tus.rs, upload/tus_server/{server,store,file_store}.rs lib/upload/tus.ts, @tus/server 2.4.1, @tus/utils 0.7.0, @tus/file-store 2.1.0 done (records interchangeable with the TS store; tus-js-client 4.3.1 resumes across both; tests/parity/tus_client.cjs)
upload/{finalize,disk,folder_target,duplicate_lookup,duplicate_check}.rs lib/upload/* done
media/{sniff,mime,urls,paths,paste_type,actions,derived_cache,convert_targets,av_targets}.rs lib/media/* same names done (derived_cache: keys, purge, mayCacheDerived; reading and writing cached conversions belongs to the media serving port)
media/serve.rs lib/media/serve.ts done: getMedia (+ lookup_media, which tells a spent budget apart), viewerRelation, consumeView, cacheHeadersFor, parseRange, streamMedia; security fix 1 (the view grant, ViewBudget)
media/{bots,render_ticket,embed,link_formats}.rs lib/media/ same names done (media serving port)
media/{convert,convert_plan,convert_size,convert_verdict,convert_notes,convert_in_place,download_menu,av_convert,av_action,motion,motion_preview,timeline,timeline_preview,poster,frame_step,source_preview,card,card_art,card_render,player_i18n,playback_health}.rs, media/pdf_lib.rs lib/media/ same names, the part of pdf-lib 1.17.1 convert.ts uses done (media processing port: section 12)
media/part_a.rs the processing half of lib/media as the serving half calls it the serving port's adapter, wired to the processing modules (section 12)
crypto/envelope.rs lib/crypto/envelope.ts server part: constants, isEncryptedUpload (the rest runs in the browser)
routes/media_action.rs app/[userCode]/[mediaCode]/[action]/route.ts done: raw, raw.<ext>, download, render, timeline.vtt/jpg, convert (the engines through part_a)
routes/{media_delete,album_zip,embed_card,oembed}.rs, archiver.rs app/[userCode]/[mediaCode]/delete/[token], app/a/[userCode]/[slug]/zip, app/embed/[version]/[card], app/oembed; archiver 8 / zip-stream 7 / compress-commons 7 done (security fixes 5 and 6; the ZIP writer is archiver's zip layout for stream entries)
http/web_response.rs Fetch new Response(text) defaults done (text/plain;charset=UTF-8, headers object semantics)
media/vips.rs + media/vips/*, media/{image,probe,avif_brand,blurhash,sharp_runtime,transform_cost,transform_queue,avatar,abort}.rs sharp 0.34.5 / libvips 8.17, lib/media/{image,probe,avif-brand,sharp-runtime,transform-cost,transform-queue,avatar}.ts, blurhash 2.0.5 done (section 11)
media/{ffmpeg,av_probe,av_cost,av_workspace,av_meta}.rs lib/media/* same names done
media/{docx,docx_support,zip_guard}.rs, media/mammoth/ lib/media/{docx,docx-support,zip-guard}.ts, mammoth 1.12.1 (+ JSZip 3.10.1, @xmldom/xmldom 0.8.13, lop 0.4.2, dingbat-to-unicode 1.0.1) done (section 13)
naming.rs, naming_words.rs, folders.rs, uploads.rs, shortlinks.rs, quota.rs, format.rs, avatar.rs, avatar_store.rs, background.rs, paste_languages.rs, js_date.rs, link_info/parse.rs, maintenance/trash.rs lib/ same names (js_date.rs: V8's new Date(string)) done
data/{uploads,upload_query,folders,duplicates,admin_files}.rs lib/data/ same names done
routes/api/{uploads,uploads_check,folders,duplicates,shortlinks,link_info,avatar,background,admin_disk}.rs, routes/{avatar,s_code,s_code_name}.rs app/api/{uploads,uploads/check,folders,duplicates,shortlinks,link-info,avatar,background,admin/disk}, app/avatar/[userCode], app/bg/[userCode], app/s/[code], app/s/[code]/[name] done (parity: library.py, library_routes.py)
runtime/limits.rs lib/runtime/limits.ts done
turnstile.rs, actions/turnstile.rs none (not upstream) optional Cloudflare Turnstile on the public entry points and its admin card (section 9, item 43)
downloads.rs, routes/api/downloads.rs, pages/dashboard_downloads.rs none (not upstream) Downloads: the desktop app's builds for macOS and Windows, published with RELEASE_PUBLISH_TOKEN (section 9, item 45)
media/strip.rs, media/strip/{jpeg,png,webp,gif,heif,raster,av,pdf,office}.rs, data/strip_metadata.rs, routes/api/strip_metadata.rs, actions/strip_metadata.rs none (not upstream) removing metadata from files on upload, one switch per account (section 9, item 47)

Reference sources (read-only, never executed) of the exact library versions upstream pins are unpacked in _scratch/srv-skeleton/src/: better-auth 1.6.13, @better-auth/core 1.6.13, better-call 1.3.5, @better-auth/utils 0.4.1, drizzle-orm 0.45.2 (integrity checked against pnpm-lock.yaml).

3. Shared vocabulary

  • Models (db/models.rs): one struct per table, field names = column names, serde camelCase like drizzle's $inferSelect. Each implements Columns (TABLE, COLUMNS in database order), so queries read format!("SELECT {} FROM {} WHERE ...", User::COLUMNS, User::TABLE).
  • Timestamps are JsDate everywhere: decodes timestamp and timestamptz as UTC (drizzle stores UTC wall time in the Better Auth timestamp columns), binds as timestamptz (every connection runs with TimeZone=UTC), serialises as toISOString() (...000Z) in JSON and as {"$date"} in RPC/page payloads. JsDate::now() has JavaScript's millisecond precision.
  • Tagged JSON (rpc/tagged.rs): build payloads with typed structs and rpc::ok(&value) / PageData::new(&props, meta), or with modlight_server::tagged_json!({...}). Plain serde_json::json! serialises its interpolated values immediately, so a JsDate inside it becomes a plain string instead of {"$date"}.
  • Errors: route handlers return AppResult<Response>; AppError::message gives {"message": ...} with a status, AppError::Internal (any anyhow/sqlx error via ?) logs and answers 500 {"message":"Internal server error"} like server.ts fail().
  • Sessions: RequestSession (one lookup per request) with the refresh policy of the context the upstream code ran in; see section 8.

4. Adding a route handler (app/**/route.ts)

  1. Create src/routes/<path>.rs named after the upstream path (app/api/uploads/route.ts -> src/routes/api/uploads.rs, app/[userCode]/[mediaCode]/[action]/route.ts -> src/routes/media_action.rs).
  2. Expose pub fn routes() -> Router<AppState> registering the upstream path in axum syntax with the upstream folder names as parameter names: /{userCode}/{mediaCode}/{action}, /api/auth/{*all} (axum keeps one name per position, and the page table already uses these names).
  3. Add one line to http::router() in src/http.rs (the central list).

Until step 3, the URL answers 501 naming the upstream file (the placeholder in pages::fallback), so unported handlers are visible. What the edge already does for every handler: the next.config.ts headers (config values win over the same header set by a handler, exactly like Next), OPTIONS -> 204 with Allow, an unsupported method -> 405 without a body, HEAD from GET, panics -> 500, cookies pushed into ResponseCookies written as Set-Cookie.

Tools for the handler body:

use crate::api_json::{json_private, read_json_object, supplied, JSON_BODY_MAX};
use crate::api_caller::resolve_api_owner;
use crate::auth_helpers::{MaybeUser, RequireUser, Sessions};

async fn patch(State(state): State<AppState>, Sessions(session): Sessions, req: Request) -> AppResult<Response> {
    let (parts, body) = req.into_parts();
    let (owner, scope) = resolve_api_owner(&state, &session, &parts.headers, TokenCapability::Write)
        .await?
        .or_respond(Some(PRIVATE_NO_STORE))?;          // 401/403 refusals, upstream wording
    let input = read_json_object(&parts.headers, body, JSON_BODY_MAX).await?; // 415/413/400 like upstream
    ...
    Ok(json_private(StatusCode::OK, &result))
}

Session extractors (auth_helpers.rs) use the route-handler policy (refresh the row AND re-issue the cookie): Sessions (the lazy RequestSession), MaybeUser (await getSession()), RequireUser / RequireAdmin (307 to /login / /dashboard, as redirect() in a route handler). Every surface that accepts an API token names its capability (api_token::authenticate_api_token, api_caller::resolve_api_caller/resolve_api_owner); keep the inventory of tests/unit/token-scope-coverage.test.ts in mind.

The raw upload paths of server.ts (/api/files*, POST /upload, POST /api/upload) bypass all of the above: replace the placeholder in http::raw_upload_router with the tus and ShareX routers. They see the request before normalisation, with no body limit and no timeouts.

5. Adding a server action (app/actions/<module>.ts)

  1. Create src/actions/<module>.rs (shortlink-unlock.ts -> shortlink_unlock.rs).
  2. Write each export as async fn name(ctx: ActionCtx) -> ActionResult and register it under the upstream names: reg.register("shortlink-unlock", "unlockShortlink", unlock_shortlink);
  3. Add <module>::register(reg); to actions::register_all.

ActionCtx: args.get::<T>(i) (missing/undefined read as null, so Option<T> is None; args.raw(i) and args.is_undefined(i) for exact checks; args.num(i) for NaN/Infinity), form (for X-Modlight-Rpc: form calls), require_user() / require_admin() / get_session(), set_cookie(SetCookie) / delete_cookie(name) / request_cookies(), revalidate_path(..), client_ip(), locale(), state. Return rpc::ok(&value), rpc::void() (resolves to undefined), Err(ActionError::Redirect(url)), Err(ActionError::NotFound) or Err(ActionError::thrown("message")); ? on anyhow/sqlx errors is a throw. Envelopes, origin check, body limit and cookies are the framework's job (rpc/mod.rs); tests/rpc_framework.rs has a complete example module. The upstream unit tests of an action port best against a small *Deps trait the way actions/tokens.rs does it.

6. Adding a page loader (app/**/page.tsx, layout.tsx)

  1. Create src/pages/<name>.rs (app/(app)/dashboard/folders/page.tsx -> src/pages/dashboard_folders.rs).
  2. Write pub async fn load(ctx: PageCtx) -> PageResult returning PageData::new(&props, meta): props is exactly the object the upstream server component passed to its client component(s), meta what its metadata/generateMetadata returned (pages/meta.rs: PageMeta::titled, Title::Absolute, Field::Clear for null, Open Graph, Twitter, alternates, .noindex(), raw extra tags).
  3. Register it in pages::register_all by its app path from pages/table.rs: reg.page("/dashboard/folders", dashboard_folders::load); Layout loaders the same way: reg.layout(Layout::App, layout_app::load);

PageCtx: param("userCode"), search.get("next") / search.to_value() (Next's searchParams object), require_user() / require_admin() / get_session(), cookies(), locale(), t("namespace").t("key") / t_with("key", &[("count", 3.into())]) (ICU like next-intl), client_ip(), mode (Document or Navigation), state. End with Err(PageError::redirect(url)), Err(PageError::NotFound) or ? on any error.

Layouts run first, outside in (root, then (app) or (auth)); a layout that redirects stops the page. A page or layout without a loader answers 501 with the upstream file name. The response shapes are in pages/mod.rs (module docs). A page URL loaded directly returns the SPA shell (web/dist/index.html, or a placeholder until the SPA is built) with the payload inlined; the SPA fetches GET /_page?path=<encodeURIComponent(location.pathname)>&<the page's own query> on client-side navigation and may send X-Modlight-Messages: <locale>:<messagesHash> to skip the dictionaries.

7. Adding a background job

Write the work in the module that owns it (upstream path), then add one entry to jobs::jobs() with its first-run delay and interval. The three upstream timers are already scheduled: tus sweep (boot, then hourly), trash purge (60 s, then every 6 hours), mail settings warm-up (once at boot); their work functions are stubs to fill in: upload::tus::sweep_tus_uploads, maintenance::trash::purge_expired_trash, mail_settings::warm_mail_settings.

8. Sessions (Better Auth compatibility)

auth/session.rs is Better Auth 1.6.13's getSession with lib/auth.ts's options; verified against the reference server (section 10). Cookie better-auth.session_token (__Secure- prefix when APP_URL is https), value encodeURIComponent(token + "." + base64(HMAC-SHA256(AUTH_SECRET, token))), 7-day expiry, refreshed when older than a day. Where a refresh may happen depends on the Next context upstream ran in, so each caller picks a RefreshPolicy:

Context Policy Upstream reason
route handler, server action, /api/auth/get-session DbAndCookie nextCookies() copies the cookie into the response
page loaded as a document DbOnly a Server Component cannot set cookies (the write is swallowed)
/_page (client navigation) Skip RSC: 1 requests skip the refresh
raw upload paths DbOnly no request scope for cookies()

A session for a banned account is still returned (Better Auth deletes the sessions when banning and refuses to create new ones: check_ban_before_session_create); resolve_api_owner refuses banned owners itself, as upstream. AuthSession::output() serialises exactly like auth.api.getSession() (same fields, same order). Accounts created through Better Auth get theme = "modern" written explicitly (its additionalFields default), not the column default.

9. Deviations from upstream and from PORT.md

Deliberate, each for a reason:

  1. Malformed percent-encoding in a path answers 400 (upstream: 500 text/plain), per "malformed requests answered 400".

  2. Thrown action errors: in production the envelope carries React's generic production message plus a digest, because that is what the upstream client saw (Next strips the message); with NODE_ENV unset the real message is sent. PORT.md's {"error":{"message":"<message>"}} shape is kept.

  3. /_page envelopes added beyond PORT.md: {"reload": true} (the URL is a route handler or a static file: load it as a document), 501 {"notImplemented": true, "route", "file"}, 500 {"error": ...}; a notFound envelope also carries layouts.root and meta; the root layout includes messagesHash, and messages is null when the SPA already has them (X-Modlight-Messages).

  4. Shortener host pass-through also covers /_page, /_rpc and /assets/ (the SPA's surface, like /_next upstream). A short code named exactly assets/... on the shortener host is the one thing shadowed.

  5. RPC origin check compares the Origin host with Host (X-Forwarded-Host only when TRUST_PROXY is on), like Next's server-action check, which survives TLS termination; X-Modlight-Rpc must be 1 or form; unknown action 404; bodies over 1 MB (Next's default server-action limit, never raised upstream) 413.

  6. Session reads are memoised per request (upstream re-queried; same answer, fewer queries).

  7. Cookie serialisation: cookies set in route handlers and actions use Next's format (Path, Expires, Max-Age, ..., SameSite=lax); Better Auth's own endpoints should use SetCookie::to_better_call_string. Same semantics; attribute order differs between the two exactly as upstream.

  8. Security fix 12.7 is in password::client_ip: X-Real-IP only with TRUST_PROXY on, else the socket peer address.

  9. Head tags: icons are linked without Next's ?icon.<hash> cache-buster; the not-found robots tag comes after the title; Next's framework tags (next-size-adjust, preload links) are gone. Link-preview relevant tags (title, description, Open Graph, Twitter, canonical, oEmbed alternates) are rendered in Next's order; diff them against the reference once the media pages are ported.

  10. Static files are served without ETag (with Last-Modified), with the upstream Cache-Control values.

  11. HTTP: HTTP/1.1 only (as Node), no X-Powered-By, no Next Vary: rsc, no Keep-Alive: timeout=5 header. Response compression is Next's (item 42).

  12. Jobs never overlap a slow previous run (upstream setInterval could).

  13. serve validates the configuration before migrating (the upstream container migrated first and refused to start afterwards).

  14. S3 transport: the Rust SDK gives up connecting after 3.1 s and fails a transfer the server stalls (no bytes for 5 s while it is being read; a reader that pauses is not a stall), where Node would wait. stat still swallows every error like upstream, so /api/health reports storage: ok on an S3 install whose endpoint is down, as upstream did. The runtime image needs a CA bundle for HTTPS endpoints (rustls reads the system store).

  15. pastes.createPaste languages: LANG_EXT[lang] upstream also finds what every JavaScript object inherits, so a language of constructor or __proto__ (the lower-cased names among them) took a function or an object as the extension, wrote the bytes under <code>.function Object() { [native code] } / <code>.[object Object] and then threw (verified on a private reference). Here only the 28 listed languages map to an extension; any other name is txt, and the paste is created.

  16. uploads.emptyTrash with a storage driver that cannot be built (an S3 install missing a variable) throws even for an empty trash; upstream's getStorage() only ran once a batch had rows. Either way the install is broken; the answer differs only for an empty trash.

  17. ShareX (security fix 2): the header token is authenticated before a byte of the body is read; a request that names no header token may still carry a legacy token field, but its body is capped at 8 MiB (LEGACY_UPLOAD_MAX_SIZE overrides; 413 with the upstream wording); quota and free space are re-checked every 8 MiB while a chunked body streams (507 "User disk quota exceeded." / the low-disk message), instead of after the whole file landed on disk.

  18. tus wire details: the status line carries a reason phrase, empty bodies go out with Content-Length: 0 (Node chunked them), and there is no Connection: keep-alive header. Kept from upstream: an upload finished at creation (zero length, or creation-with-upload of the whole body) is finalized, and its URL answers 404 afterwards.

  19. DELETE /api/uploads reads at most 1 MiB of body (upstream: req.json() without a limit); 500 ids need about 20 KB.

  20. libvips: only the loaders sharp-libvips 1.2.4 ships may run (Homebrew's ImageMagick loader would accept BMP and the avis AVIF production refuses); HEVC pixels are refused (HEVC_DECODE = false) as production cannot decode them. Built against libvips 8.18, image metadata, thumbnails and avatars are byte-identical to production's 8.17.3 except CMYK input (mean 3/255 colour shift); on Debian 13's 8.16.1, 2 of 20 blurhashes differ by a digit. The runtime image should carry libvips 8.17 or newer.

  21. ffprobe: durations come from whatever ffprobe the host has. The reference image (Debian 12, ffmpeg 5.1) counts an MP3's encoder padding (1.541224 s) where ffmpeg 9 does not (1.5 s); ship the reference's ffmpeg major for identical numbers.

  22. GET /api/link-info sends the handler's own Vary: Authorization, Cookie (Next replaces it with its rsc, ... list on every route handler; see 11).

  23. Kept on purpose (upstream behaviour worth knowing): listFolders always reports count: 0 (the subquery compares uploads with itself). (This item also listed the two behaviours items 38 and 39 now fix.)

  24. A thrown error in a route handler answers like Next: 500, empty body, no Content-Type (error::route_handler_error_response; seen on the reference on /api/duplicates with an out-of-range count and on /s/%00). Only server.ts' raw upload paths answer its fail() JSON {"message":"Internal server error"} (also for a panic there). Decided at integration (2026-10-20); before, every 500 was the JSON one.

  25. read_json_object parses with js_json::parse (V8's JSON.parse: no nesting limit, lone surrogate escapes, 1e400 = Infinity), as upstream readJsonObject (decided at integration with the reference's answers: maxViews: 1e400 -> 400 "View limit must be a number", expiresInDays: 1e400 -> a link without expiry, a field nested 3000 levels deep ignored like any unknown field). The routes still read serde_json values (api_json::js_object_to_map): a non-finite number becomes the RPC's tagged {"$num":"Infinity"} (api_json::is_js_number, rpc::tagged::raw_num read it), so a client that sends that literal object as a field value is read as Infinity; containers deeper than 64 levels are kept empty (no route reads that deep; serde_json's drop is recursive). Lone surrogates become U+FFFD as before.

  26. Security fix 1, view limits (media::serve::ViewBudget, the exact rule is in its doc comment). For a file with max_views, and anybody but its owner or an administrator: a request upstream counted is counted (unchanged); a request upstream exempted (a Range not starting at 0, Sec-Fetch-Dest: image|video, w/h/motion, a bot) is served without consuming only with the VIEW GRANT of a counted view, else it consumes one. The grant is a signed cookie ml_view_<uploadId> = <exp>.<HMAC(AUTH_SECRET, "view|<id>|<exp>")> (6 hours, httpOnly, SameSite=Lax, Secure in production), minted by every counted view of a capped file: the preview page's (document and /_page), and /raw//download's. It also lets its holder through when that view was the LAST one (views == max_views), which repairs two upstream failures: the page's own <img>/<video> on the last permitted view, and an encrypted one-time file whose second ranged read 404ed (verified on the reference). It applies to everything that carries the file's content: the stored bytes (whole or ranged) and pictures made of them (resized stills, moving tiles, timeline sheets); refusals and the cue track spend nothing. Also: a preview bot on the page of a capped file is not counted (as upstream) and gets no content (no inlined text, no render ticket), the card of a capped text file carries no excerpt, and a capped item travels in an album ZIP only by spending a view (as a download). Costs: a cross-host page (owner's preferred domain different from the host the page was opened on) cannot carry the cookie, so its <img> pays a view; public gallery tiles of a capped file pay a view each; resuming a capped download with Range pays a view.

  27. Security fix 5: the delete-by-link token is compared in constant time.

  28. Security fix 6: /oembed describes a published file with a view cap as {version, title, provider_name, provider_url, cache_age, type: "link"}: no raw URL, no dimensions, no player HTML (as embed.ts withholds og:image/og:video).

  29. Album ZIP: the reference answers every /a/<tag>/<slug>/zip with 500 (import archiver from "archiver" against archiver 8, which has only named exports: TypeError: (0, a.default) is not a function). The port serves the archive the route was written for, in archiver's zip layout for stream entries (src/archiver.rs); deflate bytes come from flate2, not Node's zlib, so archives are equivalent (same entries, order, names, flags, attributes, CRCs and contents), not byte-identical.

  30. Metadata placement: Next 16 streams a slow generateMetadata into the BODY for browsers (it blocks the head only for the bots it recognises); the port always puts it in <head>. Same tags, same order.

  31. Plain-object lookups: RENDERABLE, EXT_CATEGORY and LONG_TYPES match only their listed keys (upstream found constructor and friends on the prototype, so a stored type constructor was offered the render toggle and an extension constructor a card URL spelled from a function).

  32. Derived responses (thumbnails, motion tiles) carry Content-Length; Next sent them chunked. Same bytes.

  33. ffmpeg version (media processing): the argument vectors are upstream's and identical to the reference's (verified, tests/parity_media), but they were written for ffmpeg 5.1 (Debian 12, the production image). On ffmpeg 7+ (Homebrew's 9 on the Mac): -fflags +shortest is no longer a format flag, so audio -> MP4 fails; the mjpeg encoder refuses the limited-range YUV the timeline sheet hands it, so timeline.jpg fails; a clip offset past the end (?t=9999) fails writing the GIF trailer where 5.1 wrote a one-frame GIF; ffprobe measures VFR files, AVIF sequences and MP3 padding differently, so a few plans differ (item 21). Homebrew's build also has no libvorbis and no libwebp (ogg, and ffmpeg's animated WebP of an AVIF sequence, fail). Run the server with the reference's ffmpeg major (the Docker image on Debian 12) for identical behaviour; with it, every status, header, argument vector and deterministic output matched.

  34. Text on the embed card: libvips' Pango is put on its fontconfig backend (PANGOCAIRO_BACKEND=fc, set once at libvips start unless the operator set it). It is the only backend Linux has; on macOS Pango would otherwise pick CoreText, which never sees the fontfile and draws a system face. With it every card is pixel-identical to the reference.

  35. PDF output (media/pdf_lib.rs): pdf-lib's file structure object for object; the Flate streams are compressed by miniz rather than pako (they inflate to the same bytes). Like pdf-lib, the info dates and the XObject name (Image- + random) change on every call, so no two PDFs are equal byte for byte on either server.

  36. HEVC_DECODE = false, decided exactly as upstream decides it: by the libheif sharp-libvips 1.2.4 ships (1.20.2 with aom only, no libde265, checked in the reference container: metadata() reads a HEIC's header as compression: hevc, toBuffer() fails with "Support for this compression format has not been built in"). So HEIC gets no blurhash, no thumbnail of its own, no conversion target (convert-targets offers none) and no motion tile (not-animated), on both servers.

  37. Lookups by a caller's string in media/source_preview.rs (paste language -> extension) use exact keys, as item 31 does for the card table.

  38. Security fix: the tus URL of a finished upload (integration, 2026-10-20). On S3 a finished tus upload IS the stored object (finalize adopts the key in place) and its <id>.info record stayed, so upstream's GET /api/files/<id> streamed the file to anybody holding the tus URL whatever its later privacy, password or view limit, and DELETE (tus termination, no authentication upstream) deleted the object from under its media row. Now GET never serves a finished upload (404, as an unknown id; the option serve_finished_uploads), DELETE of a finished upload answers 400 "Cannot terminate an already completed upload" (@tus/server's own disableTerminationForFinishedUploads), and once the upload is finalized on S3 its tus record (.info, .part) is deleted, so the URL answers 404 to HEAD/PATCH/DELETE exactly as on the local driver. Unfinished uploads resume as before. Tests: upload::tus_server::server finished_uploads_are_not_served_or_terminated, tests/library_s3_e2e.rs (MinIO).

  39. Replacing or removing an avatar or a wallpaper deletes the previous object (integration, 2026-10-20). Upstream's UPDATE ... RETURNING read the NEW key, so discard(old, key) never deleted anything and every replacement left an object behind for good. The previous key is now read under the row lock of the same statement (update ... from (select ... for update) prev returning prev.<column>), and its object is deleted after the update; only this account's own keys (avatars/<userId>-..., backgrounds/<userId>-...) are ever deleted, because user.image can also be written through Better Auth's update-user. Test: tests/library_routes_db.rs replacing_avatar_or_wallpaper_deletes_the_previous_object.

  40. PORT.md section 12, fixes 1 to 7, re-checked in the integrated build (2026-10-20), each with a test that fails on upstream's behaviour: 1 view limits: items 26, tests/media_serving_db.rs fix1_every_bypass_of_the_audit_is_closed, fix1_the_legitimate_flows_ride_on_the_view_grant, fix1_preview_bots_get_no_content_of_a_capped_file (+ parity media_serve.py --only fix1); 2 ShareX: item 17, tests/security_fixes_db.rs fix2_sharex_authenticates_before_the_body_and_caps_what_streams (an unknown header token is refused with the body unread; no token: 413 after 8 MiB; a chunked body over the quota: 507 while streaming, nothing kept); 3 first administrator: auth::adapter (pg_advisory_xact_lock), tests/security_fixes_db.rs fix3_concurrent_first_sign_ups_make_exactly_one_admin (12 concurrent sign-ups on an empty install, one admin); 4 POST /api/releases/app: tests/releases.rs publishing_authenticates_before_it_reads_the_body and the 415 / 413 checks in release_channel_against_a_database; 5 constant-time delete token: item 27, tests/media_serving_db.rs fix5_delete_by_token; 6 /oembed of a capped file: item 28, tests/media_serving_db.rs fix6_oembed_describes_a_capped_file_without_its_bytes; 7 X-Real-IP only behind TRUST_PROXY: item 8, password::tests::honours_x_real_ip_only_behind_a_trusted_proxy, auth::ip::tests::real_ip_only_behind_a_trusted_proxy.

  41. Document redirects and the dashboard boundary (integration, 2026-10-20, decided with the reference's answers, CONTRACT 12.4). A page's or layout's redirect() answers 307 + Location with a whole HTML document as the body, as Next does (Next sends its __next_error__ shell; here the SPA shell with the redirect payload inlined, which the SPA follows if a client ever shows it). Under app/(app)/dashboard/, whose loading.tsx makes Next flush the page before it runs, a page's own redirect() or notFound() answers 200 like the reference: the page's own title (Admin - Modlight, User files - Modlight, else the app name), a notFound() with robots: noindex, a redirect with Next's <meta id="__next-page-redirect" http-equiv="refresh" content="1;url=..."> inside <noscript> (the SPA follows the inlined redirect payload itself; a live refresh would load the target a second time a second later). Layout redirects (the (app) layout's requireUser()) stay 307, as on the reference. The proxy gate and the trailing-slash 308 keep their text bodies (already as upstream). Malformed percent-escapes keep item 1 (400).

  42. Response compression (integration, 2026-10-20): Next's compress: true (compression 1.7.4), checked response by response on the reference: documents, /_page and /_rpc (the RSC payloads' place) and static files (/assets/*, public/, from 1 KiB) are gzip- or deflate-encoded for a client that accepts it, with Vary: Accept-Encoding; route handlers (/api/*, media, /oembed, /embed, /icon.svg), the raw upload paths and every 206 never are (http::compression). Bytes differ from Node's zlib (same content).

  43. Intended difference: optional Cloudflare Turnstile (2026-10-21, a new feature the owner asked for on top of the 1:1 port; src/turnstile.rs, src/actions/turnstile.rs, the SPA's src/turnstile/). With no complete configuration nothing changes: no check, no outbound call, the CSP and every payload as before except layouts.root.config.turnstileSiteKey: null, the turnstile dictionary in layouts.root.messages and the admin page's turnstile prop.

    • Configuration: the admin panel's "Cloudflare Turnstile" card (Admin -> Settings: switch, site key, write-only secret key, "Test") stores turnstile_enabled ("on"/"off"), turnstile_site_key and turnstile_secret_enc (secret box, purpose turnstile). Env fallback TURNSTILE_SITE_KEY + TURNSTILE_SECRET_KEY (both, trimmed; one alone warns once and stays off). Block precedence as for SMTP: once the switch row holds a value the panel decides everything (an environment secret never fills a panel block), "use the environment" sets the three rows to NULL. On = switch on + site key + a secret that decrypts. A 30 s snapshot per process (AppState::turnstile), re-read by the root layout and the admin page, warmed at boot; a database failure keeps the last known configuration (or the environment's) for one period.
    • Where: POST /api/auth/sign-in/email, /sign-up/email, /request-password-reset (any POST whose path below /api/auth contains one of them, better-auth's captcha plugin's defaultEndpoints and includes match), in auth::api::handle after the rate limiter and before the body is read; and the actions login-identifier.signInWithIdentifier (after its own throttle) and invite.acceptInvitation (after its throttle, before the invitation is read). Server-side calls (auth::server::call_endpoint) are not checked, so the sign-in those actions perform does not spend a second siteverify on a single-use token. Not: 2FA, reset-password itself, verification links, API tokens.
    • Token and refusals: header x-captcha-response (the plugin's), so no action signature or body changes. Better Auth routes answer the plugin's codes as better-call APIErrors: missing 400 MISSING_RESPONSE, refused 403 VERIFICATION_FAILED; Cloudflare unreachable 503 SERVICE_UNAVAILABLE "CAPTCHA service unavailable" (the plugin answers 500 UNKNOWN_ERROR "Something went wrong" and logs that code). Actions answer in their own shape: {status: "captcha" | "captcha-unavailable"}, {ok: false, reason: "captcha" | "captcha-unavailable"}. The SPA shows turnstile.widget.* texts (en, pl) for each.
    • Verification: POST https://challenges.cloudflare.com/turnstile/v0/siteverify, form-encoded secret, response, remoteip (the plugin posts JSON; Cloudflare takes both). remoteip = password::client_ip (the action throttles' rule: X-Real-IP only behind TRUST_PROXY, else the socket peer) when it is an IP, sent unnormalised (Better Auth's limiter key cuts IPv6 to /64, which is no address). hyper-util client with hyper-rustls (ring, Mozilla roots like the mail transport), 10 s for the whole exchange, answer capped at 64 KiB; only HTTP 200 with "success": true passes; anything else unanswered is "unavailable" and refuses (fails closed). A token over 2048 characters is refused without a call, an empty one is "missing" without a call. Logged: Cloudflare's error codes and transport errors, never the secret or the token.
    • Order: the plugin runs before the rate limiter; here after it, so a flood of bad tokens is throttled before it reaches Cloudflare, and a refusal is recorded against the bucket like any other answer.
    • Saving: a save that turns the check on first sends the secret with Cloudflare's dummy token (XXXX.DUMMY.TOKEN.XXXX): invalid-input-secret / missing-input-secret refuses the save (secret-rejected), no answer refuses it (unreachable), because a wrong secret would refuse every sign-in, the administrator's included. A changed site key forgets the stored secret. The "Test" button does the same probe (also reporting a test secret that accepts everyone); both share a throttle of 10 per 10 minutes per administrator. Audit entries turnstile.settings_changed (row names, secretChanged) and turnstile.settings_cleared.
    • CSP: while on, appCsp and the /authorize policy get https://challenges.cloudflare.com appended to script-src and frame-src (http/headers.rs, both variants built at startup, the snapshot picks one per response), on every page, because the SPA reaches /login from any page without a new document. connect-src is not needed (the widget talks to Cloudflare from its own frame; checked in Chrome with the test keys: no violation).
    • SPA and i18n: the four auth forms became ports in web/src/overrides/components/auth/ (CONTRACT.md 11); the strings are messages/<locale>/turnstile.json, an 18th client namespace (i18n::NAMESPACES), so upstream's files stay verbatim.
    • Locked out (a widget whose host names do not match, a secret revoked at Cloudflare): UPDATE settings SET value = 'off' WHERE key = 'turnstile_enabled'; takes effect within 30 s (or restart); setting the three turnstile_* rows to NULL hands control back to the environment.
    • Tests: turnstile::tests (precedence, codes, CSP in http::headers), tests/turnstile.rs (a siteverify stub on 127.0.0.1: pass, refused, missing, oversized, network error, timeout, off; every entry point), tests/turnstile_db.rs (the card, the secret box, the audit, block precedence, real sign-ups and sign-ins), web/tests/turnstile.test.tsx.
  44. Intended addition: desktop sign-in pickup (2026-10-24). The loopback handoff of RFC 8252 depends on the BROWSER posting the token from an https page to http://127.0.0.1:<port>/, and browsers refuse: Safari blocks it (mixed content / loopback), Chrome's Local Network Access drops it once its permission was dismissed. The app's listener is fine; the bytes never come. So the app also collects the outcome from the server (PKCE, RFC 7636, cut down), and whichever answer arrives first wins.

    • Flow: the app makes verifier (32 random bytes, base64url) and sends challenge = base64url(SHA-256(verifier)) (43 characters) in /authorize?port=&name=&state=&challenge=. parse_authorize_request_with_challenge validates it (malformed -> invalid-challenge, an invalid request), authorize_path carries it through /login?next=. The page loader registers {state -> challenge, user id} as pending (the only thing it writes; register_render, rate-limited per account and per address); approveDesktopClient with a challenge stores the token, the new denyDesktopClient stores access_denied (actions/authorize.rs), both ONLY on a live entry that the same account registered with the same challenge (security review 2026-10-24, P1: the challenge is in the URL, so it proves nothing about who decides). Otherwise nothing is stored and the approval answers pickup: false: the page then uses the loopback POST and the manual key, which reach only the approver's own machine. The app polls POST /api/authorize/pickup {state, verifier} every 2 s.
    • Confirmation code (review 2026-10-24, P2): confirmation_code(challenge) = first 6 base32 characters of SHA-256("modlight-confirm:" || challenge), shown as XXX-XXX by the app while it waits and by the prompt next to Approve ("Approve only if this code matches the one in your Modlight app"); the page props carry it as confirmCode. A phishing link carries somebody else's challenge, so its code is one no app of the victim's shows. The app computes the same (modlight-api signin::confirmation_code, same test vector 554-LIO).
    • Store (authorize/pickup.rs): in memory, state -> {challenge, user id, outcome, expires}, 5 minutes, at most 4096 entries (swept every 30 s and when full). The first account and challenge seen for a state bind it; another account or challenge never replaces a live entry. One account holds at most 5 live entries (its own oldest pending one makes room); when the map is full the oldest PENDING entry goes, and a decided entry (a token waiting for its app) is never evicted before it expires (review P3). Renders: 30 per account, 120 per address per 10 minutes. Handed out once, then deleted. A restart loses pending pickups (loopback and manual key remain).
    • Route (routes/api/authorize_pickup.rs): read_json_object (1 KiB), constant-time compare of the hashed verifier; 200 {"token"} or 200 {"error":"access_denied"} once, 202 {"status":"pending"}, 404 {"error":"not_found"} for unknown, expired and mismatched alike, 400 {"error":"invalid_request"}, 429 {"error":"rate_limited"} + Retry-After (120 per minute per client IP). Cache-Control: no-store. No session or API key: the verifier is the credential. Tokens and verifiers are never logged.
    • SPA: components/authorize/authorize-prompt.tsx is a port (web/src/overrides/), calling web/src/rpc/authorize-pickup.ts; with a challenge the page shows the confirmation code next to Approve, says the key will be handed to the Modlight app that shows that code (wherever it runs), after approving says the app picks the key up by itself, still tries the loopback POST, and keeps the key under "App still not signed in?" to copy; with pickup: false it falls back to upstream's flow. Strings: messages/<locale>/authorizePickup.json, a 19th client namespace. Without a challenge (older apps) everything is as before.
    • Tests: authorize::pickup::tests, routes::api::authorize_pickup::tests, actions::authorize::tests (challenge, deny), authorize::loopback (challenge parsing and the login round trip), web/tests/authorize-pickup.test.tsx, app tests/signin_pickup_fake.rs and the opt-in real-server tests/signin_pickup.rs (MODLIGHT_PICKUP_E2E); tests/secreview_pickup.rs (the review's attacks, asserting that each fails).
  45. Intended addition: Downloads (PL "Do pobrania", 2026-10-27). The server keeps the desktop app's builds for macOS and Windows; signed-in accounts download them, build machines publish them with a token, the desktop app reads the list for updates. Nothing upstream changes: new routes, a new table, a new page.

    • Table releases (migration 0015, drizzle journal and snapshot like the others): id text pk (32 hex), platform (macos|windows, CHECK), arch (arm64|x64, CHECK), version, build (may be empty), built, published (timestamptz), size bigint, sha256, filename, uploader, note; index (platform, arch, published). Model db::models::Release.
    • Files: through the storage layer at releases/<id>/<filename> (local and S3 alike: ingest_local_file, read_stream with ranges, delete_prefix). The storage scan of the admin page skips releases/ like derived/, avatars/ and backgrounds/. Chunks of an unfinished upload are local files under UPLOAD_TMP_PATH/releases/<upload>/ (upload.json, chunk-<n>); finish joins them in order while hashing, checks the declared size and SHA-256 (else 400 and the folder is removed), hands the joined file to the driver, inserts the row and drops every build of the same platform and architecture beyond the newest 5 in one transaction (advisory lock per platform+arch), then deletes their files. A finished upload keeps a done marker for a day so a repeated finish answers the same build; an upload untouched for a day is swept at the next start. One finish per upload at a time (409 busy). published is monotonic within the process.
    • Routes (routes/api/downloads.rs; errors JSON {"error","code"}, Cache-Control: private, no-store): GET /api/downloads and GET /downloads/{id}/{filename} for any signed-in account, by session or by an API token of any scope (capability identify: the list reveals nothing about an account, and the desktop app must find its updates whatever its token's scope); the file with Content-Length, Content-Disposition: attachment, ETag = the quoted SHA-256, If-None-Match (304), Range/If-Range (206). POST /api/downloads/upload (read_json_object rules, 16 KiB), PUT .../upload/{upload}/{n} (chunks of at most 50 MiB, 413 beyond, or beyond the declared size), POST .../upload/{upload}/finish for a publisher: RELEASE_PUBLISH_TOKEN as a bearer token (x-api-key too; SHA-256 of both sides compared in constant time, config::PublishToken keeps only the digest), an administrator's session, or an administrator's full API token (capability admin, as /api/releases/app). DELETE /api/downloads/{id} for administrators only. The publish token opens nothing else (401 elsewhere). The credential is checked before any body is read; a token decides alone when one is offered (a wrong one is never rescued by a cookie). 401 unauthorized/account_disabled, 403 forbidden/insufficient_scope/invalid_scope.
    • Shadowing: /downloads/{id}/{filename} has the shape of upstream's /{userCode}/{mediaCode}/{action}; a path that names no build (wrong id or file name) is handed to the media action unchanged, so an account tagged downloads from before keeps its links. The tags downloads and releases are added to RESERVED_TAGS (a new account cannot take them; releases because user files live under <tag>/ in storage).
    • Audit (audit::record_audit_by, a NULL user_id for the token): downloads.published, downloads.pruned, downloads.deleted, downloads.rejected, with by/byEmail/via and the build.
    • Page /dashboard/downloads (pages/dashboard_downloads.rs, a 28th page in pages/table.rs; CONTRACT.md 6.28): props are the GET /api/downloads answer. The SPA (web/src/downloads/) shows a card per platform, newest build on top, older folded; administrators also delete and see how to publish (with window.location.origin). Strings: messages/<locale>/downloads.json, a 20th client namespace, and nav.downloads.
    • Configuration: RELEASE_PUBLISH_TOKEN (optional; setup-local.sh generates it; a value under 24 characters warns at start).
    • Tests: downloads::tests, routes::api::downloads::tests, config::tests (publish_token_*), tests/downloads.rs (auth rules, chunked publish with a full 50 MiB chunk, wrong SHA-256, wrong size, missing chunk, keep-5, delete, ranges, the page, the media-action fallback), web/tests/downloads.test.tsx.
  46. Intended addition: per-account file encryption (2026-10-27, the owner's request; the design, the threat model, the API for bearer clients and the limits are in docs/ENCRYPTION.md). An account without a vault sees nothing new: no query beyond a subselect in lookup_media, no cookie, no header, the same payloads (the new props are absent), except the Settings page's encryption prop and the empty uploads.encryption column.

    • Code: src/vault/ (cipher, keys, store, access, view, write, gate, unlock, status, pass), src/actions/encryption.rs (RPC module encryption), src/routes/api/encryption.rs (/api/encryption, /unlock, /lock), migration 0016 (uploads.encryption, file_vaults, vault_grants, vault_cleanup), the SPA's src/encryption/.
    • Where it hooks in: the writers (upload::finalize, pastes.createPaste, media::convert_in_place) encrypt with the public key; the readers get a decrypting storage view only behind the gate (vault::access::open_files in routes/media_action.rs and its render, pages/media.rs, routes/album_zip.rs, actions/avatar.rs, convert_in_place); the galleries (pages/public_{profile,album}.rs); lookup_media marks every file of a gated account (Upload::vault_gated), which makes cache_headers_for private, may_cache_derived false, and the embed plan, /oembed and /api/link-info treat it as password-protected; http.rs lets the unlock form keep its own CSP on a raw URL (vault::gate::GateDocumentCsp); jobs::start runs vault::pass::worker.
    • Answers: locked content is 401 for a browser (an HTML form for a document, text otherwise) and 423 Locked JSON {"error": "files_locked"} for a bearer client (401 would sign the desktop app out).
    • Tests: vault::*::tests (crypto round trips, Range across chunk borders, tampering, wrong passwords, grants, gate escaping, rate-limit buckets), tests/vault_db.rs (bearer uploads, every public surface, grants, the pass with an interruption and a restart, rate limits), web/tests/encryption.test.tsx.
  47. Intended addition: removing metadata on upload (2026-11-07, the owner's request). One switch per account, "Remove metadata from files on upload", off by default: an account that never turns it on sees no change (its uploads are stored byte for byte, and no new query runs: the switch is a column of the owner row finalize already holds).

    • Columns (migration 0017, drizzle journal and snapshot like the others): user.strip_metadata boolean NOT NULL DEFAULT false (the switch) and uploads.metadata_kept varchar(16) (NULL, or why a file uploaded with the switch on kept its metadata). Models User::strip_metadata and Upload::metadata_kept, both #[sqlx(default)] and never serialised. The TypeScript server ignores both.
    • Where it runs: upload::finalize, right after the quota reservation and before anything describes the file: the type sniffing, the dimensions and blurhash, the ffprobe metadata, the checksum, the vault's encryption and the row all see the cleaned copy (media::strip::strip_upload, which works on a copy in a scratch folder of UPLOAD_TMP_PATH, removed with it, and never changes the upload's own file). The reservation is moved to the cleaned size (a larger cleaned copy that does not fit the quota is not used). The cleaned copy replaces the temp file (removed after the commit, as the ingest would have moved it); an S3 tus upload's object is copied out, cleaned, stored under a new key and deleted after the commit, as the vault does with its plaintext. The answers of every upload path keep their shape.
    • Writers: every upload goes through finalize, so the switch covers the panel (tus /api/files, session), the desktop app (tus with its token, and its plain POST /upload), ShareX and scripts (POST /upload, POST /api/upload). It does not apply to the other writers, on purpose: pastes (pastes.createPaste) store text typed in the panel, with no file metadata to remove; convert in place or as a copy stores what libvips (keep: none) or ffmpeg (-map_metadata -1) produced, which carries none of the source's metadata (in place, metadata_kept is cleared); avatars and wallpapers are re-encoded by libvips; Downloads' builds are not user files; the vault's pass rewrites ciphertext of files already stored. Files already uploaded are never touched.
    • Duplicates: a ShareX answer's duplicate_of is looked up after the insert with the cleaned checksum, leaving the new row out (duplicate_lookup::first_match_except); a tus upload, which upstream never hashes, gets the cleaned file's checksum on its row. Cleaning is deterministic (no time stamps, -fflags +bitexact), so the same photo uploaded twice is two copies of the same bytes.
    • What each format loses and keeps (told by the bytes, never by the name; the picture is never decoded where the format allows it):
      • JPEG (segments): EXIF and XMP (APP1), IPTC/Photoshop (APP13), MPF and every other APPn, comments, a JFIF thumbnail, and whatever follows the end of the image (MPF's second pictures, a motion photo's video) go; JFIF APP0, Adobe APP14, the ICC profile chunks, the tables and the entropy-coded data stay byte for byte. A rotated photo keeps an EXIF holding only Orientation.
      • PNG and APNG (chunks): tEXt, zTXt, iTXt, tIME, eXIf, every unknown ancillary chunk and anything after IEND go; IHDR PLTE IDAT IEND tRNS gAMA cHRM sRGB iCCP sBIT pHYs bKGD cICP mDCV cLLI acTL fcTL fdAT stay with their CRCs. An unknown critical chunk makes the file unknown (kept).
      • WebP (RIFF): EXIF, XMP and unknown chunks go, the VP8X flags and the RIFF size are put right; VP8 VP8L VP8X ALPH ANIM ANMF ICCP stay.
      • GIF (blocks): comment, unknown and application extensions other than NETSCAPE2.0/ANIMEXTS1.0 go, and anything after the trailer; frames, delays, colour tables and the loop stay.
      • HEIC and AVIF (HEIF): the EXIF and XMP items are overwritten in place at the same length (EXIF by a minimal block holding only Orientation, or none; XMP by an empty packet; then zeros). No box moves, so the coded picture, irot/imir, colr, depth and gain maps are the same bytes, and no codec is needed (the image's libvips cannot decode HEVC, so a re-encode could never clean an iPhone photo). An item may only be overwritten inside mdat or idat and away from every other item's bytes. An image sequence also loses its movie times and its udta and meta (renamed free, zeroed). An AVIF this cannot follow is saved again by libvips (quality 90, 4:4:4, keep=icc, orientation applied): the only lossy step of the feature, for unusual files.
      • TIFF: libvips, losslessly (the same pixels in their own depth; keep=icc; the Orientation tag carried; the compression kept, or deflate for a lossy one). Camera raws (TIFF outside) are kept as they came (format).
      • Audio and video (MP4, MOV, M4V, 3GP, M4A, MKV, WebM, AVI, MP3, FLAC, Ogg, Opus, WAV): ffmpeg 5.1 in the image, -map 0 -map -0:d? -map_metadata -1 -map_chapters -1 -c copy -fflags +bitexact -strict unofficial, the demuxer forced from the allowlist, +faststart and the major brand kept on the mov family. Every global and stream tag (GPS, dates, device, title, artist) and every chapter goes, and so do data streams (phone telemetry, a GoPro GPS track, an MP4's chapter track); every audio and picture stream is copied packet for packet, cover art included, and the rotation of a phone video (display matrix side data) survives. The result is checked with ffprobe before it is used: the same audio and picture streams, the same rotation, no tag naming a place, a time or a person; otherwise it is thrown away. A bare MPEG frame sync that ffprobe does not read as media is left alone. Other containers (ASF, FLV, MPEG-TS, AAC, APE, WavPack) are left alone.
      • PDF (lopdf 0.47): the trailer's Info dictionary and every Metadata (XMP) stream go, and so do older revisions of an incrementally saved file (unreferenced objects are pruned); pages, fonts and pictures are the same objects. Kept as they came: password-protected files (protected), signed ones (signed: a rewrite would break the signature), files whose cross-reference table had to be rebuilt (damaged). A PDF/A file loses its conformance mark with its XMP. Pictures inside a PDF keep any EXIF they carry.
      • Office Open XML (docx, xlsx, pptx and relatives with a main part) and OpenDocument (odt, ods, odp...): docProps/core.xml and docProps/app.xml become empty property sets, docProps/custom.xml goes with its content type and relationship, meta.xml becomes an empty office:meta; every other entry is copied byte for byte, local header included, so ODF's first stored mimetype stays first. Signed packages and password-protected ODF are kept as they came. Pictures inside keep any EXIF they carry; comments and tracked changes inside the document body are content, not metadata, and stay.
      • Anything else (text, archives, executables, BMP, SVG, PSD...): untouched.
    • Fails open: a file that cannot be cleaned is stored exactly as it came; the upload never fails because of this step. The reason is logged (with the code, not the name) and stored in uploads.metadata_kept: damaged (a broken or unusual layout), format (a camera raw, an AVIF sequence libvips refuses), protected, signed, too-large (over a limit below, a deadline, or a cleaned copy that does not fit the quota), busy (the media queue was full or the disk had no room for a second copy), tool (ffmpeg or libvips missing or failing). The Settings card lists those files (live ones, newest first, 20 of them and the total).
    • Limits: the byte-level rewriters (JPEG, PNG, WebP, GIF, HEIF, the zip of a document) stream file to file in constant memory, whatever the size; ffmpeg works on files (AV_WEIGHT on the media limiter, the 45 s copy deadline, 10 s probes, 2 GiB at most: beyond, too-large at once); libvips (TIFF, the AVIF fallback) and lopdf (64 MiB at most, decompressed object streams bounded at 64 MiB) hold a conversion's slot with a 30 s deadline; an S3 object is staged within 45 s; a zip is only read when the end of its directory says it may be a document. The parsers are fuzzed (media::strip::fuzz: every truncation and 3000 mutations per format must end in Ok or Err, read nothing past the end, not loop, and be final: cleaning a result again changes nothing).
    • Switch: the Settings page (stripMetadata prop, the card in "Appearance & security"), the RPC module strip-metadata (loadStripMetadata, setStripMetadata), and GET/PUT /api/strip-metadata for the desktop app and scripts (CONTRACT.md section 11; capabilities read and write). Strings: messages/<locale>/stripMetadata.json, a 21st client namespace.
    • Crate: lopdf 0.47 (MIT), with no default features; it brings aes/cbc/ecb/cipher 0.5 (MIT or Apache-2.0, for encrypted PDFs it only detects here), rand 0.10, rangemap, brotli-decompressor (BSD-3-Clause or MIT), weezl, miniz_oxide 0.9 (flate2 still uses zlib-rs: any_zlib wins over its default backend). All compatible with AGPL-3.0. No new system package: libvips (tiffsave, heifsave with AV1, autorot) and ffmpeg/ffprobe 5.1 are already in the image.
    • Tests: media::strip::*::tests (each rewriter, the HEIF item confinement, tiff_look, the ffmpeg arguments and checks), media::strip::fuzz, tests/media_strip.rs (real files: JPEG with a real ICC profile and rotated EXIF, PNG, still and animated WebP, animated GIF, AVIF and HEIC from libvips, TIFF from ImageMagick, twelve audio and video containers made with ffmpeg and compared packet for packet, cover art, PDF; skipped when a tool is missing), tests/strip_db.rs (the switch and token scopes, ShareX, tus from the panel and from the app, checksums and duplicates of the cleaned file, a broken file kept and listed, an encrypted account, a docx, the switch off storing byte for byte), routes::api::strip_metadata::tests, web/tests/strip-metadata.test.tsx.

10. Verified on 2026-10-18

  • Fresh dev database -> modlight-server migrate -> pg_dump --schema-only identical to the reference database (only pg_dump's random \restrict key differs); drizzle.__drizzle_migrations rows (id, hash, created_at) and the settings seed identical.
  • Reference dump restored into the dev server -> the migrator applies nothing.
  • A session created by signing in on the reference server (8420) is accepted by the Rust reader on a copy of that database (same AUTH_SECRET), its getSession output is identical field by field and in order, the cookie re-issued on refresh is byte-identical to the one the reference issued, and the three refresh policies write what the table in section 8 says. A cookie with a tampered signature is refused.
  • tests/db_compat.rs checks every model against information_schema column by column.

To repeat the session check: sign in on 8420 (QA credentials in _wzorzec/qa-accounts.env), pg_dump -Fc the reference database into a new database on 55421, run the server against it with the reference AUTH_SECRET, and send the cookie to /_page?path=/login (the root layout's theme comes from the session user) or to POST /_rpc/tokens/myTokens.

11. The libvips binding (src/media/vips.rs)

Upstream does all image work with sharp 0.34.5, a thin C++ layer over libvips (8.17.3 in production). Calling libvips with the same operations and arguments gives the same pixels, so the port binds libvips directly: hand-written extern "C" declarations in vips/ffi.rs (no bindgen, no extra crates); build.rs links what pkg-config --libs vips names (libvips 8.15 or newer; brew install vips on macOS, libvips-dev in the image).

Layers: ffi.rs (raw declarations, GValue, VipsProgress) -> operation.rs (Operation: any libvips operation by nickname, like vips8's VImage::call) -> image.rs (Image: one owned reference, Blob, typed wrappers, the timeout) -> sharp.rs (Sharp: sharp's pipeline.cc and metadata() for the options the server uses).

Rules:

  1. init() runs vips_init once with sharp-runtime's settings (threads from effective_cpu_count(), cache 32 MB / 10 files / 50 operations).
  2. Everything is blocking: use the async methods or run_blocking (tokio's blocking pool, then vips_thread_shutdown). Expensive work goes through transform_queue::image_transforms() like upstream's imageTransforms.
  3. Image holds one GObject reference, Send but not Sync; metadata is written only through Image::mutate on an uncached private copy.
  4. Inputs are copied into vips_tracked_malloc memory so the 32 MB cache ceiling bounds what cached operations can pin.
  5. Errors take and clear the process-wide error buffer (concurrent failures can mix messages); VIPS warnings go to tracing at debug level.
  6. Loaders: JPEG, PNG, WebP, GIF, TIFF, HEIF/AVIF and SVG only (section 9, item 20); loads are sequential, as in sharp 0.34.5.
  7. Image::set_timeout is sharp's .timeout() (an eval handler that kills the evaluation); encoders that never report progress run to the end.
  8. Argument names and types are checked against the operation's own declarations before anything reaches GObject.

Adding an operation needs no new FFI: Operation::new("flatten")?.set("in", self).set("background", background).call()?.image("out"). Scalars are i32/f64/bool/Arg::Str, enums and flags Arg::Nick; arguments only newer libvips releases have go through set_if_known with a comment naming the release; encoders are *save_buffer operations passed to Image::save_buffer. Copy the order and arguments from sharp's pipeline.cc (sources in _scratch/srv-library/ref-src/sharp@0.34.5). The media processing port added flatten, join: { animated }, composite and text input (section 12).

12. Media processing API (for the media serving port)

What the serving routes (/[userCode]/[mediaCode]/[action]: convert, raw?motion=1, raw?w= for a video, timeline.vtt, timeline.jpg; the render of a .docx; /embed/[version]/[card]) and the media page call, by upstream name. MediaRow is db::models::Upload (with rowMimeType applied, as get_media returns it). Engine errors that upstream let escape (BusyError, TimeoutError, AbortedError from the limiter; AvTimeoutError, AvAbortedError from ffmpeg/ffprobe) come back as anyhow::Errors to downcast_ref - the route's transformRefusal.

Upstream Rust Notes
avConvert(req) (av-action.ts) media::av_action::av_convert(&AppState, AvConvertRequest) -> anyhow::Result<Option<Response>> None = not audio/video, take the image branch. AvConvertRequest { media, privileged, to: Option<String>, offset_seconds (?t=), clip_seconds (?d=), signal, admit: Option<AdmitHook> (consumeView, download: true), dry_run (?check=1) }. Builds the whole response (refusals with X-Convert-Refused/limit, 204 with X-Convert-Note, streamed download); runs inside the limiter itself.
convertOptionFor, conversionBudget, probeImage, convertImageBuffer (convert.ts) media::convert::{convert_option_for, conversion_budget_for(&ImageProbe, ConvertTarget), conversion_budget(animated, frame_pixels, total_pixels, to), probe_image(Bytes), convert_image_buffer(Bytes, ConvertTarget, ConvertOptions { animated, timeout_ms, signal }) -> anyhow::Result<Converted { buffer, mime, ext }>} the image branch of /convert; the ?check=1 row shortcut is conversion_budget(frames > 1, w*h, w*h*frames, to).
derivedKey, contentHash, mayCacheDerived, readDerived, writeDerived, readDerivedStream, writeDerivedFile media::derived_cache::* (read_derived(&Storage, key) -> Option<Bytes>, write_derived(&Storage, key, Bytes), ...) best effort, as upstream.
motionPreview(media, opts, signal) (motion.ts) media::motion::motion_preview(&AppState, &Upload, MotionOptions { width, height, fit: MotionFit }, AbortSignal) -> anyhow::Result<Result<(Bytes, u32 frames), MotionRefusal>> run it inside image_transforms().run(.., MOTION_WEIGHT, MOTION_TIMEOUT_MS); X-Motion-Frames is the u32; refusal tokens MotionRefusal::as_str(). Query parsing: motion_preview::{wants_motion, MOTION_DEFAULT_SIZE, clamp_motion_edge}.
timelineTrack, timelineSheet (timeline.ts) media::timeline::{timeline_track, timeline_sheet}(&AppState, &Upload, AbortSignal) -> anyhow::Result<Result<String / Bytes, TimelineRefusal>> inside the limiter with TIMELINE_TRACK_* / TIMELINE_*; actions timeline_preview::{TIMELINE_TRACK_ACTION, TIMELINE_SHEET_ACTION}.
videoPoster(media, signal) (poster.ts) media::poster::video_poster(&AppState, &Upload, AbortSignal) -> Option<Bytes> None = no poster (the route answers 415, never the video). open_video_source is shared with the timeline.
docxToHtml(buf), MAX_DOCX_BYTES, docxSupport (docx.ts) media::docx::docx_to_html(Bytes) -> DocxResult, media::docx_support::{MAX_DOCX_BYTES, docx_support} section 13; run it inside the limiter at thumbnail weight with DOCX_TIMEOUT_MS, as the route does.
renderCard(choice), parseCardFile, cardChoiceFor, cardUrl, EMBED_CARD_VERSION (card-render.ts, card.ts) media::card_render::render_card(&CardChoice) -> Result<Bytes, String>, media::card::{parse_card_file, card_choice_for, card_path, card_url, EMBED_CARD_VERSION, CARD_WIDTH, CARD_HEIGHT} render_card caches every pair it drew for the life of the process (shared futures, failures not cached).
diagnosePlayback(media) (playback-health.ts) media::playback_health::diagnose_playback(&PlaybackSource) -> Option<PlaybackDiagnosis> (PlaybackSource::from(&Upload)) serialises like upstream's object (playback prop of the media page).
showsSource, inlineTextPlan, sourceReadings, codeLanguageLabel, MAX_INLINE_TEXT (source-preview.ts) media::source_preview::*
downloadOptionsFor, downloadHintKey, downloadLabelKey, sectionHeadingKey (download-menu.ts) media::download_menu::* (DownloadOption serialises as upstream's literals) the media page's menu (canEncode = owner or admin).
estimateTargetSize, formatEstimate, ... (convert-size.ts) media::convert_size::*
playerLang, buildPlayerDictionary (player-i18n.ts) media::player_i18n::*
readVerdict/rememberVerdict/verdictKey (convert-verdict.ts), noteKey/REFUSAL_KEYS (convert-notes.ts) media::convert_verdict::*, media::convert_notes::* process-wide memo, as upstream.
REASON_TEXT, refusal, verdictResponse (av-action.ts) media::av_action::{REASON_TEXT, refusal, verdict_response, base_name, source_tag, source_kind_of} base_name is the download name rule (filename.replace(/\.[^.]+$/, "") || code).

Also ported and used inside: av_convert (planAv, runAvPlan, avVariant, frameOffset, animationFilters, webScaleFilter, audioCoverPath), convert_plan and convert_in_place (convertUploadInPlace behind the convert RPC module), frame_step (the player's arithmetic, server twin), pdf_lib (section 9, item 35).

Verification (re-runnable, see each script's header):

S=_scratch/srv-media-a      # scratch reference on 8446, Rust server on 8426
python3 server/tests/parity_media/corpus.py $S/corpus
python3 server/tests/parity_media/media_parity.py ref              # 595 cases on the reference
python3 server/tests/parity_media/media_parity.py rust --ffmpeg ref # the reference image's ffmpeg 5.1
python3 server/tests/parity_media/media_parity.py compare --ffmpeg ref
python3 server/tests/parity_media/media_parity.py cards            # all 155 embed cards
python3 server/tests/parity_media/convert_actions.py               # the convert RPC module, 20 checks

Through the routes: part B's tests/parity/media_serve.py on the merged binary, 1113/1113 (with the reference image's ffmpeg; encoder-dependent bytes compared by decoded content).

Measured on 2026-10-20 (libvips 8.18.7 here, 8.17.3 in the reference): statuses, meaningful headers and ffmpeg/ffprobe argument vectors identical in all 595 cases; outputs byte-identical for every JPEG, WebP (still and animated, including every motion tile), GIF, MP4/MOV remux and encode, MP3, WAV, poster and timeline sheet and track; PNG and AVIF differ in their bytes only (encoder builds; same pixels), Matroska and Ogg differ by their random segment/serial ids (same streams), PDFs as item 35; CMYK input shifts colour by about 3/255 (section 11). All 155 embed cards are pixel-identical. With Homebrew's ffmpeg 9 instead, see item 33.

13. Document preview (docx)

media::docx::docx_to_html(Bytes) -> DocxResult is docxToHtml (Result<DocxPreview { html, images, images_dropped }, DocxRefusal>, DocxRefusal::as_str() gives upstream's token: not-a-zip, malformed, too-big, too-many-entries, zip64, unsupported-method, too-large, legacy-doc, not-a-document); media::zip_guard::inspect_zip and media::docx_support::{docx_support, MAX_DOCX_BYTES} are the two helpers. The route (/{tag}/{code}/render, gating, status codes) is not part of it.

media/mammoth/ is mammoth 1.12.1 ported file by file (docx/body_reader.rs = lib/docx/body-reader.js, ...), with what it runs of JSZip (jszip.rs: central directory, local names, size check, no CRC check), @xmldom/xmldom (xml/xmldom.rs: sax.js and the DOM rules, every warning/error is a failure because mammoth throws on them), lop (lop.rs, the style-map grammar's combinators) and dingbat-to-unicode. mammoth/js.rs holds the JavaScript semantics it needs (parseInt, ToNumber, {} as a map with Object.prototype showing through, UTF-16 strings). The conversion runs on a thread of its own (256 MB stack, the readers recurse); pictures go through the libvips binding one at a time in document order (mapSeries upstream), so the 3 MB budget drops the same ones. New dependency: flate2 on zlib-rs (zlib's inflate semantics, as node:zlib and pako).

Verified by tests/parity_media/: docx_corpus.py builds 229 documents (hand-built OOXML per mammoth feature, embedded style maps, pictures, hostile archives, xmldom edge cases, Object.prototype names, macOS textutil output, mammoth's own Word/LibreOffice test documents); docx_parity.py runs each through upstream's own docx.ts, zip-guard.ts, docx-support.ts, sharp-runtime.ts and limits.ts inside the reference container (node 24 strips the types; imports rewritten to the container's packages) and through docx_to_html (tests/media_docx.rs, ignored test parity_dump), and compares the JSON. On 2026-10-20: 222 identical byte for byte, pictures included (libvips 8.18 here and 8.17.3 in the reference wrote the same WebP); 3 known differences (items 3 and 4 below); 4 that never finish upstream (item 1). docx_route_e2e.py fetches a sample through the reference's real /render route with minted tickets: same bytes as the harness.

To repeat (from the main checkout; the reference app runs on 8446, see _scratch/srv-media-a/):

python3 server/tests/parity_media/docx_corpus.py _scratch/srv-media-a/docx/corpus
python3 server/tests/parity_media/docx_parity.py --corpus _scratch/srv-media-a/docx/corpus \
  --work _scratch/srv-media-a/docx/run --container modlight-srvmedia-a-ref-app-1
python3 server/tests/parity_media/docx_route_e2e.py --corpus _scratch/srv-media-a/docx/corpus \
  --work _scratch/srv-media-a/docx/run f-basic.docx i-budget.docx h-bomb-lying.docx

Differences, all deliberate or unavoidable:

  1. Hangs. Upstream never finishes on some inputs the guard lets through; the port answers. JSZip loops forever on ZIP64 "extensible data" whose length walks the reader backwards (not-a-document here); xmldom's error recovery is quadratic (the port stops at the first report, which is the same answer); some error-free scans are quadratic (many unterminated <? or CDATA sections, many distinct element names: memoised here, same tree).
  2. Entries the guard never inflated (a central record beyond the declared count, or the directory JSZip finds after its prepended-data shift) are expanded with a ceiling of 40 MiB + 1 (JSZip: none) and fail past it.
  3. Prototype pollution not reproduced. A style with w:type="__proto__" writes itself onto Object.prototype in upstream (every object in the Node process gains a property, for the rest of its life: e.g. a style id w:type removes every <w:br/>). Here such a style is ignored. An object assigned to a __proto__ key of mammoth's maps does not become the prototype of the other keys (inherited Object.prototype members are reproduced: constructor, toString, ... as ids, element names, relationship types).
  4. Nesting depth. Upstream runs out of V8 stack at a depth that depends on what nests and on JIT warm-up (fresh process: 1898 nested elements in a paragraph, 909 nested w:sdt, 306 nested tables; warm: 2499, 1249, 312). The port fails an XML part nested deeper than 1901 elements (the fresh limit for plain nesting); deeper w:sdt/table nesting below that renders here and fails upstream.
  5. Lone UTF-16 surrogates (from character references) are carried as U+10F800..U+10FFFF inside the port and resolved at the end exactly as the JavaScript string would be encoded; a document containing those private-use code points literally would see them treated as surrogates.