Skip to content

database-runbook.md

Latest commit

 

History

History
250 lines (203 loc) · 11.2 KB

database-runbook.md

File metadata and controls

250 lines (203 loc) · 11.2 KB
Raw

Database backup, upgrade, restore, and rollback

Status: implemented and integration-tested, pre-alpha

The supported v1 backup procedure creates one verified file while the daemon continues running. It uses SQLite's online backup API; it does not copy a live irc.db, irc.db-wal, and irc.db-shm set with generic filesystem tools. Restore, migration, and every other database mutation remain deliberately stopped-service operations protected by the daemon's exclusive lease.

The database contains plaintext retained conversations, canonical account avatars, account presentation choices, account and channel relationships, password verifiers, invitation metadata, and operational state. A backup has the same confidentiality as the live server and can outlive every configured retention rule. Store it in a mode-0700 directory, encrypt off-host copies, restrict the encryption key separately, and define a backup deletion policy explicitly. Deleting a backup is not a forensic-erasure guarantee for snapshots or storage-provider replicas.

State permissions

Every daemon and offline database open creates a missing state directory as mode 0700 and new database and lease files as mode 0600. Existing database, lease, WAL, SHM, and rollback-journal files must have no group or other permissions. The binary refuses a broader existing state set before opening SQLite; it never silently changes permissions because that could hide a prior exposure or break an intentionally shared path.

After upgrading an older direct-launch installation, inspect and repair its state while the service is stopped:

sudo systemctl stop telex-ircd.service
sudo chmod 0700 /var/lib/telex
sudo chmod 0600 /var/lib/telex/irc.db \
  /var/lib/telex/irc.db.lock
for suffix in -wal -shm -journal; do
  if sudo test -e "/var/lib/telex/irc.db$suffix"; then
    sudo chmod 0600 "/var/lib/telex/irc.db$suffix"
  fi
done

Confirm ownership is the dedicated service account before restarting. Treat a previously group/world-readable database as potentially disclosed; tightening the mode prevents future reads but cannot retract an earlier copy. The packaged systemd unit's UMask=0077 and mode-0700 StateDirectory remain defense in depth rather than the sole privacy control.

The two maintenance commands

database backup <new-file>:

  • opens the source read-only and can run while the daemon owns its advisory application lease;
  • refuses a missing/non-file source and refuses to overwrite the destination or any destination sidecar;
  • runs full SQLite and foreign-key checks, and uses SQLite's online backup API in bounded page batches so committed WAL content is included consistently while writers continue;
  • creates one mode-0600 destination, verifies the copy, syncs it and its parent directory, and removes an incomplete destination on failure; and
  • copies the source schema exactly. In particular, running a newer binary's backup command does not migrate an older source.

The command produces a verified plaintext SQLite file. It does not compress or encrypt that file; the deployment must apply encryption before transferring a backup to storage whose confidentiality boundary differs from the live state directory.

database prepare:

  • requires an existing database selected by IRC_CONFIG/IRC_DATABASE and the exclusive lease;
  • intentionally opens it with the current binary, applies supported migrations and ordinary bounded metadata cleanup, then runs integrity and foreign-key checks; and
  • requires a complete WAL checkpoint and sync before reporting success, making the prepared main file safe for the candidate swap below.

prepare is mutating. Never point it at the only copy of an old database when rehearsing an upgrade.

Create and verify a backup

These examples use the production paths expected by the checked-in systemd unit. Choose a UTC-stamped filename that does not exist; the command will not replace it.

sudo install -d -o telex -g telex -m 0700 \
  /var/backups/telex

BACKUP=/var/backups/telex/irc-2026-07-18T180000Z.db
sudo -u telex env \
  IRC_CONFIG=/etc/telex/server.toml \
  /usr/local/bin/telex-ircd database backup "$BACKUP"
sudo sha256sum "$BACKUP"

Success reports backup=complete, the copied schema version, destination, and integrity=ok. Record that output, the binary version or commit, UTC time, and checksum outside the backup file. Do not restart the service merely to test a backup; prepare and inspect a separate candidate first.

SQLite may briefly retry a page batch when a writer is active, but the finished destination is one consistent snapshot. Do not checkpoint, copy, or discard the live source sidecars around this operation.

Rehearse an upgrade without touching production state

Build the prospective binary with Cargo.lock and run its tests first. Copy the verified backup to a new candidate on the same filesystem as the production database and prepare that copy with the prospective binary; the live service may remain active because the candidate has its own database and lease:

CANDIDATE=/var/lib/telex/irc.upgrade-candidate.db
sudo -u telex install -m 0600 "$BACKUP" "$CANDIDATE"
sudo -u telex env \
  IRC_DATABASE="$CANDIDATE" \
  /path/to/prospective/telex-ircd database prepare

Check schema_version, content_epoch, integrity=ok, and wal_checkpoint=complete. The content epoch should equal the source epoch after preparation. A migration failure leaves the production database and the original backup untouched; discard the failed candidate and investigate.

The repository continuously exercises the fresh standalone baseline. Earlier pre-release database lineages are intentionally unsupported because none were deployed with user content. database prepare is the host-specific rehearsal: it proves the selected backup, filesystem, ownership, and prospective binary together.

Install a prepared candidate

Before any switch, make a fresh pre-switch backup of the current production database. Create a new rollback directory on the same filesystem. Keep the canonical .lock file in place; it is an empty reusable lease sidecar, not database content.

With the service stopped and the candidate already prepared:

DB=/var/lib/telex/irc.db
ROLLBACK=/var/lib/telex/rollback-2026-07-18T180000Z
sudo install -d -o telex -g telex -m 0700 "$ROLLBACK"
sudo mv -- "$DB" "$ROLLBACK/irc.db"
for suffix in -wal -shm -journal; do
  if sudo test -e "$DB$suffix"; then
    sudo mv -- "$DB$suffix" "$ROLLBACK/irc.db$suffix"
  fi
done
sudo mv -- "$CANDIDATE" "$DB"

The service is stopped, the candidate is on the same filesystem, and the final mv is therefore one atomic rename. A crash between the two renames leaves an obvious stopped-service state with both files recoverable; it cannot expose a partially copied database to the daemon.

For an upgrade, retain the previous binary before atomically installing the new one. Start the service and require all of the following before accepting normal traffic:

sudo systemctl start telex-ircd.service
sudo systemctl status telex-ircd.service
sudo journalctl -u telex-ircd.service -n 100 --no-pager
curl --fail https://irc.example.net:8098/v1/health

Also perform a certificate-validated IRC login, join, message, disconnect, and resume with a disposable account. The health endpoint proves a storage-worker round trip, not authentication, history, or protocol behavior.

Restore and rollback

Restoring an older backup is the same candidate procedure:

  1. stop the service and preserve a new backup of the current state;
  2. copy the selected backup to a new candidate beside the production database;
  3. run the destination binary's database prepare against the candidate;
  4. verify its schema and content epoch against the restore ticket or operator record;
  5. quarantine the current main database and any -wal, -shm, or -journal sidecars exactly as above;
  6. atomically rename the prepared candidate to the configured database path;
  7. start, inspect logs and health, then perform authenticated IRC and resume smoke tests.

To roll back an upgrade, stop the new service, reinstall the retained old binary, quarantine the failed current database set, and move the untouched old database set back from its rollback directory. If that set is unavailable, prepare a copy of the pre-upgrade backup with the old binary and install that candidate instead.

Migrations after the baseline are forward-only. An older baseline-era binary will reject a newer schema; never try to repair that by editing PRAGMA user_version. Pre-baseline binaries and databases are intentionally incompatible with the collapsed schema even though their old version numbers were higher. Do not run an old pre-baseline binary against current state or prepare one of its databases with a current binary. Restoring a compatible pre-upgrade checkpoint also discards every message, cursor, account change, and invitation created after that checkpoint. Keep the acceptance window bounded and make the tradeoff explicit before a live upgrade.

Recovery and reset consequences

Offline account-password changes and content reset use the same lease and must run while the service is stopped. Their detailed state transitions are documented in account password changes and content reset.

A backup made before a password change contains the old active verifier. Restoring it undoes the change and can make a compromised password valid again. Treat such a restore as a new security incident and change the password again before reopening the service.

A backup made before content reset contains the prior conversations and prior content epoch. Restoring it deliberately brings both back. If reset is being used to minimize retained content rather than to rehearse operations, retaining that backup defeats the server-side purpose; destroy it according to the backup policy after the explicit rollback window closes.

Automated evidence

The runbook boundary is covered by these repository checks:

  • cargo test --locked --test database_maintenance performs a real backup while a live store and its application lease remain active, proves later writes continue, rejects overwrite, prepares a baseline candidate, swaps it into place, and proves that only the saved checkpoint returns;
  • cargo test --locked --test history_store_conformance exercises retained schema integrity, explicit version rejection, restart, credentials, recovery, retention, and reset invariants;
  • cargo test --locked --test cli_admin covers destructive confirmations, lease refusal, account/channel authority, invitations, and content reset; and
  • dev/container-lab/lab.sh acceptance exercises packaged release builds, baseline creation, TLS onboarding, retained-state restart, backup candidate preparation/swap/rollback, and content reset in disposable service state.

These tests prove software behavior, not a particular deployment. Before launch, exercise the real disk, private filesystem, verified backup and candidate preparation, fresh-schema swap, off-host round trip, systemd unit, TLS renewal, firewall, edge reboot/recovery, and operator access path. Run the online backup timer once under real service conditions before relying on it.