Skip to content

operations.md

Latest commit

 

History

History
275 lines (223 loc) · 12.4 KB

operations.md

File metadata and controls

275 lines (223 loc) · 12.4 KB
Raw

Standalone service operations

Status: implemented packaging, pre-alpha runbook

The supported production shape is one release binary supervised by systemd under a dedicated unprivileged account. Containers remain a development and test tool, not the recommended production deployment artifact.

The checked-in unit assumes Linux with systemd, an executable installed at /usr/local/bin/telex-ircd, configuration at /etc/telex/server.toml, state under /var/lib/telex, and unprivileged listener ports such as 6697, 8097, and 8098. Review every path and hardening directive against the target distribution before installing it.

The current deployment baseline is Ubuntu 24.04 or 26.04 with systemd and the stable Rust 1.95 toolchain declared in Cargo.toml. Builds use Cargo.lock and must pass the container build at that exact minimum. Newer stable toolchains may be used for development, but nightly-only language or library features are out of scope.

Build and install

Build on the target architecture with the repository lockfile:

cargo test --locked --all-targets --all-features
cargo build --locked --release
sudo install -Dm0755 target/release/telex-ircd \
  /usr/local/bin/telex-ircd
sudo install -Dm0644 docs/operations.md \
  /usr/local/share/doc/telex-ircd/operations.md

Create the service identity and configuration directory. The sysusers file does not grant a shell, supplementary groups, or login credentials.

sudo install -Dm0644 packaging/systemd/telex-ircd.sysusers \
  /etc/sysusers.d/telex-ircd.conf
sudo systemd-sysusers
sudo install -d -o root -g telex -m 0750 /etc/telex
sudo install -d -o telex -g telex -m 0700 /var/lib/telex
sudo install -o root -g telex -m 0640 \
  config/examples/public.toml \
  /etc/telex/server.toml

Edit the copied profile before starting it. In particular, verify the public names, listener topology, WebSocket origin, database path, TLS paths, admission policy, and retention bounds. Do not make the service account or configuration file writable merely to simplify deployment.

Run the same policy preflight as the service, as the eventual service user:

sudo -u telex env \
  IRC_CONFIG=/etc/telex/server.toml \
  /usr/local/bin/telex-ircd config check

The preflight deliberately does not create the database, read certificates, or bind ports. The real start remains the runtime-resource check.

The public example names a placeholder configured operator account. Replace that name in the configuration, create the matching first human account while the daemon is stopped, and optionally bootstrap only invitation issue/revoke so that account can onboard the next person:

sudo -u telex env \
  IRC_CONFIG=/etc/telex/server.toml \
  /usr/local/bin/telex-ircd account create operator
sudo -u telex env \
  IRC_CONFIG=/etc/telex/server.toml \
  /usr/local/bin/telex-ircd account capability bootstrap-invite-steward operator

The configured-operator status and the invite-steward capabilities are independent. After startup, the operator panel can grant the small supported global capabilities without creating a broad administrator role.

Install and start the unit only after the profile passes review:

sudo install -Dm0644 packaging/systemd/telex-ircd.service \
  /etc/systemd/system/telex-ircd.service
sudo systemctl daemon-reload
sudo systemctl enable --now telex-ircd.service
sudo systemctl status telex-ircd.service
sudo journalctl -u telex-ircd.service -n 100 --no-pager

The unit creates /var/lib/telex with mode 0700, drops all Linux capabilities, constrains writable paths, limits address families, and applies a 512 MiB memory ceiling. The ceiling leaves headroom above the daemon's three independently bounded four-worker Argon2 pools; remeasure it if those concurrency limits or password-hashing parameters change. The unit cannot bind privileged ports below 1024. Keep the daemon unprivileged and use a firewall redirect or a reviewed front end if a legacy port is required.

Readiness and smoke checks

The process logs IRC listeners ready only after configuration, TLS identity, database migration and integrity checks, content pruning, the exclusive database lease, and every configured socket bind have succeeded. systemd's Type=exec confirms process execution, while this log line is the application readiness boundary.

For a TLS IRC listener, complete a certificate-validated connection and issue an IRC CAP LS 302/registration exchange with a disposable test account. When the native control listener is enabled, GET /v1/health is a public, credential-free readiness probe that performs a bounded SQLite-worker round trip and discloses only ready or unavailable. It is not a full integrity scan. GET /v1 remains the public server-status document. The compatibility matrix remains the release check for real IRC clients.

Exact aggregate counters and effective runtime bounds require an account with the separately granted diagnostics.read capability at GET /v1/diagnostics. Do not put that account's password in a unit file, probe URL, or command-line argument. See the native control API for provisioning and the response's deliberate omissions.

Normal logs contain listener addresses, bounded counters, stable opaque IDs, and failures needed to operate the daemon. They must not contain message bodies, passwords, invitation bearer tokens, DM relationships, remote client addresses, or private-channel membership. Keep journald access restricted anyway. Runtime rejections are recorded as sparse aggregate warnings and one complete aggregate snapshot at clean shutdown; see runtime rate limits for the exact scopes and anonymity tradeoff.

Shutdown and restart

The daemon handles both interactive SIGINT and systemd's SIGTERM. Shutdown stops accepting new IRC and WebSocket connections, tells the HTTP control listener to stop, bounds its drain to five seconds, closes client transports, orders storage shutdown behind already queued work, joins the SQLite worker, and releases the database lease. A successful sequence logs IRC server stopped cleanly and exits zero.

The unit gives that sequence 15 seconds before systemd may force termination. Clients should treat a restart as an ordinary disconnect and resume from their durable cursor after reconnecting; the daemon does not promise to deliver a final IRC error line during shutdown.

Use systemctl restart telex-ircd.service for binary, configuration, or TLS identity changes. Offline account, channel, invite, authority, and content administration commands require the service to be stopped so the database lease protects against a second writer.

TLS certificates

The daemon reads and validates the complete certificate chain and matching unencrypted private key once, before binding. It has no certificate hot reload and does not interpret SIGHUP. A renewal is active only after a clean service restart.

For direct TLS, deploy the chain and key into a root-managed directory that the telex group can traverse, with the key readable by that group but not by other users. A typical destination is /etc/telex/tls with directory mode 0750 and key mode 0640. Prefer a certificate-manager deploy hook that atomically replaces both files and then runs:

systemctl try-restart telex-ircd.service

Do not broadly relax /etc/letsencrypt permissions. Exercise the final certificate-renewal path on the real host before launch, including a failed renewal and rollback to the previous pair.

When a public edge performs only layer-4 pass-through, issue the daemon its own certificate on the daemon host. Do not copy an edge wildcard private key merely because that key already covers the name. A DNS-01 credential should be scoped to the one required zone and DNS mutation permission, stored root-only outside Git, and restricted by source address when the provider and deployment have a stable egress address.

A robust deploy hook stages a versioned certificate/key pair, validates the hostname, validity window, chain, and key match, atomically changes one current path, and only then restarts the daemon. If readiness fails, restore the prior path before returning failure. Exercise the real deploy hook during a saved renewal dry run; a successful ACME simulation that skips the hook does not prove the service can consume renewed material.

Reverse proxies

Direct daemon TLS or a layer-4 TLS pass-through is the least ambiguous v1 topology. If a front end terminates TLS, every plaintext backend listener must remain on loopback and the loopback hop becomes part of the trusted host boundary.

The transports have distinct proxy requirements:

  • raw IRC needs a TCP/stream proxy, not an HTTP reverse proxy;
  • IRC-over-WebSocket requires /irc with no query string and preservation of Origin, Upgrade, Connection, and Sec-WebSocket-Protocol;
  • the native control listener ignores all forwarded-client headers, so its privacy-safe rate limiter sees the proxy address rather than the end user;
  • browser control mutations compare Origin directly with Host and with the daemon's own TLS state. Consequently, terminating HTTPS in a generic HTTP proxy while speaking plaintext to the control listener is not a supported v1 onboarding topology. Use direct TLS or layer-4 pass-through for that listener.

Never rewrite arbitrary browser origins into an allowed value. That would erase the control API's CSRF boundary. A future trusted-proxy design must name its allowed peers and external origin explicitly before this restriction can change.

Ubuntu Nginx stream packaging

On Ubuntu, nginx -V may report stream support as a compatible dynamic module even when the runtime module package is absent. On the tested Ubuntu 24.04 edge, installing libnginx-mod-stream created the module-loader symlink under /etc/nginx/modules-enabled/; no Nginx rebuild was necessary. Confirm the module is actually loaded and run nginx -t before editing live listeners.

A conventional layout adds this block at top level, outside http {}:

stream {
    include /etc/nginx/streams-enabled/*.conf;
}

Each dedicated public port can then pass encrypted bytes directly to the same private daemon port. This example address is documentation-only and must be replaced:

server {
    listen 6697;
    proxy_connect_timeout 5s;
    proxy_timeout 24h;
    proxy_socket_keepalive on;
    proxy_pass 192.0.2.10:6697;
}

Telex also sends a protocol-level IRC PING after three minutes without client traffic and allows 90 seconds for the matching PONG. This keeps ordinary clients, NAT mappings, and the transparent proxy path active while detecting a dead application rather than trusting TCP indefinitely. Keep proxy_timeout comfortably above the combined heartbeat window; the 24-hour edge value leaves Telex responsible for liveness instead of making Nginx terminate quiet IRC sessions first.

Repeat the shape for WebSocket and control ports with appropriate idle timeouts. Do not add ssl to the edge listener, install the daemon key there, or enable PROXY protocol when the intent is transparent end-to-end TLS. Stream access logging is absent unless configured; avoid adding it casually when client addresses are not operationally required.

Back up the live Nginx configuration, test before reload, and verify both the new streams and every pre-existing HTTP site afterward. Provider security-group rules and the daemon-host firewall are separate boundaries: open only the exact TCP destination ports, and initially restricting ingress to an operator source is a useful private-acceptance stage. Never express a few non-contiguous ports as one broad range.

Firewall and recovery notes

Expose only configured public listeners. The database and service-management interfaces are never network services. A private node normally needs its TLS IRC port, and optionally its TLS WebSocket/control ports. SSH and monitoring policy belong to the host runbook.

database backup is a read-only online operation using SQLite's backup API; the daemon may remain active. An isolated restore candidate has its own lease and may also be prepared while production runs. Restore/swap, rollback, credential recovery, content reset, and every mutation of the production file remain stopped-service operations protected by the daemon's exclusive lease. Use the tested database backup, upgrade, restore, and rollback runbook; do not improvise a live file copy. In particular, restoring a pre-recovery or pre-reset backup also restores the credentials or content it contains.