Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 16 additions & 2 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,9 @@

FastAPI backend (`src/databases_ui`) serving a React/TypeScript console (`frontend/`), and the Jobs'
runner (`runner.py`) in the same image, with a Helm chart (`charts/databases-ui`). State lives in
Kubernetes (Database resources, Jobs, RBAC) and Blob Storage (snapshots and their shares, the audit
log); the server stores nothing.
Kubernetes (Database resources, Jobs, RBAC) and Blob Storage (snapshots and their shares, scripts,
the audit log); the server stores nothing. External databases (not CloudNativePG) are the
catalog's `external` entries; scripts are `sql.py` (what may run) and `runner.script`.

## Things that fail silently

Expand Down Expand Up @@ -33,6 +34,19 @@ log); the server stores nothing.
postponed annotations turn them into strings and FastAPI answers 422 everywhere.
- **The page may hold no inline script or style.** The CSP blocks it and the page renders blank;
CI greps `frontend/dist/index.html` for it.
- **The admission policy pins an external database's host, port, TLS mode and password Secret
from the rendered catalog** (`templates/jobpolicy.yaml`). A catalog in `existingConfigMap` isn't
rendered, so its external databases' Jobs are refused; a new Job variable that names another
host or Secret needs the policy changed in the same change.
- **A script's output reaches the server through the pod's termination message**, which Kubernetes
caps at 4096 bytes. `runner.termination_message` trims the output, never the JSON; writing it any
other way truncates mid-string and the server shows the raw text as the failure reason.
- **A script travels in the Job (`DBUI_SCRIPT`), and a single environment variable is capped near
128 kB by Linux** (`MAX_ARG_STRLEN`), so `sql.MAX_BYTES` stays at 100 kB. Moving the script to a
volume means the admission policy's volume rule changes too.
- **`sql.problems` is the only thing between a script and psql's backslash commands** (`\!` runs a
shell in the Job's pod). Keep it a lexer: quoted strings, dollar quotes, comments and BEGIN
ATOMIC bodies must be skipped, or ordinary function bodies get refused.
- **The image name follows the repository name** (`ghcr.io/magmamoose/databases-ui`). The chart's
default image is that name, and the Jobs run the same image; a rename means a chart change in the
same PR.
Expand Down
93 changes: 80 additions & 13 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,14 +1,16 @@
# databases-ui

Let app teams snapshot, restore and clone their own PostgreSQL databases on
[CloudNativePG](https://cloudnative-pg.io/), without asking the platform team. People sign in with
Entra ID and see the databases of the namespaces Kubernetes lets them into; every action runs as a
Kubernetes Job next to the database.

- **Access is Kubernetes RBAC.** A `Database` resource names its app's namespace in a label. Who
may see it, snapshot it or restore it is what the API server says that person may do in that
namespace, asked with a SubjectAccessReview: `view` sees, `edit` snapshots and restores, `admin`
restores production. There is no second list of who may do what.
[CloudNativePG](https://cloudnative-pg.io/), and run their migration scripts on them or on any other
PostgreSQL, without asking the platform team. People sign in with Entra ID and see the databases of
the namespaces Kubernetes lets them into; every action runs as a Kubernetes Job next to the
database.

- **Access is Kubernetes RBAC.** A `Database` resource names its app's namespace in a label (an
external database, in the catalog). Who may see it, snapshot it, restore it or run a script on it
is what the API server says that person may do in that namespace, asked with a
SubjectAccessReview: `view` sees, `edit` snapshots, restores and runs scripts, `admin` restores
and runs scripts in production. There is no second list of who may do what.
- **Snapshots on demand.** `pg_dump` in the custom format, streamed straight into Azure Blob
Storage with who took it and why. Nothing touches a disk, and nothing leaves the cluster.
- **Restore in place or into a copy, exactly.** In one transaction, everything the database's owner
Expand All @@ -18,8 +20,14 @@ Kubernetes Job next to the database.
by CloudNativePG.
- **Clone across environments, once shared.** Someone who may snapshot a database in acc shares a
snapshot with a namespace in dev; whoever may restore there can then clone it.
- **Production is careful.** A restore in a production environment waits for a second person, and
snapshots taken there can't be downloaded.
- **Scripts, checked or run.** Paste or open a `.sql` file: it runs as the database's owner in one
transaction. A check runs it, shows what it said and rolls back; a run commits, after a snapshot
if asked. One failing statement, or a lock it can't get within 15 seconds, changes nothing.
- **Any PostgreSQL.** A server outside CloudNativePG goes in the catalog with its host, a login and
that login's password Secret: it is listed, snapshotted and scripted like the others.
- **Production is careful.** A restore or a script in a production environment waits for a second
person, who reads the script before approving it; snapshots taken there can't be downloaded,
and a script's output there names its statements but shows no rows.
- **Database users**: the owner roles of the databases you see, and the Secrets with their
passwords.
- **An audit log** of who asked for what, who approved it and how it ended, per environment.
Expand Down Expand Up @@ -135,6 +143,61 @@ Without the label, a database belongs to its own namespace (`database`): only wh
usually the platform team, sees it. Without the annotation, a restore in place scales nothing; the
owner's other sessions are still ended first.

## External databases

A PostgreSQL server that isn't a CloudNativePG `Database` (a VM, a managed service, another
operator's cluster) goes under its environment's `external` in the catalog:

```yaml
environments:
- name: acc
storage:
url: https://<account>.blob.core.windows.net/acc-snapshots
external:
- name: billing # what this app calls it, unique in the environment
namespace: acc-billing # whose people may act on it; empty: the environment's own
host: billing-db.example.internal
port: 5432 # the default
database: billing # the database on that server
user: billing_owner # the login the Jobs use
passwordSecret: billing-db # a Secret in the environment's namespace, key `password`
caSecret: billing-db-ca # its `ca.crt` verifies the server
tls: verify-full # the default; or require, or disable
```

- **The login is what a script can do.** Give the Jobs the schema's owner, as migration scripts
need, not a superuser: a superuser's script can do anything to the server.
- **`verify-full` without a `caSecret`** checks the certificate against the image's public roots
(`sslrootcert=system`), for a server with a public certificate. `require` encrypts without
checking who answers; `disable` is for a server without TLS on a network you trust.
- **No restores.** A restore drops everything its login owns first, which on a server this app
doesn't manage could be anything. Its snapshots are ordinary `pg_dump` archives: download one
outside production, or restore it by hand.
- **The Jobs must reach it.** If the namespace's NetworkPolicies limit egress, admit the Jobs' pods
(`app.kubernetes.io/component: job`) to the server's address and port. The admission policy
admits exactly the catalog's host, port, TLS mode and password Secret, so the catalog has to be
in the chart's values (`catalog.content` or `catalog.text`).

## What a script does

- **One transaction, every statement or none.** `psql` runs `BEGIN`, `SET LOCAL lock_timeout =
'15s'`, the script with `ON_ERROR_STOP`, then `COMMIT` (run) or `ROLLBACK` (check). At the first
error psql stops before either and PostgreSQL rolls everything back, so a check shows exactly
what a run would do.
- **No BEGIN, COMMIT or psql commands.** A script that opens or ends a transaction of its own, or
holds a backslash command (`\!`, `\connect`, `\o`), is refused before a Job exists, with the
line. Statements that can't run in a transaction (`CREATE INDEX CONCURRENTLY`, `VACUUM`) fail.
- **A lock it can't get within 15 seconds fails the script**, rather than queueing the app's own
queries behind an `ALTER TABLE` that waits for a long one.
- **A snapshot first, when asked.** A run can take a snapshot before it starts; if the snapshot
fails, the script doesn't run.
- **What it said.** The last lines of psql's output (statement tags, notices, errors with the
script's line numbers, and rows outside production) show under Activity. The script itself is
kept beside the snapshots (`scripts/<database>/<id>.sql`) and in the Job, with its sha256 in the
audit log.
- **In production, a second person reads it first.** The request waits, suspended; whoever approves
sees the script the Job holds, which can't change after it is created.

## What a restore does

- **One transaction, the snapshot exactly.** `psql` runs `BEGIN`, `DROP OWNED BY CURRENT_USER`, then
Expand Down Expand Up @@ -257,9 +320,13 @@ and a Job's work.
the reviews ask about nobody and refuse everything.
- **No passwords in the server.** It reads no Secret; a Job's pod gets the owner's password from
the kubelet, and the admission policy keeps the server's Jobs to what it builds.
- **A production restore takes two people.** It is created suspended and starts only when someone
else who may restore there approves it; a Job's pod template can't change after creation, so
what runs is what was asked for. Unapproved requests expire after a day.
- **A production restore or script takes two people.** It is created suspended and starts only when
someone else who may restore (or run scripts) there approves it; a Job's pod template can't
change after creation, so what runs is what was asked for. Unapproved requests expire after a
day.
- **A script runs as the database's owner, never as a superuser the app chose.** For an external
database the catalog names the login, and the admission policy holds a script Job to that host,
port, TLS mode and password Secret.
- **Data leaves an environment only when someone who may snapshot it says so**: a share names one
namespace in one environment, and is in the audit log. Downloads, outside production only,
stream through the server and are recorded too.
Expand Down
6 changes: 4 additions & 2 deletions charts/databases-ui/Chart.yaml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
apiVersion: v2
name: databases-ui
description: Let app teams snapshot, restore and clone their own PostgreSQL databases on CloudNativePG, with Entra ID sign-in, Kubernetes Jobs and Azure Blob Storage.
description: Let app teams snapshot, restore and clone their own PostgreSQL databases on CloudNativePG, and run migration scripts on them or any other PostgreSQL, with Entra ID sign-in, Kubernetes Jobs and Azure Blob Storage.
type: application

# version is the CHART version and appVersion the application version. The release
Expand Down Expand Up @@ -33,4 +33,6 @@ annotations:
url: https://github.com/MagmaMoose/databases-ui
artifacthub.io/changes: |
- kind: added
description: First release.
description: Scripts, checked (rolled back) or run (committed, after a snapshot if asked), with a second person in production.
- kind: added
description: External databases, any PostgreSQL the catalog names, for snapshots and scripts.
15 changes: 15 additions & 0 deletions charts/databases-ui/ci/default-values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,21 @@ catalog:
storage:
url: https://examplesnapshots.blob.core.windows.net/dev
cloneFrom: [acc]
external:
- name: ledger
namespace: dev-ledger
host: ledger-db.example.internal
database: ledger
user: ledger_owner
passwordSecret: ledger-db
caSecret: ledger-db-ca
- name: archive
host: 10.20.30.40
port: 6432
database: archive
user: archive_owner
passwordSecret: archive-db
tls: require
- name: acc
kubernetes: none
storage:
Expand Down
21 changes: 17 additions & 4 deletions charts/databases-ui/templates/jobpolicy.yaml
Original file line number Diff line number Diff line change
@@ -1,12 +1,18 @@
{{- if .Values.jobPolicy.enabled }}
{{- $urls := list }}
{{- $servers := list }}
{{- range (default list (fromYaml (include "databases-ui.catalogContent" .)).environments) }}
{{- $urls = append $urls (printf "%q" (trimSuffix "/" .storage.url)) }}
{{- range (default list .external) }}
{{- $servers = append $servers . }}
{{- end }}
{{- end }}
# Creating Jobs in a namespace is, to Kubernetes, reading every Secret there. This keeps the Jobs
# this app creates to what it builds itself: its own image as `job`, one owner's password (never
# a superuser's), the server's CA certificate and no other Secret, TLS to a database in the Job's
# own namespace, and the catalog's storage.
# own namespace or an external database exactly as the catalog has it (host, port, TLS and its
# own password Secret), and the catalog's storage. External databases are known here only when
# the catalog is in the values (catalog.content or catalog.text), not catalog.existingConfigMap.
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
Expand Down Expand Up @@ -55,9 +61,16 @@ spec:
)))
message: databases-ui Jobs read one owner's password from its Secret, never a superuser's
- expression: >-
variables.env.exists(e, e.name == "PGSSLMODE" && e.value == "verify-full")
&& variables.env.all(e, e.name != "PGHOST" || (has(e.value) && e.value.matches("^[a-z0-9-]+-rw[.]" + request.namespace + "[.]svc$")))
message: databases-ui Jobs connect over verified TLS to a database in their own namespace
variables.env.filter(e, e.name == "PGHOST").size() == 1 && variables.env.exists(e, e.name == "PGHOST" && has(e.value) && (
(e.value.matches("^[a-z0-9-]+-rw[.]" + request.namespace + "[.]svc$") && variables.env.exists(m, m.name == "PGSSLMODE" && m.value == "verify-full"))
{{- range $servers }}
|| (e.value == {{ .host | quote }}
&& variables.env.exists(m, m.name == "PGSSLMODE" && m.value == {{ default "verify-full" .tls | quote }})
&& variables.env.exists(m, m.name == "PGPORT" && m.value == {{ default 5432 .port | toString | quote }})
&& variables.env.exists(m, m.name == "PGPASSWORD" && has(m.valueFrom) && has(m.valueFrom.secretKeyRef) && m.valueFrom.secretKeyRef.name == {{ .passwordSecret | quote }}))
{{- end }}
))
message: databases-ui Jobs connect over verified TLS to a database in their own namespace, or to an external database exactly as the catalog has it
{{- if $urls }}
- expression: >-
variables.env.all(e, !(e.name in ["DBUI_BLOB_URL", "DBUI_AUDIT_URL"]) || (has(e.value) && e.value in [{{ join ", " $urls }}]))
Expand Down
3 changes: 2 additions & 1 deletion charts/databases-ui/values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,8 @@ config:

# The environments (see the README). Rendered into a ConfigMap from `content`, a map here, or from
# `text`, the file as it is (--set-file catalog.text=catalog.yaml); or read from your own ConfigMap
# in the release namespace.
# in the release namespace. An environment's `external` databases need content or text: the Jobs'
# admission policy admits their hosts from the rendered catalog, and refuses them otherwise.
catalog:
existingConfigMap: ""
key: catalog.yaml
Expand Down
11 changes: 11 additions & 0 deletions examples/catalog.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,17 @@ environments:
url: https://examplesnapshots.blob.core.windows.net/dev
# Snapshots tst and acc share with a namespace here can be cloned into it.
cloneFrom: [tst, acc]
# PostgreSQL servers outside CloudNativePG: snapshots and scripts here, restores elsewhere.
# The Jobs log in as user, with the password in passwordSecret (key password) in this
# environment's namespace: the schema's owner, not a superuser.
external:
- name: billing
namespace: dev-billing
host: billing-db.example.internal
database: billing
user: billing_owner
passwordSecret: billing-db
caSecret: billing-db-ca
- name: tst
kubernetes: none
storage:
Expand Down
2 changes: 1 addition & 1 deletion frontend/src/App.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -89,7 +89,7 @@ export function App() {
<div className="card sign-in">
<Logo />
<h1>Databases UI</h1>
<p className="muted">{state === "ended" ? "Your session has ended." : "Sign in with your work account to snapshot, restore and clone the databases of your namespaces."}</p>
<p className="muted">{state === "ended" ? "Your session has ended." : "Sign in with your work account to snapshot, restore and clone the databases of your namespaces, and run scripts on them."}</p>
{authError && (
<p className="notice error" role="alert">
{authError}
Expand Down
21 changes: 17 additions & 4 deletions frontend/src/api.ts
Original file line number Diff line number Diff line change
Expand Up @@ -18,18 +18,23 @@ export type Me = {

export type Database = {
name: string;
// cloudnativepg: a Database resource. external: a server the catalog names.
kind: "cloudnativepg" | "external";
namespace: string;
owner: string;
owner_secret: string;
ready: boolean;
message: string;
server: { host: string; port: number; database: string; tls: string } | null;
workloads: string[];
copy: { name: string; ready: boolean } | null;
can_snapshot: boolean;
can_restore: boolean;
can_script: boolean;
};

export type User = { name: string; managed: boolean; login: boolean; secret: string; databases: string[]; namespaces: string[] };
// `server` is host:port/database for an external database's login, empty for a managed role.
export type User = { name: string; managed: boolean; login: boolean; secret: string; databases: string[]; namespaces: string[]; server: string };

export type Snapshot = {
environment: string;
Expand All @@ -47,7 +52,7 @@ export type JobState = "waiting" | "starting" | "running" | "succeeded" | "faile

export type Job = {
name: string;
action: "snapshot" | "restore";
action: "snapshot" | "restore" | "script";
database: string;
namespace: string;
state: JobState;
Expand All @@ -63,17 +68,24 @@ export type Job = {
into: "" | "database" | "copy";
target_database: string;
approval: string;
// A script's: check or run, its id, its sha256 and the script itself, as the Job holds it.
mode: "" | "check" | "run";
script: string;
script_sha256: string;
script_text: string;
started: string;
finished: string;
result: { result?: string; bytes?: number; seconds?: number; reason?: string };
result: { result?: string; bytes?: number; seconds?: number; reason?: string; mode?: string; output?: string };
can_approve: boolean;
can_decline: boolean;
};

export type AuditEntry = { at: string; environment: string; actor: string; action: string; database?: string; namespace?: string; job?: string; snapshot?: string; target?: string; detail?: string };
export type AuditEntry = { at: string; environment: string; actor: string; action: string; database?: string; namespace?: string; job?: string; snapshot?: string; script?: string; target?: string; detail?: string };

export type RestoreRequest = { source_environment: string; source_database: string; snapshot: string; into: "database" | "copy"; reason: string };

export type ScriptRequest = { script: string; mode: "check" | "run"; reason: string; snapshot: boolean };

export class ApiError extends Error {
constructor(
readonly status: number,
Expand Down Expand Up @@ -128,6 +140,7 @@ export const api = {
snapshots: (e: string, d: string) => request<{ snapshots: Snapshot[] }>("GET", `${db(e, d)}/snapshots`).then((r) => r.snapshots),
snapshot: (e: string, d: string, note: string) => request<Job>("POST", `${db(e, d)}/snapshots`, { note }),
restore: (e: string, d: string, body: RestoreRequest) => request<Job>("POST", `${db(e, d)}/restore`, body),
script: (e: string, d: string, body: ScriptRequest) => request<Job>("POST", `${db(e, d)}/scripts`, body),
shared: (e: string, d: string) => request<{ snapshots: Snapshot[] }>("GET", `${db(e, d)}/shared`).then((r) => r.snapshots),
share: (e: string, d: string, id: string, target: { environment: string; namespace: string }) =>
request<Snapshot>("POST", `${db(e, d)}/snapshots/${encodeURIComponent(id)}/share`, target),
Expand Down
Loading
Loading