Skip to content

retro: record first-night verification #429

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
espadonne merged 1 commit into from
Sep 3, 2026
Merged

retro: record first-night verification #429

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 11 additions & 1 deletion docs/internal/retro/2026-09-02-availability-sitrep.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -455,7 +455,17 @@ ssh root@shithub.sh '
# expect: pprof listener started (loopback only) addr=127.0.0.1:6060
```

### 10. Phase 0 verification, still open
### 10. Phase 0 verification — DONE 2026-09-03 06:47 UTC

First full night after the campaign: OOM kills 0; overnight peak
`%memused` 20.9% (was 88.2% on 2026-09-02); swap used 42 MB of 4 GB;
postgres/web/worker active; `backup-daily` ok at 03:17:45 with
heartbeat; spaces-sync ok at 02:19 (01:23 run, ~56 min without
`--fast-list`); AIDE check 06:00–06:29; DO uptime probe 98/98 `200`
over two hours; 429s are crawler-only. DR check on the backups
bucket: 120 objects match, 1 difference = the 03:17 dump not yet
mirrored by the 07:23 sync. Traffic tables 717k/770k rows after the
first purge (DB 1057 MB; space is reclaimed by autovacuum over time).

```sh
ssh root@shithub.sh '
Expand Down
Loading