Benchmarks Website V3: Admin and Auto-Deploy by connortsui20 · Pull Request #7849 · vortex-data/vortex

connortsui20 · 2026-05-08T18:47:25Z

Summary

This is the (hopefully) the last PR for the new benchmarks website, and you can see the prototype here: http://ec2-18-219-54-101.us-east-2.compute.amazonaws.com:3000/

With this PR, an admin can simply git clone the vortex repository on an EC2 instance, set the INGEST_BEARER_TOKEN and ADMIN_BEARER_TOKEN accordingly (these are set as GitHub secrets in the repository), and then run just a few commands

Here are some of the features of the server itself:

Every 60s, vortex-bench-deploy.timer polls origin/$DEPLOY_BRANCH, rebuilds + atomically swaps the binary if website code changed, verifies /health, and rolls back on failure.
Every hour, vortex-bench-backup.timer calls a new POST /api/admin/snapshot endpoint (CSV EXPORT DATABASE), tar czfs the result, uploads the backup s3://vortex-benchmark-results-database/v3-backups/<ts>.tar.gz.
POST /api/admin/sql runs read-only queries on the backing database.

Admin endpoints are gated by a separate ADMIN_BEARER_TOKEN, mounted only when the env var is set.

Quick start for a new admin

There are only a handful of steps. The full walkthrough is in benchmarks-website/ops/README.md.

On the AWS console (I already did this):

IAM → Policies → create VortexBenchV3Backups from the JSON in the runbook → IAM → Roles → create VortexBenchServerRole (EC2 trust) → attach to the instance.
S3 → vortex-benchmark-results-database → Management → add lifecycle rule expiring v3-backups/ after 7 days.

On the EC2 instance:

cd ~ && git clone https://github.com/vortex-data/vortex.git && cd vortex
./benchmarks-website/ops/install.sh

sudo $EDITOR /etc/vortex-bench.env                 # set INGEST_BEARER_TOKEN + ADMIN_BEARER_TOKEN

sudo systemctl start vortex-bench-deploy.service   # fire the first build now
journalctl -fu vortex-bench-deploy.service         # watch it (Ctrl+C when "deploy ok")

# Run migration from the old data.json.gz
/var/lib/vortex-bench/ops/migrate.sh run --output /var/lib/vortex-bench/bench.duckdb

sudo systemctl start vortex-bench-backup.service   # start the backup loop

AdamGS · 2026-05-08T18:52:50Z

@@ -0,0 +1,22 @@
+# SPDX-License-Identifier: Apache-2.0


vortex-k8s @robert3005

AdamGS · 2026-05-08T19:05:27Z

    let bearer_token =
        env::var("INGEST_BEARER_TOKEN").context("INGEST_BEARER_TOKEN env var must be set")?;
+    let admin_bearer_token = env::var("ADMIN_BEARER_TOKEN").ok();
    let bind_addr = env::var("VORTEX_BENCH_BIND").unwrap_or_else(|_| "127.0.0.1:3000".to_string());


nit - make port configurable? can default to whatever but respecting PORT is usually useful

AdamGS · 2026-05-08T19:06:04Z

+        snapshot_dir = %snapshot_dir.display(),
        "bench server listening"
    );
    axum::serve(listener, app).await?;


realized I commented on tests before - but having ctrl-c handling here is a good practice, gotta make sure it actually shuts down.

I just use pkill directly on the server, and if the admin wants to have a server that they can SIGINT, then they might as well just do cargo run vortex-bench-server?

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>

Replaces the ad-hoc SSH-and-`nohup` deploy of the v3 benchmarks site with a systemd timer that polls origin/develop every 60s, builds and atomically swaps the binary, and verifies /health. Adds an hourly gzipped-snapshot timer and two server-side admin endpoints so backups and ad-hoc reads no longer need to stop the server. Two new routes mounted only when ADMIN_BEARER_TOKEN is set: - POST /api/admin/snapshot?ts=<id>: runs `EXPORT DATABASE … (FORMAT csv)` against the live DuckDB connection, into a fresh subdirectory under AppState::snapshot_dir. ts must match [A-Za-z0-9_-]{1,64}. CSV is the only EXPORT format that ships with libduckdb-sys's `bundled` feature; flipping to parquet or a Vortex layout later is a one-line change. - POST /api/admin/sql {sql, ?format=json|table}: runs read-only SQL (SELECT/WITH/PRAGMA/SHOW/DESCRIBE/EXPLAIN, anything else 403) and renders either JSON or a duckdb-cli-style ASCII table. Uses the same connection mutex as ingest, so a slow SELECT briefly delays writes. Auth is independent of the ingest token (separate ADMIN_BEARER_TOKEN env var) so the two rotate separately. Both use constant-time eq. Everything an EC2 host needs lives under benchmarks-website/ops/: - install.sh: idempotent one-time bootstrap (state dirs under /var/lib/vortex-bench, sudoers fragment, env-file template, systemd units, enable + start the timers). Recommended first-time path is "wait for the deploy timer to build, then run migrate.sh"; preserving an existing $HOME/bench.duckdb is documented as a side note. - deploy.sh: called by vortex-bench-deploy.timer every 60s. Cheap fast path (sha == stamp → exit 0). Path filter on benchmarks-website/{server, migrate}, Cargo.toml, Cargo.lock — vortex-array PRs fast-forward the working tree but skip the rebuild. Atomic versioned-binary symlink swap, sudo systemctl restart, /health verification with rollback to previous binary on failure, stamp updated only on success so failures retry on next tick. Keeps last KEEP_BINARIES (default 3) versions. - migrate.sh: stops server, snapshots current DB to bench.prev-<ts>.duckdb, passes args through to `cargo run -p vortex-bench-migrate --`, restarts. - backup.sh: hourly. Calls /api/admin/snapshot, `tar czf`s the CSV directory into <ts>.tar.gz (gzip reclaims ~5–7× on this shape since most data lands in BIGINT[] runtime arrays serialised as text), uploads with `aws s3 cp` to s3://vortex-ci-benchmark-results/v3-backups/, and cleans up both local copies. Logs the compression ratio so a future regression shows up in `journalctl -u vortex-bench-backup`. - inspect.sh: thin wrapper around /api/admin/sql, no server stop. - systemd/ units: server (Type=simple, Restart=on-failure, hardening via ProtectSystem=strict), deploy oneshot + 60s timer, backup oneshot + hourly timer (Persistent=true so a missed hour catches up after reboot). A symlink at /var/lib/vortex-bench/ops -> .../benchmarks-website/ops keeps the systemd ExecStart paths stable as the repo location changes. - ops/README.md: full operator runbook — first-time install, day-to-day ops (push to develop → live in 60s, monitor a deploy, force a deploy, re-run migration, ad-hoc SQL, backup/restore, token rotation), and failure modes (deploy retry loop, /health stuck, disk filling up, backup not running, host migration). Targeted at a fresh admin who has never seen the system before. - benchmarks-website/README.md, benchmarks-website/AGENTS.md: updated to point at ops/ and to reflect the systemd-based deploy. - server/src/{lib.rs, app.rs, main.rs, admin.rs}: module map, route table, env-var list, and admin module doc all updated. The previous v3 docker artifacts are removed: - benchmarks-website/ec2-init.txt: replaced by ops/README.md. - benchmarks-website/server/Dockerfile: v3 isn't containerised any more. - benchmarks-website/server/scripts/backup.sh: replaced by ops/backup.sh. The v2 React/Vite stack is untouched. docker-compose.yml is left in place; its v3 service entry is now orphaned but harmless and the v2 service is unaffected. server/tests/admin.rs (9 tests): - SQL round-trip (JSON + ASCII table format) - Read-only allow-list (DELETE/UPDATE/DROP/INSERT/CREATE/ATTACH → 403) - Allowed verbs (PRAGMA/SHOW/DESCRIBE/EXPLAIN/WITH) - Bearer enforcement: missing/wrong/ingest-token-on-admin → 401 - Admin router not mounted when ADMIN_BEARER_TOKEN unset → 404 - Snapshot creates the export dir + schema.sql - Snapshot of an existing dir → 409 - ts validation: empty / "../oops" / "with space" / 65 chars → 400 cargo test -p vortex-bench-server passes (admin: 9, rest: 18 pre-existing). cargo clippy -p vortex-bench-server --all-targets --all-features clean. Signed-off-by: Claude <noreply@anthropic.com> Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>

codspeed-hq · 2026-05-08T21:20:57Z

Merging this PR will improve performance by 14.02%

⚠️

Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚡ 2 improved benchmarks
✅ 1206 untouched benchmarks

Performance Changes

	Mode	Benchmark	`BASE`	`HEAD`	Efficiency
⚡	Simulation	`take_map[(0.1, 0.5)]`	1,111.5 µs	974.8 µs	+14.02%
⚡	Simulation	`take_map[(0.1, 1.0)]`	1.8 ms	1.7 ms	+10.72%

_{Comparing ct/bench-v3-autopilot (3bc4736) with develop (ff12040)}

Switches /api/admin/snapshot from `EXPORT DATABASE … (FORMAT csv)` to per-table `COPY (SELECT * FROM <table>) TO … (FORMAT vortex)`. Dogfoods the project's own format and compresses an order of magnitude better than gzipped CSV on this shape (BIGINT[] runtime arrays + short strings). - schema.rs grows a `pub const TABLES: &[&str]` so the snapshot loop has a stable list to iterate. - admin::snapshot writes `schema.sql` from `SCHEMA_DDL` verbatim, then `INSTALL vortex FROM community; LOAD vortex;` (idempotent — autoload is enabled in the bundled libduckdb-sys), then one COPY per table. - ops/README.md restore section rewritten: untar → `.read schema.sql` → `INSERT … SELECT * FROM read_vortex(<file>)` per table. - Two snapshot tests are marked `#[ignore]` because they need outbound network to fetch the vortex extension. Run them by hand before merge: cargo test -p vortex-bench-server --test admin -- --ignored Signed-off-by: Claude <noreply@anthropic.com>

…docs - build.rs runs `git rev-parse --short=12 HEAD` at compile time and exports VORTEX_BENCH_BUILD_SHA, with rerun-if-changed on .git/HEAD and .git/refs/heads so the SHA stays fresh across deploys. Falls back to "unknown" outside a git checkout. - HealthResponse grows a `build_sha` field (`&'static str`) and the /health handler populates it from env!(). Lets an operator run `curl /health | jq .build_sha` and compare against `cat /var/lib/vortex-bench/last-deployed-sha` to confirm the live process is the one the deploy timer last rolled out. - ops/README.md grows two sections under day-to-day: - "Which build is actually running?" — three identifiers, increasing levels of certainty. - "How do I manually rebuild and restart, outside the timer?" — three knobs (restart-only, force-deploy, manual checkout-and-build with the timer paused). Signed-off-by: Claude <noreply@anthropic.com>

connortsui20 requested review from AdamGS, lwwmanning and palaska May 8, 2026 18:47

connortsui20 added the changelog/feature A new feature label May 8, 2026

AdamGS reviewed May 8, 2026

View reviewed changes

connortsui20 and others added 2 commits May 8, 2026 17:13

use many connections on the benchmarks server

c00f12f

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>

connortsui20 force-pushed the ct/bench-v3-autopilot branch from 1477314 to 07656cc Compare May 8, 2026 21:13

claude added 2 commits May 8, 2026 17:38

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Benchmarks Website V3: Admin and Auto-Deploy#7849

Benchmarks Website V3: Admin and Auto-Deploy#7849
connortsui20 wants to merge 4 commits intodevelopfrom
ct/bench-v3-autopilot

connortsui20 commented May 8, 2026 •

edited

Loading

Uh oh!

AdamGS May 8, 2026

Uh oh!

AdamGS May 8, 2026

Uh oh!

AdamGS May 8, 2026

Uh oh!

connortsui20 May 8, 2026

Uh oh!

codspeed-hq Bot commented May 8, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

connortsui20 commented May 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Quick start for a new admin

Uh oh!

AdamGS May 8, 2026

Choose a reason for hiding this comment

Uh oh!

AdamGS May 8, 2026

Choose a reason for hiding this comment

Uh oh!

AdamGS May 8, 2026

Choose a reason for hiding this comment

Uh oh!

connortsui20 May 8, 2026

Choose a reason for hiding this comment

Uh oh!

codspeed-hq Bot commented May 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will improve performance by 14.02%

Performance Changes

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

connortsui20 commented May 8, 2026 •

edited

Loading

codspeed-hq Bot commented May 8, 2026 •

edited

Loading