diff --git a/cmd/stellar-rpc/scripts/bench-fullhistory/README.md b/cmd/stellar-rpc/scripts/bench-fullhistory/README.md index 37a2c1cf9..a4c8024c3 100644 --- a/cmd/stellar-rpc/scripts/bench-fullhistory/README.md +++ b/cmd/stellar-rpc/scripts/bench-fullhistory/README.md @@ -121,9 +121,13 @@ Shared flags: | flag | meaning | |---|---| | `--types=ledgers,txhash,events` | which data types to ingest (any subset; required) | -| `--source=pack\|bsb` | `pack` reads a local cold packfile; `bsb` reads from a GCS `BufferedStorageBackend` | +| `--source=pack\|bsb\|lcm` | `pack` reads a local cold packfile; `bsb` reads from a GCS `BufferedStorageBackend`; `lcm` reads a framed `LedgerCloseMeta` file from stellar-core `apply-load` (see [Synthetic ledgers](#synthetic-ledgers-via-apply-load)) | | `--cold-dir=DIR` | source cold-store dir (required for `--source=pack`) | | `--bucket-path=...` | GCS `destination_bucket_path` (for `--source=bsb`); ADC credentials required | +| `--lcm-file=FILE` | apply-load `meta.xdr` (required for `--source=lcm`) | +| `--lcm-checkpoint=N` | skip leading ledgers with seq ≤ N (apply-load setup ledgers; for `--source=lcm`) | +| `--lcm-fix-tx-hashes` | repair apply-load's tx-hash/envelope mismatch so the roundtrip reader can consume the meta (default `true`; `--source=lcm`) | +| `--lcm-allow-partial` | allow a short final chunk when the run was sized below 10k ledgers (default `true`; `--source=lcm`) | | `--bsb-buffer-size`, `--bsb-num-workers` | BSB prefetch tuning | | `--chunk=N` | first chunk ID to ingest (required) | | `--xdr-views` | extract via zero-copy XDR views instead of `UnmarshalBinary` + struct walk | @@ -192,6 +196,80 @@ bench-fullhistory cold-ingest --types=txhash --source=pack \ bench-fullhistory build-txhash-index --in-dir=/path/to/out/cold/txhash ``` +## Synthetic ledgers via `apply-load` + +When you don't have (or don't want) real pubnet chunks, you can generate +**fully synthetic, density-controlled** packfiles with stellar-core's +`apply-load` command. `apply-load-gen.sh` drives the whole pipeline: + +``` +apply-load → meta.xdr (framed LedgerCloseMeta) → cold-ingest --source=lcm → packfiles → build-txhash-index +``` + +```sh +# A small SAC run is enough to exercise the read benches: TPS is set by +# per-ledger DENSITY, not ledger count, so a few hundred ledgers already hit +# the profile's target throughput. +CORE_BIN=/path/to/stellar-core PROFILE=sac NUM_LEDGERS=300 \ + ./apply-load-gen.sh +``` + +**Workload profiles** (`PROFILE=`) map to apply-load's model transactions and +target throughputs. TPS = txs-per-ledger ÷ block-time, and the target is taken at +the network's **600 ms block time** (`CLOSE_TIME_MS` default), so the per-ledger +tx count = `TPS × 0.6`: + +| `PROFILE` | model tx (`APPLY_LOAD_MODEL_TX`) | target | txs/ledger @600ms | +|---|---|---|---| +| `sac` | `sac` (Stellar Asset Contract transfer) | ~10k SAC TPS | 6,000 | +| `token` (`oz`) | `custom_token` (OpenZeppelin-style token) | ~9k OZ TPS | 5,400 | +| `soroswap` | `soroswap` (AMM swap, real mainnet wasm) | ~2.5k TPS | 1,500 | + +> The ledger header `closeTime` is whole **seconds** in XDR, so a 600 ms block +> cadence can't be a timestamp — it's modeled purely by per-ledger density. + +Key env knobs: `NUM_LEDGERS` (total ledgers to generate; **prefer this for a +quick run** — the final chunk may be partial), `CHUNKS` (10k-ledger chunks to +fill, default 16; ignored when `NUM_LEDGERS` is set), `CLOSE_TIME_MS` (block +time for the TPS math, default 600), `TXS_PER_LEDGER` (override the derived +density), `CLUSTERS` (`APPLY_LOAD_LEDGER_MAX_DEPENDENT_TX_CLUSTERS` — parallel +apply threads; **generation-speed only**, default 8, don't exceed 8), +`TYPES`, `CHUNK_WORKERS`, `OUT_ROOT`, `KEEP_META`, `BENCH_BIN`. + +**Requirements & caveats:** + +- Needs a stellar-core built with **`BUILD_TESTS`** (the CI build tagged + `…~buildtests`) — `apply-load` + `ARTIFICIALLY_GENERATE_LOAD_FOR_TESTING` + are test-only. +- **Cost scales with density, not just count.** apply-load close time grows with + txs/ledger and accumulated state: `sac` (1 fat batched tx/ledger) runs at + ~0.1 s/ledger, but `token`/`soroswap` apply ~9k txs/ledger at ~9 s/ledger and + rising. A full 10k-ledger chunk of dense Soroban load is **hours to days** — + so size dense profiles with a small `NUM_LEDGERS` (a few hundred), which still + meets the TPS target. +- **apply-load tx-hash fixup (automatic).** `apply-load`'s streamed meta records + the same transactions in the tx-set and in `TxProcessing`, but the stored + result hash does **not** equal the envelope's real hash, so the go-stellar-sdk + ingest `LedgerTransactionReader` (which pairs envelope↔result by hash) rejects + it with *"unknown tx hash in LedgerCloseMeta"* — breaking the roundtrip + tx-page / tx-hash benches. `cold-ingest --source=lcm` repairs this by default + (`--lcm-fix-tx-hashes`): it pairs each result to its envelope via the + fee-charged account and stamps the correct hash. See `lcm_fixup.go`. +- The `lcm` source assigns ledger sequences **positionally** per chunk (chunk 1 + → seqs 10002…20001, etc.), skipping apply-load setup ledgers + (`--lcm-checkpoint`). The final chunk may be **partial** when the run was + sized below a full chunk (`--lcm-allow-partial`, on by default); the read + benches clamp their cursors to each chunk's actual ledger range. +- **`cold-events` works for `sac` and `soroswap`, not `token`.** The corpus + builder needs enough unique *terms* (contract anchors + topic values) to fill + the K-bucket sweep (≥ max K, default 15) — it does **not** require 3 distinct + contracts. `sac` (one SAC contract whose `transfer` events vary `from`/`to` + over thousands of accounts) and `soroswap` (router + pair contracts) both + reach 15 terms from a single/few contracts. `token` (`custom_token`) emits + events that are not 4-topic, so it yields no usable terms — use `sac` or + `soroswap` for event benches. `cold-ledgers`/`cold-txpage`/`cold-txhash` work + for all profiles. + ## Interpreting ingest output - **`total wall`** — end-to-end wall time. For multi-chunk cold runs it is @@ -210,7 +288,8 @@ bench-fullhistory build-txhash-index --in-dir=/path/to/out/cold/txhash - `bench_concurrent_runner.go`, `bench_grid.go` — the `--query-concurrency` sweep scaffolding. - `bench_{hot,cold}_ingest.go` — ingest drivers. - `ingest_{ledgers,txhash,events}.go` — per-type ingesters + collectors. -- `ingester.go`, `ledger.go`, `extract_{views,parsed}.go`, `sources.go` — ingest plumbing. +- `ingester.go`, `ledger.go`, `extract_{views,parsed}.go`, `sources.go` — ingest plumbing (`sources.go` has the `pack`/`bsb`/`lcm` ledger sources). +- `apply-load-gen.sh` — synthetic-ledger driver: stellar-core `apply-load` → `meta.xdr` → packfiles. - `bench_build_txhash_index.go`, `streamhash_merge.go` — phase-2 index build. - `corpus.go`, `cache*.go`, `tx_hash_helpers.go`, `metrics_helpers.go` — shared helpers. - `run-all-benches.sh` — suite driver (builds once, runs every read + ingest diff --git a/cmd/stellar-rpc/scripts/bench-fullhistory/SYNTHETIC-LEDGERS.md b/cmd/stellar-rpc/scripts/bench-fullhistory/SYNTHETIC-LEDGERS.md new file mode 100644 index 000000000..cfaf30619 --- /dev/null +++ b/cmd/stellar-rpc/scripts/bench-fullhistory/SYNTHETIC-LEDGERS.md @@ -0,0 +1,129 @@ +# Synthetic-ledger generation + benchmarking — runbook + +End-to-end recipe to generate controllable synthetic full-history datasets with +stellar-core `apply-load` and run the read bench suite on them. This is the +hands-off path: prepare the host once, then `synthetic-run.sh` does +generate → bench → (optional) upload for every profile. + +Scripts (all in this directory): +- `apply-load-gen.sh` — generate ONE profile (apply-load → meta.xdr → cold packfiles + tx-hash index) +- `bench-suite.sh` — run the cold/hot read benches against generated stores +- `synthetic-run.sh` — orchestrator: loop profiles → generate → bench → optional GCS upload + +## 1. Host prerequisites + +### a) stellar-core with `apply-load` (BUILD_TESTS) +Released/Docker cores **strip** `apply-load`. Install the `~buildtests` build from +SDF's unstable apt channel (Ubuntu 24.04 / noble shown): + +```sh +sudo wget -qO /etc/apt/keyrings/SDF.asc https://apt.stellar.org/SDF.asc +echo "deb [signed-by=/etc/apt/keyrings/SDF.asc] https://apt.stellar.org noble unstable" \ + | sudo tee /etc/apt/sources.list.d/SDF-unstable.list +sudo apt-get update +apt-cache madison stellar-core | grep buildtests # pick newest, protocol you want +sudo apt-get install -y stellar-core= # pin: it sorts below stable +stellar-core apply-load --help # must succeed +``` + +### b) Go + RocksDB (to build the `bench-fullhistory` binary) +The bench binary uses cgo against RocksDB v10 (grocksdb v1.10.x). The system +`librocksdb` (8.x) is too old. + +```sh +# Go: match go.mod's toolchain (1.26 at time of writing) — e.g. /usr/local/go +# RocksDB v10.9.1 (shared lib + headers): +PREFIX=$HOME/.rocksdb ./scripts/install-rocksdb.sh # repo root script + +export CGO_CFLAGS="-I$HOME/.rocksdb/include" +export CGO_LDFLAGS="-L$HOME/.rocksdb/lib -lrocksdb" +export LD_LIBRARY_PATH="$HOME/.rocksdb/lib" # needed at RUN time too +``` + +### c) Disk + RAM — the two real constraints +- **Disk:** use a fast **local** volume (NVMe instance store, not network EBS) for + `OUT_ROOT`. The transient `meta.xdr` is large (a 10k-ledger SAC chunk ≈ ~100+ GB + before it's deleted post-ingest). Budget hundreds of GB free. +- **RAM — this caps how many ledgers you can generate.** Each dense apply-load holds + in-memory soroban state that **grows with ledger count**. Measured: SAC at + 6000 tx/ledger ≈ **8.5 MB/ledger** → ~32 GB at 3,760 ledgers, ~85 GB at 10,000. + + | box RAM | sac/token (6000/5400 tx/ledger) | soroswap (1500 tx/ledger) | + |---|---|---| + | 61 GB (c6id.8xlarge) | ~6,000 ledgers | ~20,000 (2 chunks) | + | 128 GB | ~14,000 | full chunks easily | + | 256 GB | ~28,000 (≈3 chunks) | many chunks | + + **A full 10k-ledger chunk of 10k-TPS SAC needs ~96–128 GB RAM.** If a run exceeds + RAM the kernel OOM-kills apply-load mid-generation. Size `NUM_LEDGERS` to the box. + +## 2. Profiles and the TPS model + +`MODEL_TX` + per-ledger density define the workload. TPS is taken at a **600 ms** +block time (`CLOSE_TIME_MS`), so per-ledger tx count = `TPS × 0.6`: + +| PROFILE | model tx | target | tx/ledger @600ms | +|---|---|---|---| +| `sac` | SAC transfer | 10,000 TPS | 6,000 | +| `token` (`oz`) | custom_token | 9,000 TPS | 5,400 | +| `soroswap` | AMM swap | 2,500 TPS | 1,500 | + +Notes baked into the scripts: +- `BATCH_SAC=1` so each transfer is its own tx (pack tx-density == TPS target). +- `CLUSTERS=8` (`APPLY_LOAD_LEDGER_MAX_DEPENDENT_TX_CLUSTERS`) — generation-speed + only; don't exceed 8 (known multi-threaded-apply perf issues above that). +- `HTTP_PORT=0` so parallel generations don't collide on core's HTTP port. +- The streamed meta needs a **tx-hash fixup** (cold-ingest does it by default, + `--lcm-fix-tx-hashes`) or the roundtrip txpage/txhash benches reject it; the + passphrase is pubnet to match the bench binary. (Details in this dir's README.) +- `cold-events`/`hot-events` work for `sac` and `soroswap`; **not** `token` + (custom_token events aren't 4-topic). + +## 3. Run it + +```sh +cd cmd/stellar-rpc/scripts/bench-fullhistory + +# env from §1b (CGO_*, LD_LIBRARY_PATH) must be exported in this shell. +CORE_BIN=/usr/bin/stellar-core \ +OUT_ROOT=/mnt/nvme/synth \ +PROFILES="sac token soroswap" \ +NUM_LEDGERS=6000 \ # size to your RAM (see §1c) +PARALLEL=0 \ # sequential (safe); 1 only if combined RSS fits RAM +GCS_DEST=gs://rpc-full-history/synthetic-ledgers/ \ # optional upload + ./synthetic-run.sh +``` + +For a long unattended run, detach it: +```sh +setsid nohup env CORE_BIN=… OUT_ROOT=… NUM_LEDGERS=… ./synthetic-run.sh > run.out 2>&1 < /dev/null & +``` + +`soroswap` reaches full 10k chunks on modest RAM, so a common split is: +`NUM_LEDGERS=20000 PROFILES=soroswap` (2 chunks) plus +`NUM_LEDGERS= PROFILES="sac token"`. + +## 4. Outputs + +``` +$OUT_ROOT//cold/{ledgers/00000/*.pack, txhash.idx, events/00000/*} +$OUT_ROOT//work/apply-load.cfg # exact config (reproducibility input) +$OUT_ROOT/bench-results/run-//*.csv # latency/throughput sweeps +``` + +Point the read benches at a cold store directly, e.g.: +```sh +LD_LIBRARY_PATH=$HOME/.rocksdb/lib ./bench-fullhistory cold-txpage \ + --cold-dir=$OUT_ROOT/sac/cold/ledgers --page-size=20 --iters=200 \ + --query-concurrency=1,4,8,16 --xdr-views --out=results-sac +``` + +## 5. Reproducibility caveat + +The **config + genesis are deterministic** (same root account each run), but the +**transactions are not byte-reproducible**: stellar-core seeds its RNG from +wall-clock time and `apply-load` exposes no seed. Runs match in *shape* +(profile/density/op-mix), not bytes. To pin an exact dataset, keep the generated +cold packs (and their SHA256s) — that's the canonical artifact. Uploading to GCS +(`GCS_DEST`) is also how you make NVMe-instance-store output durable (it's wiped +on instance stop/terminate). diff --git a/cmd/stellar-rpc/scripts/bench-fullhistory/apply-load-gen.sh b/cmd/stellar-rpc/scripts/bench-fullhistory/apply-load-gen.sh new file mode 100755 index 000000000..008149697 --- /dev/null +++ b/cmd/stellar-rpc/scripts/bench-fullhistory/apply-load-gen.sh @@ -0,0 +1,261 @@ +#!/usr/bin/env bash +# +# apply-load-gen.sh — generate synthetic full-history packfiles for the +# bench-fullhistory suite using stellar-core's `apply-load`. +# +# Pipeline: +# 1. stellar-core new-db + new-hist + apply-load -> meta.xdr (framed +# LedgerCloseMeta stream of dense synthetic ledgers) +# 2. bench-fullhistory cold-ingest --source=lcm -> cold packfiles +# (ledgers/, txhash/, events/) in the layout the read benches expect +# 3. build-txhash-index -> cold tx-hash MPHF +# +# The cold-* read benches then point --cold-dir at /cold/ledgers (etc). +# +# WORKLOAD PROFILES (model transaction + density). Targets are interpreted at +# the network's 600ms block time (CLOSE_TIME_MS default), so the per-ledger tx +# count = TPS * 0.6: +# profile model tx target TPS txs/ledger @600ms +# sac sac 10,000 6,000 +# token (oz) custom_token 9,000 5,400 +# soroswap soroswap 2,500 1,500 +# +# TPS = txs-per-ledger / block-time. Block time is CLOSE_TIME_MS (default 600). +# The ledger header closeTime is whole SECONDS in XDR, so 600ms blocks cannot be +# represented as timestamps — the cadence is modeled by per-ledger DENSITY only. +# +# NOTE on batching: APPLY_LOAD_BATCH_SAC_COUNT>1 folds N SAC transfers into a +# single InvokeHostFunction tx, so the CLOSED/streamed ledger ends up with +# ~(TXS_PER_LEDGER) txs but only ~TXS_PER_LEDGER batched transfers reach the +# meta as 1 tx each — i.e. the usable pack carries far fewer txs than the TPS +# target (verified: BATCH_SAC=100 streamed 1 tx/ledger). Keep BATCH_SAC=1 so +# every transfer is its own tx and the pack's tx density equals the TPS target. +# +# REQUIREMENTS +# * stellar-core built with BUILD_TESTS (apply-load + ARTIFICIALLY_GENERATE_ +# LOAD_FOR_TESTING). The public CI build is tagged e.g. +# `26.x.y-NNNN..noble~buildtests`. +# * The bench-fullhistory binary (this script builds it if --bench-bin unset). +# +# COST WARNING (real full chunks): each chunk is 10,000 ledgers, and a chunk +# at 9k txs/ledger applies 90M transactions. 16 chunks of dense Soroban load is +# many hours to days of apply time and tens of GB of meta. Start with +# CHUNKS=1 to validate the pipeline before scaling up. +# +set -euo pipefail + +# ---- knobs (env-overridable) ----------------------------------------------- +PROFILE="${PROFILE:-sac}" # sac | token | soroswap +CHUNKS="${CHUNKS:-16}" # number of 10k-ledger chunks to fill +NUM_LEDGERS="${NUM_LEDGERS:-}" # override total ledgers (else CHUNKS*10k). + # Use a small value for a quick run that + # still hits the profile's TPS (TPS is set + # by density, not ledger count). The final + # chunk is then partial (cold-ingest's + # --lcm-allow-partial handles it). +CLOSE_TIME_MS="${CLOSE_TIME_MS:-600}" # modeled block time in ms for TPS math. + # Default 600ms — the network target. The + # ledger header closeTime is whole SECONDS + # in XDR, so sub-second cadence cannot be a + # timestamp; it is modeled purely as + # per-ledger density = TPS * CLOSE_TIME_MS/1000. +TXS_PER_LEDGER="${TXS_PER_LEDGER:-}" # override the profile's per-ledger tx count +CORE_BIN="${CORE_BIN:-$(command -v stellar-core || true)}" +BENCH_BIN="${BENCH_BIN:-}" # prebuilt bench-fullhistory; built if empty +OUT_ROOT="${OUT_ROOT:-./apply-load-out}" # work + output root +TYPES="${TYPES:-ledgers,txhash,events}" # cold-ingest types +CHUNK_WORKERS="${CHUNK_WORKERS:-4}" # cold-ingest chunk concurrency +HTTP_PORT="${HTTP_PORT:-0}" # stellar-core HTTP port. 0 = disabled, which + # apply-load doesn't need and which lets many + # generations run in PARALLEL without colliding + # on the default 11626 (bind: address in use). +CLUSTERS="${CLUSTERS:-8}" # APPLY_LOAD_LEDGER_MAX_DEPENDENT_TX_CLUSTERS: + # parallel apply threads — a GENERATION-SPEED + # knob only (does not change the workload). Cap + # at 8; multi-threaded apply has known perf + # issues above that even on bigger machines. +KEEP_META="${KEEP_META:-0}" # 1 = keep meta.xdr after ingest +# Must match the passphrase the bench binary hardcodes (main.go: pubnetPassphrase). +# The ingest reader recomputes each tx hash from its envelope under this +# passphrase and matches it against the result entries; a mismatch makes the +# roundtrip txpage/txhash paths fail with "unknown tx hash in LedgerCloseMeta". +NETWORK_PASSPHRASE="${NETWORK_PASSPHRASE:-Public Global Stellar Network ; September 2015}" + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +LEDGERS_PER_CHUNK=10000 + +log() { printf '\033[1;34m[apply-load-gen]\033[0m %s\n' "$*" >&2; } +die() { printf '\033[1;31m[apply-load-gen] ERROR:\033[0m %s\n' "$*" >&2; exit 1; } + +# ---- resolve core + verify BUILD_TESTS ------------------------------------- +[ -n "$CORE_BIN" ] || die "stellar-core not found; set CORE_BIN=/path/to/stellar-core (must be a BUILD_TESTS build)" +[ -x "$CORE_BIN" ] || die "CORE_BIN=$CORE_BIN is not executable" +CORE_VER="$("$CORE_BIN" version 2>/dev/null | head -1 || true)" +log "stellar-core: $CORE_BIN ($CORE_VER)" +if ! "$CORE_BIN" apply-load --help >/dev/null 2>&1; then + die "this stellar-core lacks the apply-load command — you need a BUILD_TESTS build (…~buildtests)" +fi + +# ---- per-profile density --------------------------------------------------- +# model_tx, batch_sac_count, target_tps. (Dependent-tx clusters is a generation- +# speed knob, not per-profile — see CLUSTERS above.) +case "$PROFILE" in + sac) MODEL_TX="sac"; BATCH_SAC=1; TARGET_TPS=10000 ;; + token|oz) MODEL_TX="custom_token"; BATCH_SAC=1; TARGET_TPS=9000 ;; + soroswap) MODEL_TX="soroswap"; BATCH_SAC=1; TARGET_TPS=2500 ;; + *) die "unknown PROFILE=$PROFILE (expected sac|token|soroswap)" ;; +esac + +# txs-per-ledger so that (txs * batch) / (close_time_ms/1000) == target_tps, +# i.e. txs = target_tps * close_time_ms / 1000 / batch (ceil division). +if [ -z "$TXS_PER_LEDGER" ]; then + TXS_PER_LEDGER=$(( (TARGET_TPS * CLOSE_TIME_MS + 1000 * BATCH_SAC - 1) / (1000 * BATCH_SAC) )) +fi +# NUM_LEDGERS override: when set, it drives generation directly and CHUNKS is +# derived as the number of (10k-ledger) chunks needed to cover it (the last is +# partial). Otherwise NUM_LEDGERS = CHUNKS full chunks. +if [ -n "$NUM_LEDGERS" ]; then + CHUNKS=$(( (NUM_LEDGERS + LEDGERS_PER_CHUNK - 1) / LEDGERS_PER_CHUNK )) + [ "$CHUNKS" -lt 1 ] && CHUNKS=1 +else + NUM_LEDGERS=$(( CHUNKS * LEDGERS_PER_CHUNK )) +fi +GENESIS_ACCOUNTS=$(( TXS_PER_LEDGER * 2 )) +[ "$GENESIS_ACCOUNTS" -lt 21000 ] && GENESIS_ACCOUNTS=21000 + +log "profile=$PROFILE model_tx=$MODEL_TX txs/ledger=$TXS_PER_LEDGER batch_sac=$BATCH_SAC -> ~$(( TXS_PER_LEDGER * BATCH_SAC * 1000 / CLOSE_TIME_MS )) TPS @ ${CLOSE_TIME_MS}ms blocks" +log "chunks=$CHUNKS num_ledgers=$NUM_LEDGERS (this is the slow part — apply-load closes every ledger)" + +# ---- workspace + config ---------------------------------------------------- +# Absolutize OUT_ROOT: the steps below `cd` into $WORK_DIR before invoking core +# and the bench binary, so any WORK_DIR-relative path (CONF, META, …) would no +# longer resolve from inside it. +mkdir -p "$OUT_ROOT" +OUT_ROOT="$(cd "$OUT_ROOT" && pwd)" +WORK_DIR="$OUT_ROOT/$PROFILE/work" +COLD_OUT="$OUT_ROOT/$PROFILE/cold" +mkdir -p "$WORK_DIR" "$COLD_OUT" +CONF="$WORK_DIR/apply-load.cfg" +META="$WORK_DIR/meta.xdr" + +cat > "$CONF" </dev/null +log "stellar-core new-hist local…" +( cd "$WORK_DIR" && "$CORE_BIN" new-hist local --conf "$CONF" ) >/dev/null +log "stellar-core apply-load… (this is the long-running step)" +APPLY_LOG="$WORK_DIR/apply-load.log" +( cd "$WORK_DIR" && "$CORE_BIN" apply-load --conf "$CONF" ) 2>&1 | tee "$APPLY_LOG" + +[ -s "$META" ] || die "apply-load produced no meta at $META. + Your core may not emit metadata in benchmark mode. Workaround: base the config + on docs/apply-load-for-meta.cfg (APPLY_LOAD_MODE=ledger-limits), which is the + proven meta-emitting path, and re-run with --source=lcm." + +# Pre-benchmark checkpoint: ledgers with seq <= this are setup, not benchmark. +CHECKPOINT="$(grep -oE 'Published final checkpoint before benchmark: ledger [0-9]+' "$APPLY_LOG" \ + | grep -oE '[0-9]+$' | tail -1 || true)" +CHECKPOINT="${CHECKPOINT:-0}" +log "pre-benchmark checkpoint = $CHECKPOINT (skipping ledgers with seq <= $CHECKPOINT)" +log "meta.xdr size: $(du -h "$META" | cut -f1)" + +# ---- build bench binary if needed ------------------------------------------ +if [ -z "$BENCH_BIN" ]; then + BENCH_BIN="$WORK_DIR/bench-fullhistory" + log "building bench-fullhistory -> $BENCH_BIN" + ( cd "$SCRIPT_DIR" && go build -o "$BENCH_BIN" . ) +fi + +# ---- cold-ingest via the lcm source ---------------------------------------- +# --chunk=1 (chunk 0 is reserved/“unset”); baseChunk=1 maps chunk 1 -> block 0. +log "cold-ingest --source=lcm (types=$TYPES, chunks=$CHUNKS)…" +"$BENCH_BIN" cold-ingest \ + --source=lcm \ + --lcm-file="$META" \ + --lcm-checkpoint="$CHECKPOINT" \ + --types="$TYPES" \ + --chunk=1 \ + --num-chunks="$CHUNKS" \ + --chunk-workers="$CHUNK_WORKERS" \ + --cold-out-dir="$COLD_OUT" \ + --xdr-views \ + --out="$WORK_DIR/bench-out" + +# ---- build the cold tx-hash index ------------------------------------------ +if [[ ",$TYPES," == *",txhash,"* ]]; then + log "build-txhash-index…" + "$BENCH_BIN" build-txhash-index \ + --in-dir="$COLD_OUT/txhash" \ + --idx-out="$COLD_OUT/txhash.idx" \ + --out="$WORK_DIR/bench-out" +fi + +[ "$KEEP_META" = "1" ] || { log "removing meta.xdr (set KEEP_META=1 to keep)"; rm -f "$META"; } + +log "DONE. Cold packfiles: $COLD_OUT/{ledgers,txhash,events}" +cat >&2 </cold/{ledgers,txhash.idx,events/00000} +# +# Iter counts are env-overridable so the same script does a quick smoke +# (small *_ITERS) or a full run (defaults below). cold-events is skipped for a +# profile whose events are not 4-topic (apply-load's custom_token); the bench +# itself errors out cleanly otherwise. +# +# Required env: BENCH_BIN (path to the bench-fullhistory binary). On Linux the +# caller must also export LD_LIBRARY_PATH to the RocksDB v10 .so dir. +set -uo pipefail + +BENCH_BIN="${BENCH_BIN:?set BENCH_BIN=/path/to/bench-fullhistory}" +ROOT="${ROOT:?set ROOT=/cold>}" +RESULTS="${RESULTS:-$ROOT/bench-results/run-$(date -u +%Y%m%dT%H%M%SZ)}" +PROFILES="${PROFILES:-sac token soroswap}" +QC="${QC:-1,4,8,16}" +LEDGER_NS="${LEDGER_NS:-1 10 20}" +LEDGERS_ITERS="${LEDGERS_ITERS:-60}" +TXPAGE_ITERS="${TXPAGE_ITERS:-200}" +TXHASH_ITERS="${TXHASH_ITERS:-1000}" +EVENTS_ITERS="${EVENTS_ITERS:-500}" +PAGE_SIZE="${PAGE_SIZE:-20}" +HOT="${HOT:-1}" # 1 = also build a hot store per profile and run hot-* benches +# Profiles whose events are NOT 4-topic (skip cold/hot-events). apply-load's +# custom_token emits non-4-topic events; sac/soroswap are fine. +NO_EVENTS="${NO_EVENTS:-token}" + +mkdir -p "$RESULTS" +echo "bench-suite -> $RESULTS (profiles: $PROFILES)" + +skip_events() { case " $NO_EVENTS " in *" $1 "*) return 0;; *) return 1;; esac; } + +for P in $PROFILES; do + COLD="$ROOT/$P/cold" + if [ ! -d "$COLD/ledgers" ]; then echo "skip $P (no cold store at $COLD)"; continue; fi + O="$RESULTS/$P"; mkdir -p "$O" + echo "================= $P =================" + + # ---- COLD read benches (auto-discover chunk range) ---- + for n in $LEDGER_NS; do + "$BENCH_BIN" cold-ledgers --cold-dir="$COLD/ledgers" --n="$n" --iters="$LEDGERS_ITERS" \ + --query-concurrency="$QC" --out="$O" > "$O/cold-ledgers-n$n.log" 2>&1 || echo " cold-ledgers n=$n FAILED" + done + for mode in "" "--xdr-views"; do + tag=$([ -z "$mode" ] && echo roundtrip || echo xdrviews) + "$BENCH_BIN" cold-txpage --cold-dir="$COLD/ledgers" --page-size="$PAGE_SIZE" --iters="$TXPAGE_ITERS" \ + --query-concurrency="$QC" $mode --out="$O" > "$O/cold-txpage-$tag.log" 2>&1 || echo " cold-txpage $tag FAILED" + "$BENCH_BIN" cold-txhash --cold-dir="$COLD/ledgers" --txhash-cold-mphf="$COLD/txhash.idx" --iters="$TXHASH_ITERS" \ + --query-concurrency="$QC" $mode --out="$O" > "$O/cold-txhash-$tag.log" 2>&1 || echo " cold-txhash $tag FAILED" + done + if ! skip_events "$P"; then + "$BENCH_BIN" cold-events --cold-events-dir="$COLD/events/00000" --iters="$EVENTS_ITERS" \ + --query-concurrency="$QC" --out="$O" > "$O/cold-events.log" 2>&1 || echo " cold-events FAILED" + fi + + # ---- HOT: build a hot store from the cold pack (chunk 1), then hot reads ---- + if [ "$HOT" = "1" ]; then + H="$O/hot" + "$BENCH_BIN" hot-ingest --types=ledgers,txhash,events --source=pack --cold-dir="$COLD/ledgers" \ + --chunk=1 --hot-dir="$H" --out="$O" > "$O/hot-ingest.log" 2>&1 || echo " hot-ingest FAILED (skipping hot reads)" + if [ -d "$H/ledgers" ]; then + for n in $LEDGER_NS; do + "$BENCH_BIN" hot-ledgers --hot-dir="$H/ledgers" --chunk=1 --n="$n" --iters="$LEDGERS_ITERS" \ + --query-concurrency="$QC" --out="$O" > "$O/hot-ledgers-n$n.log" 2>&1 || echo " hot-ledgers n=$n FAILED" + done + for mode in "" "--xdr-views"; do + tag=$([ -z "$mode" ] && echo roundtrip || echo xdrviews) + "$BENCH_BIN" hot-txpage --hot-dir="$H/ledgers" --chunk=1 --page-size="$PAGE_SIZE" --iters="$TXPAGE_ITERS" \ + --query-concurrency="$QC" $mode --out="$O" > "$O/hot-txpage-$tag.log" 2>&1 || echo " hot-txpage $tag FAILED" + "$BENCH_BIN" hot-txhash --hot-dir="$H/ledgers" --txhash-hot="$H/txhash" --cold-dir="$COLD/ledgers" --chunk=1 \ + --iters="$TXHASH_ITERS" --query-concurrency="$QC" $mode --out="$O" > "$O/hot-txhash-$tag.log" 2>&1 || echo " hot-txhash $tag FAILED" + done + if ! skip_events "$P"; then + "$BENCH_BIN" hot-events --hot-dir="$H/events" --chunk=1 --iters="$EVENTS_ITERS" \ + --query-concurrency="$QC" --out="$O" > "$O/hot-events.log" 2>&1 || echo " hot-events FAILED" + fi + fi + fi + echo " $P done -> $O" +done +echo "ALL BENCHES DONE -> $RESULTS" diff --git a/cmd/stellar-rpc/scripts/bench-fullhistory/bench_cold_ingest.go b/cmd/stellar-rpc/scripts/bench-fullhistory/bench_cold_ingest.go index 9ea470fd4..857e2f040 100644 --- a/cmd/stellar-rpc/scripts/bench-fullhistory/bench_cold_ingest.go +++ b/cmd/stellar-rpc/scripts/bench-fullhistory/bench_cold_ingest.go @@ -50,9 +50,11 @@ func cmdColdIngest() int { // worker too (a pack reader or a BSB session), so nothing here is shared // across workers — BSB's single sequential cursor cannot be. type coldDeps struct { + logger *supportlog.Entry source string coldDir string bucketPath string + lcm lcmOpts startChunk chunk.ID numChunks int chunkWorkers int @@ -73,10 +75,18 @@ type coldDeps struct { func buildColdDeps(logger *supportlog.Entry) (context.Context, coldDeps, func(), error) { fs := flag.NewFlagSet("cold-ingest", flag.ExitOnError) typesArg := fs.String("types", "", "comma-separated subset of ledgers,events,txhash (required)") - source := fs.String("source", sourcePack, "ledger source: pack | bsb") + source := fs.String("source", sourcePack, "ledger source: pack | bsb | lcm") coldDir := fs.String("cold-dir", "", "source cold-store dir (required iff --source=pack)") bucketPath := fs.String("bucket-path", "sdf-ledger-close-meta/v1/ledgers/pubnet", "GCS destination_bucket_path (used iff --source=bsb)") + lcmFile := fs.String("lcm-file", "", + "framed-XDR LedgerCloseMeta file from apply-load (required iff --source=lcm)") + lcmCheckpoint := fs.Uint("lcm-checkpoint", 0, + "apply-load pre-benchmark checkpoint: skip leading ledgers with seq <= this (used iff --source=lcm)") + lcmFixTxHashes := fs.Bool("lcm-fix-tx-hashes", true, + "repair apply-load's tx-hash/envelope mismatch so the roundtrip ingest reader can consume the meta (used iff --source=lcm)") + lcmAllowPartial := fs.Bool("lcm-allow-partial", true, + "allow the final chunk to be shorter than a full chunk when the apply-load run was sized below 10k ledgers (used iff --source=lcm)") bsbBufferSize := fs.Uint("bsb-buffer-size", 5000, "BSB prefetch buffer depth, PER chunk worker (total buffered ledgers ≈ this × --chunk-workers)") bsbNumWorkers := fs.Uint("bsb-num-workers", 50, @@ -133,6 +143,10 @@ func buildColdDeps(logger *supportlog.Entry) (context.Context, coldDeps, func(), startChunk := chunk.ID(uint32(*chunkArg)) mode := modeString(*xdrViews) + if *source == sourceLCM && *lcmFile == "" { + return nil, coldDeps{}, nil, errors.New("--lcm-file is required when --source=lcm") + } + if *source == sourcePack && enabled["ledgers"] { if samePath(*coldDir, filepath.Join(*coldOutDir, "ledgers")) { return nil, coldDeps{}, nil, fmt.Errorf("--cold-out-dir/ledgers must differ from --cold-dir (%s)", *coldDir) @@ -175,7 +189,12 @@ func buildColdDeps(logger *supportlog.Entry) (context.Context, coldDeps, func(), ctx, cancel := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM) deps := coldDeps{ + logger: logger, source: *source, coldDir: *coldDir, bucketPath: *bucketPath, + lcm: lcmOpts{ + file: *lcmFile, checkpoint: uint32(*lcmCheckpoint), baseChunk: startChunk, + fixTxHashes: *lcmFixTxHashes, passphrase: pubnetPassphrase, allowPartial: *lcmAllowPartial, + }, startChunk: startChunk, numChunks: *numChunks, chunkWorkers: *chunkWorkers, outRoot: *coldOutDir, subdirs: subdirs, enabled: enabled, xdrViews: *xdrViews, parallel: *parallel, mode: mode, @@ -261,7 +280,7 @@ func runOneChunkCold(ctx context.Context, d coldDeps, chunkID chunk.ID) (_ *chun // Acquire this chunk's ledger stream. Each chunk gets its own INDEPENDENT // stream so chunk workers run fully in parallel, and the stream owns its own // setup + teardown (no separate prepare/close to manage here). - stream, oerr := openChunkStream(d.source, d.coldDir, d.bucketPath, d.bsbOpts, chunkID) + stream, oerr := openChunkStream(d.logger, d.source, d.coldDir, d.bucketPath, d.bsbOpts, d.lcm, chunkID) if oerr != nil { return nil, oerr } diff --git a/cmd/stellar-rpc/scripts/bench-fullhistory/bench_cold_ledgers.go b/cmd/stellar-rpc/scripts/bench-fullhistory/bench_cold_ledgers.go index 84cb8cdb4..af3441fae 100644 --- a/cmd/stellar-rpc/scripts/bench-fullhistory/bench_cold_ledgers.go +++ b/cmd/stellar-rpc/scripts/bench-fullhistory/bench_cold_ledgers.go @@ -96,7 +96,6 @@ func coldRangeOp( chunkLo, chunkSpan uint32, n int, ) iterOp { - startSpan := ledgersPerChunk - uint32(n) + 1 return func(rng *rand.Rand, _ bool) (time.Duration, error) { c := chunkLo + rng.Uint32N(chunkSpan) path := packPath(coldDir, c) @@ -113,7 +112,23 @@ func coldRangeOp( } defer r.Close() - start := chunkFirstLedger(c) + rng.Uint32N(startSpan) + // Clamp the start-cursor span to the chunk's ACTUAL ledger range. A + // chunk from a synthetic run sized below LedgersPerChunk is partial, so + // its ledgers occupy only the start of the nominal range; using the full + // nominal span would pick start seqs past the end and short-read. + firstSeq := chunkFirstLedger(c) + if fs, ferr := r.FirstSeq(); ferr == nil && fs > firstSeq { + firstSeq = fs + } + lastSeq := chunkLastLedger(c) + if ls, lerr := r.LastSeq(); lerr == nil && ls < lastSeq { + lastSeq = ls + } + avail := int(lastSeq) - int(firstSeq) + 1 + if avail < n { + return 0, fmt.Errorf("chunk %d has %d ledgers, fewer than n=%d", c, avail, n) + } + start := firstSeq + rng.Uint32N(uint32(avail-n+1)) end := start + uint32(n) - 1 seen := 0 for entry, ierr := range r.IterateLedgers(start, end) { diff --git a/cmd/stellar-rpc/scripts/bench-fullhistory/bench_hot_ingest.go b/cmd/stellar-rpc/scripts/bench-fullhistory/bench_hot_ingest.go index 5384e7a2a..6a729a713 100644 --- a/cmd/stellar-rpc/scripts/bench-fullhistory/bench_hot_ingest.go +++ b/cmd/stellar-rpc/scripts/bench-fullhistory/bench_hot_ingest.go @@ -75,10 +75,14 @@ type hotDeps struct { func buildHotDeps(logger *supportlog.Entry) (context.Context, hotDeps, func(), error) { fs := flag.NewFlagSet("hot-ingest", flag.ExitOnError) typesArg := fs.String("types", "", "comma-separated subset of ledgers,txhash,events (required)") - source := fs.String("source", sourcePack, "ledger source: pack | bsb") + source := fs.String("source", sourcePack, "ledger source: pack | bsb | lcm") coldDir := fs.String("cold-dir", "", "source cold-store dir (required iff --source=pack)") bucketPath := fs.String("bucket-path", "sdf-ledger-close-meta/v1/ledgers/pubnet", "GCS destination_bucket_path (used iff --source=bsb)") + lcmFile := fs.String("lcm-file", "", + "framed-XDR LedgerCloseMeta file from apply-load (required iff --source=lcm)") + lcmCheckpoint := fs.Uint("lcm-checkpoint", 0, + "apply-load pre-benchmark checkpoint: skip leading ledgers with seq <= this (used iff --source=lcm)") bsbBufferSize := fs.Uint("bsb-buffer-size", 5000, "BSB prefetch buffer depth") bsbNumWorkers := fs.Uint("bsb-num-workers", 50, "BSB download workers") retryLimit := fs.Uint("retry-limit", 3, "BSB retry attempts on transient backend failure") @@ -106,6 +110,9 @@ func buildHotDeps(logger *supportlog.Entry) (context.Context, hotDeps, func(), e if *hotDir == "" { return nil, hotDeps{}, nil, errors.New("--hot-dir is required") } + if *source == sourceLCM && *lcmFile == "" { + return nil, hotDeps{}, nil, errors.New("--lcm-file is required when --source=lcm") + } chunkID := chunk.ID(uint32(*chunkArg)) mode := modeString(*xdrViews) @@ -126,8 +133,12 @@ func buildHotDeps(logger *supportlog.Entry) (context.Context, hotDeps, func(), e // Open the single-chunk ledger stream. Hot ingest is single-chunk; the // stream owns its own setup + teardown, so there is nothing to close here // beyond the ingesters. - stream, err := openChunkStream(*source, *coldDir, *bucketPath, + stream, err := openChunkStream(logger, *source, *coldDir, *bucketPath, BSBOpts{BufferSize: *bsbBufferSize, NumWorkers: *bsbNumWorkers, RetryLimit: *retryLimit, RetryWait: *retryWait}, + lcmOpts{ + file: *lcmFile, checkpoint: uint32(*lcmCheckpoint), baseChunk: chunkID, + fixTxHashes: true, passphrase: pubnetPassphrase, allowPartial: true, + }, chunkID) if err != nil { cancel() diff --git a/cmd/stellar-rpc/scripts/bench-fullhistory/corpus.go b/cmd/stellar-rpc/scripts/bench-fullhistory/corpus.go index 2c0cb0fff..4b27e8aee 100644 --- a/cmd/stellar-rpc/scripts/bench-fullhistory/corpus.go +++ b/cmd/stellar-rpc/scripts/bench-fullhistory/corpus.go @@ -22,7 +22,9 @@ import ( "errors" "fmt" "math/rand/v2" + "os" "sort" + "strconv" supportlog "github.com/stellar/go-stellar-sdk/support/log" "github.com/stellar/go-stellar-sdk/xdr" @@ -36,6 +38,31 @@ import ( // one contract-per-filter without forcing a collision. const termsPerCategory = 3 +// maxTopics is Soroban's max contract-event topic count. +const maxTopics = 4 + +// wantTopics is how many topic positions a contract event must have to qualify +// for the corpus. Default 4 (SAC / soroswap transfer-style events). Override +// via EVENTS_TOPIC_COUNT — e.g. 3 for apply-load's custom_token, whose transfer +// events carry 3 topics. Clamped to [1, maxTopics]. +var wantTopics = topicCountFromEnv() + +func topicCountFromEnv() int { + n := maxTopics + if v := os.Getenv("EVENTS_TOPIC_COUNT"); v != "" { + if p, err := strconv.Atoi(v); err == nil { + n = p + } + } + if n < 1 { + n = 1 + } + if n > maxTopics { + n = maxTopics + } + return n +} + // totalTerms is the budget of the picked term universe; matches // eventstore.Query's documented ≤15-unique-term caller ceiling. const totalTerms = 15 @@ -116,13 +143,49 @@ func newCorpus( if err != nil { return nil, fmt.Errorf("corpus: scan: %w", err) } + if len(terms) == 0 { + return nil, errors.New("corpus: 0 filterable terms — no contract events with topics found") + } + // The K-filter sweep needs at least K distinct terms (contract anchors + + // topic values) to place one term per filter. Rather than fail when the + // workload can't reach max(buckets), CAP the sweep to the terms available: + // keep buckets ≤ len(terms) and clamp the rest down to len(terms) (dedup). + // This lets low-diversity workloads (e.g. custom_token's 3-topic events) + // still run at the largest K they can support. The actual per-iter unique + // term count is recorded in nUniqueTerms regardless. + capK := len(terms) + kept := make([]int, 0, len(buckets)) + seen := map[int]bool{} + for _, k := range buckets { + if k > capK { + k = capK + } + if !seen[k] { + seen[k] = true + kept = append(kept, k) + } + } + if capK < buckets[len(buckets)-1] || capK < maxOf(buckets) { + logger.Warnf("corpus: only %d filterable terms; capping K-bucket sweep to ≤%d (requested up to %d)", + len(terms), capK, maxOf(buckets)) + } return &corpus{ terms: terms, - buckets: append([]int(nil), buckets...), + buckets: kept, maxEvents: maxEvents, }, nil } +func maxOf(xs []int) int { + m := xs[0] + for _, x := range xs { + if x > m { + m = x + } + } + return m +} + // Next produces the next request via round-robin partition with // collision-recovery search. Each call advances the RNG; the // sequence is deterministic given the seed. @@ -281,16 +344,20 @@ func scanForTopTerms( stats[cid] = ci } ci.events4Topic++ - for d := range 4 { + for d := range wantTopics { ci.posCounts[d][raws[d]]++ } } - // Anchors: top termsPerCategory contracts by 4-topic event count. - // Each anchor lets a K=3 partition place one contract-constraint - // per filter, ensuring filters AND a specific contract bitmap - // against their topic bitmaps (otherwise filters would only - // constrain topics and the cardinality model degenerates). + // Anchors: up to termsPerCategory contracts by 4-topic event count. + // Each anchor lets a partition place one contract-constraint per filter, + // so filters AND a specific contract bitmap against their topic bitmaps. + // Real pubnet chunks have many contracts; synthetic apply-load workloads + // drive a SINGLE contract, so we accept as few as one anchor and make up + // the term budget from that contract's topic-value diversity (e.g. a SAC's + // `transfer` events vary `from`/`to` over thousands of accounts). The total + // usable-term count — not the contract count — is what the K-filter sweep + // needs (validated against the bucket set in newCorpus). ranked := make([]*contractInfo, 0, len(stats)) for _, ci := range stats { if ci.events4Topic > 0 { @@ -300,11 +367,11 @@ func scanForTopTerms( sort.Slice(ranked, func(i, j int) bool { return ranked[i].events4Topic > ranked[j].events4Topic }) - if len(ranked) < termsPerCategory { - return nil, fmt.Errorf("corpus: only %d contracts emit 4-topic events; need ≥%d", - len(ranked), termsPerCategory) + if len(ranked) == 0 { + return nil, fmt.Errorf("corpus: no contracts emit 4-topic events") } - picked := ranked[:termsPerCategory] + nAnchors := min(termsPerCategory, len(ranked)) + picked := ranked[:nAnchors] // Topic budget: remaining-budget (position, value) pairs aggregated // over the picked contracts, ranked by frequency across positions. @@ -316,7 +383,7 @@ func scanForTopTerms( count int } allValues := make([]posValue, 0, 64) - for d := range 4 { + for d := range wantTopics { agg := map[string]int{} for _, ci := range picked { for v, c := range ci.posCounts[d] { @@ -328,9 +395,9 @@ func scanForTopTerms( } } sort.Slice(allValues, func(i, j int) bool { return allValues[i].count > allValues[j].count }) - topicBudget := min(totalTerms-termsPerCategory, len(allValues)) + topicBudget := min(totalTerms-nAnchors, len(allValues)) - terms := make([]termSpec, 0, termsPerCategory+topicBudget) + terms := make([]termSpec, 0, nAnchors+topicBudget) for _, ci := range picked { cid := ci.id terms = append(terms, termSpec{category: 0, value: append([]byte(nil), cid[:]...)}) @@ -341,8 +408,8 @@ func scanForTopTerms( terms = append(terms, termSpec{category: v.pos + 1, value: []byte(v.value)}) posCount[v.pos]++ } - logger.Infof("corpus: picker emitted %d contracts + topic positions [%d,%d,%d,%d] (%d terms total)", - termsPerCategory, posCount[0], posCount[1], posCount[2], posCount[3], len(terms)) + logger.Infof("corpus: picker emitted %d contract(s) + topic positions [%d,%d,%d,%d] (%d terms total)", + nAnchors, posCount[0], posCount[1], posCount[2], posCount[3], len(terms)) return terms, nil } @@ -386,10 +453,10 @@ func extractContract4TopicsStruct(ev *xdr.ContractEvent) ([32]byte, [4]string, b return zero, raws, false } topics := ev.Body.V0.Topics - if len(topics) != 4 { + if len(topics) != wantTopics { return zero, raws, false } - for d := range 4 { + for d := range wantTopics { b, err := topics[d].MarshalBinary() if err != nil { return zero, raws, false @@ -438,7 +505,7 @@ func extractContract4TopicsView(raw []byte) ([32]byte, [4]string, bool) { return zero, raws, false } count, err := topicsArr.Count() - if err != nil || count != 4 { + if err != nil || int(count) != wantTopics { return zero, raws, false } i := 0 @@ -446,7 +513,7 @@ func extractContract4TopicsView(raw []byte) ([32]byte, [4]string, bool) { if ierr != nil { return zero, raws, false } - if i >= 4 { + if i >= wantTopics { break } topicRaw, rerr := topic.Raw() @@ -462,7 +529,7 @@ func extractContract4TopicsView(raw []byte) ([32]byte, [4]string, bool) { raws[i] = string(topicRaw) i++ } - if i != 4 { + if i != wantTopics { return zero, raws, false } var cid [32]byte diff --git a/cmd/stellar-rpc/scripts/bench-fullhistory/lcm_fixup.go b/cmd/stellar-rpc/scripts/bench-fullhistory/lcm_fixup.go new file mode 100644 index 000000000..fe797b6e2 --- /dev/null +++ b/cmd/stellar-rpc/scripts/bench-fullhistory/lcm_fixup.go @@ -0,0 +1,168 @@ +package main + +// apply-load tx-hash fixup. +// +// stellar-core's `apply-load` streams a LedgerCloseMeta whose transaction SET +// (the generalized tx set's parallel-soroban phase) and transaction RESULTS +// (TxProcessing) are the same transactions but in different order, and whose +// stored result hash (TxProcessing[i].Result.TransactionHash) does NOT equal +// the hash of any envelope under the network passphrase. (Confirmed empirically +// against core 26.1.1: for a dense ledger, 0/N result hashes matched an +// envelope hash under pubnet/testnet/standalone, yet every envelope's source +// account was charged a fee in exactly one TxProcessing entry — i.e. a clean +// bijection via the fee-charged account.) +// +// The go-stellar-sdk ingest LedgerTransactionReader pairs envelopes to results +// BY HASH (it hashes each envelope under the passphrase and looks the stored +// result hash up in that map), so it fails with "unknown tx hash in +// LedgerCloseMeta" on raw apply-load meta. That breaks the roundtrip +// tx-page / tx-hash read benches (the xdr-views path, which pairs positionally, +// is unaffected). +// +// fixupModelTxHashes repairs the meta so the standard reader can consume it: +// for each TxProcessing[i] it finds the fee-charged account, maps it back to +// the unique envelope with that source account, and stamps +// TxProcessing[i].Result.TransactionHash with that envelope's real hash. This +// is a CORRECT pairing (not merely self-consistent): the result/meta stays +// attached to the transaction it actually belongs to. + +import ( + "github.com/stellar/go-stellar-sdk/network" + "github.com/stellar/go-stellar-sdk/xdr" +) + +// fixupStats is returned for logging/validation. +type fixupStats struct { + ledgers int + txs int + fixed int + skipped int // txs that could not be uniquely paired (left untouched) +} + +func (s *fixupStats) add(o fixupStats) { + s.ledgers += o.ledgers + s.txs += o.txs + s.fixed += o.fixed + s.skipped += o.skipped +} + +// feeChargedAccount returns the single account whose entry appears in a tx's +// fee-processing changes (the source account that paid the fee), or "" if the +// changes reference zero or more than one distinct account. +func feeChargedAccount(changes xdr.LedgerEntryChanges) string { + acct := "" + for _, ch := range changes { + var le *xdr.LedgerEntry + switch ch.Type { + case xdr.LedgerEntryChangeTypeLedgerEntryState: + if s, ok := ch.GetState(); ok { + le = &s + } + case xdr.LedgerEntryChangeTypeLedgerEntryUpdated: + if s, ok := ch.GetUpdated(); ok { + le = &s + } + case xdr.LedgerEntryChangeTypeLedgerEntryCreated: + if s, ok := ch.GetCreated(); ok { + le = &s + } + } + if le == nil { + continue + } + ae, ok := le.Data.GetAccount() + if !ok { + continue + } + a := ae.AccountId.Address() + if acct != "" && acct != a { + return "" // more than one distinct account — ambiguous + } + acct = a + } + return acct +} + +// fixupModelTxHashes rewrites a raw framed LedgerCloseMeta payload so the +// ingest LedgerTransactionReader can pair envelopes to results by hash. It +// returns the (possibly rewritten) payload and per-ledger stats. On any decode +// error it returns the input unchanged. +func fixupModelTxHashes(raw []byte, passphrase string) ([]byte, fixupStats, error) { + var lcm xdr.LedgerCloseMeta + if err := lcm.UnmarshalBinary(raw); err != nil { + return nil, fixupStats{}, err + } + n := lcm.CountTransactions() + if n == 0 { + return raw, fixupStats{ledgers: 1}, nil + } + + // source account -> envelope hash, tracking duplicates so we never pair + // ambiguously (an account submitting >1 tx in the ledger). + type ent struct { + h xdr.Hash + count int + } + bySource := make(map[string]*ent, n) + for _, e := range lcm.TransactionEnvelopes() { + h, err := network.HashTransactionInEnvelope(e, passphrase) + if err != nil { + return nil, fixupStats{}, err + } + src := e.SourceAccount().ToAccountId().Address() + if x, ok := bySource[src]; ok { + x.count++ + } else { + bySource[src] = &ent{h: xdr.Hash(h)} + bySource[src].count = 1 + } + } + + st := fixupStats{ledgers: 1, txs: n} + stamp := func(fee xdr.LedgerEntryChanges) (xdr.Hash, bool) { + acct := feeChargedAccount(fee) + if acct == "" { + return xdr.Hash{}, false + } + x, ok := bySource[acct] + if !ok || x.count != 1 { + return xdr.Hash{}, false + } + return x.h, true + } + + switch lcm.V { + case 1: + v1 := lcm.MustV1() + for i := range v1.TxProcessing { + if h, ok := stamp(v1.TxProcessing[i].FeeProcessing); ok { + v1.TxProcessing[i].Result.TransactionHash = h + st.fixed++ + } else { + st.skipped++ + } + } + lcm.V1 = &v1 + case 2: + v2 := lcm.MustV2() + for i := range v2.TxProcessing { + if h, ok := stamp(v2.TxProcessing[i].FeeProcessing); ok { + v2.TxProcessing[i].Result.TransactionHash = h + st.fixed++ + } else { + st.skipped++ + } + } + lcm.V2 = &v2 + default: + // V0 (non-generalized tx set): the SDK reader handles these directly; + // nothing to fix. + return raw, st, nil + } + + out, err := lcm.MarshalBinary() + if err != nil { + return nil, fixupStats{}, err + } + return out, st, nil +} diff --git a/cmd/stellar-rpc/scripts/bench-fullhistory/lcm_source_test.go b/cmd/stellar-rpc/scripts/bench-fullhistory/lcm_source_test.go new file mode 100644 index 000000000..4327a9623 --- /dev/null +++ b/cmd/stellar-rpc/scripts/bench-fullhistory/lcm_source_test.go @@ -0,0 +1,122 @@ +package main + +import ( + "context" + "os" + "path/filepath" + "testing" + + "github.com/stellar/go-stellar-sdk/ingest/ledgerbackend" + "github.com/stellar/go-stellar-sdk/xdr" + + chunkPkg "github.com/stellar/stellar-rpc/cmd/stellar-rpc/internal/fullhistory/pkg/chunk" +) + +// writeLCMFile writes a framed-XDR LedgerCloseMeta stream whose ledgers carry +// the given sequence numbers, mirroring apply-load's METADATA_OUTPUT_STREAM. +func writeLCMFile(t *testing.T, seqs []uint32) string { + t.Helper() + path := filepath.Join(t.TempDir(), "meta.xdr") + f, err := os.Create(path) + if err != nil { + t.Fatalf("create: %v", err) + } + defer f.Close() + for _, s := range seqs { + lcm := xdr.LedgerCloseMeta{ + V: 0, + V0: &xdr.LedgerCloseMetaV0{ + LedgerHeader: xdr.LedgerHeaderHistoryEntry{ + Header: xdr.LedgerHeader{LedgerSeq: xdr.Uint32(s)}, + }, + }, + } + if err := xdr.MarshalFramed(f, lcm); err != nil { + t.Fatalf("MarshalFramed seq=%d: %v", s, err) + } + } + return path +} + +// collectSeqs drains an lcmStream for one chunk and returns the decoded +// ledger sequences it yielded (or the first error). +func collectSeqs(t *testing.T, st *lcmStream, want int) ([]uint32, error) { + t.Helper() + rng := ledgerbackend.BoundedRange(1, uint32(want)) + var got []uint32 + for raw, err := range st.RawLedgers(context.Background(), rng) { + if err != nil { + return got, err + } + var lcm xdr.LedgerCloseMeta + if err := lcm.UnmarshalBinary(raw); err != nil { + t.Fatalf("unmarshal yielded payload: %v", err) + } + got = append(got, lcm.LedgerSequence()) + } + return got, nil +} + +func TestLCMStreamSkipsSetupAndMapsBlocks(t *testing.T) { + // 3 setup ledgers (seq 1..3) then 12 benchmark ledgers (seq 100..111). + // checkpoint=3 means everything <=3 is setup. + setup := []uint32{1, 2, 3} + bench := []uint32{100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111} + file := writeLCMFile(t, append(append([]uint32{}, setup...), bench...)) + + const base = chunkPkg.ID(7) // arbitrary base; block = chunkID - base + + tests := []struct { + name string + chunkID chunkPkg.ID + want int + expect []uint32 + }{ + {"block0", base, 4, []uint32{100, 101, 102, 103}}, + {"block1", base + 1, 4, []uint32{104, 105, 106, 107}}, + {"block2", base + 2, 4, []uint32{108, 109, 110, 111}}, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + st := &lcmStream{ + opts: lcmOpts{file: file, checkpoint: 3, baseChunk: base}, + chunkID: tc.chunkID, + } + got, err := collectSeqs(t, st, tc.want) + if err != nil { + t.Fatalf("unexpected error: %v", err) + } + if len(got) != len(tc.expect) { + t.Fatalf("got %v, want %v", got, tc.expect) + } + for i := range got { + if got[i] != tc.expect[i] { + t.Fatalf("got %v, want %v", got, tc.expect) + } + } + }) + } +} + +func TestLCMStreamNoCheckpointSkip(t *testing.T) { + // checkpoint=0: nothing is skipped, the first frame is benchmark block 0. + file := writeLCMFile(t, []uint32{10, 11, 12}) + st := &lcmStream{opts: lcmOpts{file: file, checkpoint: 0, baseChunk: 1}, chunkID: 1} + got, err := collectSeqs(t, st, 3) + if err != nil { + t.Fatalf("unexpected error: %v", err) + } + if len(got) != 3 || got[0] != 10 || got[2] != 12 { + t.Fatalf("got %v, want [10 11 12]", got) + } +} + +func TestLCMStreamNotEnoughLedgers(t *testing.T) { + // 2 benchmark ledgers but the chunk asks for 4 → short-read error. + file := writeLCMFile(t, []uint32{1, 100, 101}) + st := &lcmStream{opts: lcmOpts{file: file, checkpoint: 1, baseChunk: 1}, chunkID: 1} + _, err := collectSeqs(t, st, 4) + if err == nil { + t.Fatalf("expected short-read error, got nil") + } +} diff --git a/cmd/stellar-rpc/scripts/bench-fullhistory/results/2026-06-09-synthetic-apply-load.md b/cmd/stellar-rpc/scripts/bench-fullhistory/results/2026-06-09-synthetic-apply-load.md new file mode 100644 index 000000000..c5dd5b5c9 --- /dev/null +++ b/cmd/stellar-rpc/scripts/bench-fullhistory/results/2026-06-09-synthetic-apply-load.md @@ -0,0 +1,137 @@ +# stellar-rpc full-history bench — synthetic apply-load datasets (2026-06-09) + +Addresses **[#762](https://github.com/stellar/stellar-rpc/issues/762)** — a +*controllable* synthetic dataset whose transaction profile we set deliberately, +so we can characterize ingest/query behavior under specific load shapes instead +of only whatever pubnet happened to produce. + +Three synthetic datasets were generated with stellar-core `apply-load` +(`apply-load-gen.sh`) and run through the full `bench-fullhistory` read + ingest +suite. Datasets, configs, per-iter/per-sweep CSVs, and the machine-readable +`RESULTS.md` live at +`gs://rpc-full-history/synthetic-ledgers/2026-06-04-apply-load-20k/`; every +number here is recomputed from those CSVs. + +> **Relation to #762.** The issue proposed a `--source=synthetic` bench source +> backed by `ingest/loadtest.ApplyLoad`. The pinned `go-stellar-sdk` doesn't yet +> include `ingest/loadtest`, so this uses the equivalent **`--source=lcm`** path +> (read the framed `LedgerCloseMeta` `apply-load` streams) — same acceptance +> criteria (generate N-chunk datasets → `cold-ingest`/`hot-ingest` → +> `cold-*`/`hot-*` query benches), no dependency bump. Swapping in +> `loadtest.ApplyLoad` later is a drop-in producer change. + +**Interactive explorer:** [`2026-06-09-synthetic-vs-pubnet-explorer.html`](./2026-06-09-synthetic-vs-pubnet-explorer.html) +— a self-contained (offline) HTML with all sweep data embedded for the three +synthetic datasets **plus the pubnet baseline**; toggle tier / decode-path / +percentiles / dataset / concurrency, sort, and overlay series. Also at +`gs://rpc-full-history/synthetic-ledgers/2026-06-04-apply-load-20k/explorer.html`. + +## Setup + +- **machine:** AWS c6id.8xlarge — 32 vCPU (Intel Ice Lake), 61 GB RAM, local NVMe instance store +- **core:** `stellar-core 26.1.1-3289.51ecab1e7.noble~buildtests` (commit `51ecab1e7`, ledger protocol 27) +- **gen:** `APPLY_LOAD_MODE=benchmark`, `BATCH_SAC=1`, `CLUSTERS=8`, pubnet passphrase, tx-hash fixup applied at ingest +- **block model:** 600 ms (per-ledger tx count = TPS × 0.6) +- **bench:** concurrency sweep `1,4,8,16`; iters — ledgers 60, txpage 200, txhash 1000, events 500; both decode paths (roundtrip + xdr-views) + +### Datasets (single machine; all reads run on the c6id.8xlarge above) + +| profile | model tx | target | tx/ledger | ledgers | chunks | txs | cold size | +|---|---|---|---|---|---|---|---| +| **sac** | SAC transfer | 10,000 TPS | 6,000 | 10,000 | 1 | 59.87 M | 28 GB | +| **token** (oz) | custom_token | 9,000 TPS | 5,400 | 10,000 | 1 | 53.85 M | 23 GB | +| **soroswap** | AMM swap | 2,500 TPS | 1,500 | 20,000 | 2 | 29.92 M | 16 GB | + +A full 10k-ledger chunk of 10k-TPS SAC needs ~96–128 GB RAM (in-memory soroban +state grows ~8.5 MB/ledger); on the 61 GB box sac/token were generated at 10k +ledgers (1 chunk) and soroswap — far lighter at 1,500 tx/ledger — at 20k (2 +chunks). See `SYNTHETIC-LEDGERS.md` for the per-RAM sizing table. + +**`pubnet`** = real pubnet chunk 5860 on the same c6id.8xlarge (corrected harness, +`results/2026-06-03-cross-machine.md`) — the non-synthetic baseline, for contrast. + +## Table 1 — Query latency, p50 / p99 @ c=1 (ms). `cold / hot` + +**xdr-views** path (the realistic server path): + +| workload | pubnet (baseline) | sac | token | soroswap | +|---|---|---|---|---| +| tx-page | 3.0 / 1.5 | 26.2 / 21.2 | 25.6 / 21.3 | 14.1 / 11.5 | +| tx-hash | 2.2 / 1.2 | 16.8 / 14.3 | 17.6 / 14.0 | 9.1 / 7.1 | +| events | 14.4 / 4.4 | 229 / 14.0 | 235 / 13.6 | 210 / 32.5 | +| ledgers (n=20) | 15.1 / 13.3 | 101 / 91 | 92 / 84 | 53 / 51 | + +**roundtrip** path (production `UnmarshalBinary` + `ParseTransaction`): + +| workload | pubnet (baseline) | sac | token | soroswap | +|---|---|---|---|---| +| tx-page | 13.2 / 11.1 | 129 / 121 | 135 / 128 | 89 / 84 | +| tx-hash | 11.9 / 10.6 | 117 / 120 | 132 / 114 | 82 / 81 | + +## Table 2 — Peak query throughput, ops/s (best across c=1→16, xdr-views). `cold / hot` + +| workload | pubnet (baseline) | sac | token | soroswap | +|---|---|---|---|---| +| tx-page | 3,456 / 4,830 | 411 / 491 | 412 / 498 | 732 / 888 | +| tx-hash | 4,170 / 7,253 | 575 / 734 | 610 / 789 | 992 / 1,277 | +| events | 512 / 1,843 | 120 / 740 | 112 / 1,140 | 38 / 286 | + +events note: `custom_token` emits **3-topic** events (others 4-topic), so token's +corpus is built with `EVENTS_TOPIC_COUNT=3`; it still reaches the full K=15 +universe. Events latency falls sharply with K (token cold 250 ms @ K=1 → 47 ms @ +K=15) as more filters select fewer events. + +## Table 3 — Ingest throughput (cold-ingest from pack, `--parallel --xdr-views`) + +| profile | total wall | ledgers/s (e2e) | per-ledger p50 / p99 | txhash items/s | events items/s | build-txhash-index | +|---|---|---|---|---|---|---| +| sac | 8m28s | 20 | 24 / 54 ms | 701 k | 142 k | 59.9 M keys @ 23.3 M/s | +| token | 4m22s | 38 | 18 / 33 ms | 707 k | 291 k | 53.8 M keys @ 11.8 M/s | +| soroswap | 3m06s | 108 | 14 / 24 ms | 371 k | 506 k | 29.9 M keys @ 17.1 M/s | + +RocksDB effective config for the hot stores: [`rocksdb-config.md`](./rocksdb-config.md). Note the **events** and **ledgers** CFs run on RocksDB defaults (auto-compaction on, `max_background_jobs=2`, L0 slowdown@20/stop@36), while **txhash** is tuned write-once (compaction off) — which is why the events hot-ingest p99 tail (~8× p50) comes from compaction throttling on the events CF. + +cold-ingest end-to-end is **events-stage-bound** (term-index + cold append); +per-ledger cost scales with events/ledger. Per-ledger cold-ingest stays **under +~55 ms through p99** even on 6k-tx ledgers (rare max-tail spikes to ~0.4–1 s on +packfile flush). + +## How these compare to production (pubnet) chunks + +Same machine + harness, vs the pubnet chunk-5860 run +(`results/2026-06-03-cross-machine.md`): + +- **Per-query latency is ~5–9× higher** and **throughput ~5–8× lower** than the + pubnet chunk. Cause is **per-ledger density**: every query touches a whole + `LedgerCloseMeta`, and a 1.5k–6k-tx synthetic LCM is ~50–300× larger than a + sparse pubnet ledger. The clean gradient *within* the synthetic set — + soroswap (1.5k) ~2× faster than sac/token (6k) on the identical code path — + confirms density, not "synthetic-ness", is the driver. +- **Ingest is item-bound, not ledger-bound:** synthetic ledgers/s is ~15–80× + lower, but per-item rates (keys/s, items/s) match pubnet's order. Same work + per tx, packed into fewer, fatter ledgers. +- **Qualitative findings match pubnet** across all four workloads: xdr-views is + 4–9× faster than roundtrip; hot ≈ cold for point reads but hot wins big on + events; throughput scales with concurrency. All benches ran with **0 errors** + and tx-hash **miss-rate 0**. + +These datasets are a **density stress test** — they exercise sustained 2.5k–10k +TPS regimes pubnet rarely produces, so absolute latencies are higher than +typical pubnet serving. They are not a substitute for the pubnet baseline; they +characterize the read/ingest path under deliberate high-TPS load shapes. + +## Reproducing + +See [`SYNTHETIC-LEDGERS.md`](../SYNTHETIC-LEDGERS.md). End to end: + +```sh +CORE_BIN=/usr/bin/stellar-core OUT_ROOT=/mnt/nvme/synth \ +PROFILES="sac token soroswap" NUM_LEDGERS= \ +GCS_DEST=gs://rpc-full-history/synthetic-ledgers/ \ + ./synthetic-run.sh +``` + +Generation is reproducible from `(config + profile)`; exact transactions are +**not** byte-reproducible (apply-load seeds its RNG from wall-clock and exposes +no seed), so the generated cold packs are the canonical pinned artifact — kept +in GCS at the path above. diff --git a/cmd/stellar-rpc/scripts/bench-fullhistory/results/2026-06-09-synthetic-vs-pubnet-explorer.html b/cmd/stellar-rpc/scripts/bench-fullhistory/results/2026-06-09-synthetic-vs-pubnet-explorer.html new file mode 100644 index 000000000..4dc2729f5 --- /dev/null +++ b/cmd/stellar-rpc/scripts/bench-fullhistory/results/2026-06-09-synthetic-vs-pubnet-explorer.html @@ -0,0 +1,271 @@ + + + +stellar-rpc full-history bench — synthetic (sac/token/soroswap) vs pubnet, c6id.8xlarge (2026-06-09) + + +

stellar-rpc full-history bench — synthetic (sac/token/soroswap) vs pubnet, c6id.8xlarge (2026-06-09)

synthetic: gs://rpc-full-history/synthetic-ledgers/2026-06-04-apply-load-20k/ · pubnet: gs://rpc-full-history/benchmarks/2026-06-03/ · interactive explorer · all data embedded (offline-capable). Toggle tier / decode-path / percentiles / machines / concurrency; click a column header to sort.
+
Queries
Ingest
Ingest totals
+
+
+
+
+ chart metric:
+
+

Line graph — one line per series (dataset · tier · workload · path), x-axis = concurrency. Filter to one workload/path/tier to overlay datasets; click a legend entry to hide/show a line.

+

Bar chart — one bar per visible row.

+
+
+ + +
+ + \ No newline at end of file diff --git a/cmd/stellar-rpc/scripts/bench-fullhistory/results/make_explorer.py b/cmd/stellar-rpc/scripts/bench-fullhistory/results/make_explorer.py new file mode 100644 index 000000000..f25588234 --- /dev/null +++ b/cmd/stellar-rpc/scripts/bench-fullhistory/results/make_explorer.py @@ -0,0 +1,398 @@ +#!/usr/bin/env python3 +"""Build a self-contained interactive HTML explorer from bench-fullhistory runs. + +Usage: + python3 make_explorer.py [--title "..."] [--source "gs://..."] + + contains one subdir per machine (the per-machine result dirs as +uploaded to gs://rpc-full-history/benchmarks//). The output is a single +HTML file with all data embedded — no server, no build step, works offline. +""" +import csv, json, os, sys, argparse + +# machine dir prefix -> short label +def machine_label(dirname): + return dirname.split("-2026")[0].split("-2025")[0] + +def rc(path): + if not os.path.exists(path): + return [] + with open(path) as f: + return list(csv.DictReader(f)) + +def sweep_rows(d, fn, machine, tier, workload, path): + out = [] + for r in rc(os.path.join(d, fn)): + out.append({ + "machine": machine, "tier": tier, "workload": workload, "path": path, + "c": int(r["query_concurrency"]), + "p50": float(r["p50_ms"]), "p90": float(r["p90_ms"]), + "p99": float(r["p99_ms"]), "max": float(r["max_ms"]), + "ops": float(r["ops_per_sec"]), + }) + return out + +def stage_rows(d, fn, machine, tier, mode, mapping): + """mapping: {csv_stage: display_stage}""" + out = [] + for r in rc(os.path.join(d, fn)): + st = r["stage"] + if st not in mapping: + continue + out.append({ + "machine": machine, "tier": tier, "mode": mode, "stage": mapping[st], + "p50": float(r["p50_ns"]) / 1e6, "p90": float(r["p90_ns"]) / 1e6, + "p99": float(r["p99_ns"]) / 1e6, "max": float(r["max_ns"]) / 1e6, + }) + return out + +def total_ns(d, fn, stage="total_per_ledger"): + for r in rc(os.path.join(d, fn)): + if r["stage"] == stage: + return int(r["total_ns"]) + return None + +def build(run_root): + machines, queries, ingest, throughput, buildidx = [], [], [], [], [] + dirs = sorted(x for x in os.listdir(run_root) if os.path.isdir(os.path.join(run_root, x))) + # stable machine ordering: 2x,4x,8x,arm + order = {"c6id.2xlarge": 0, "c6id.4xlarge": 1, "c6id.8xlarge": 2, "im4gn.4xlarge": 3} + dirs.sort(key=lambda x: order.get(machine_label(x), 99)) + for dn in dirs: + d = os.path.join(run_root, dn) + m = machine_label(dn) + machines.append(m) + for tier in ("cold", "hot"): + queries += sweep_rows(d, f"{tier}-ledgers.csv", m, tier, "ledgers", "raw") + queries += sweep_rows(d, f"{tier}-txpage-20-roundtrip-sweep.csv", m, tier, "tx-page", "roundtrip") + queries += sweep_rows(d, f"{tier}-txpage-20-xdrviews-sweep.csv", m, tier, "tx-page", "xdr-views") + queries += sweep_rows(d, f"{tier}-txhash-roundtrip-sweep.csv", m, tier, "tx-hash", "roundtrip") + queries += sweep_rows(d, f"{tier}-txhash-xdrviews-sweep.csv", m, tier, "tx-hash", "xdr-views") + queries += sweep_rows(d, f"{tier}-events-query-sweep.csv", m, tier, "events", "roundtrip") + queries += sweep_rows(d, f"{tier}-events-query-xdrviews-sweep.csv", m, tier, "events", "xdr-views") + # hot ingest: view + parsed + for mode in ("view", "parsed"): + ingest += stage_rows(d, f"hot-ledgers-{mode}.csv", m, "hot", mode, {"write": "ledgers.write"}) + ingest += stage_rows(d, f"hot-txhash-{mode}.csv", m, "hot", mode, {"extract": "txhash.extract", "hot_write": "txhash.write"}) + ingest += stage_rows(d, f"hot-events-{mode}.csv", m, "hot", mode, {"extract": "events.extract", "hot_write": "events.write"}) + ingest += stage_rows(d, f"hot-driver-{mode}.csv", m, "hot", mode, { + "read_blocked": "driver.read_blocked", "fan_out_per_ledger": "driver.fan_out", + "lcm_decode": "driver.lcm_decode", "total_per_ledger": "driver.total_per_ledger"}) + tn = total_ns(d, f"hot-driver-{mode}.csv") + if tn: + throughput.append({"machine": m, "tier": "hot", "mode": mode, "ledgers_per_s": round(10000 / (tn / 1e9), 1)}) + # cold ingest: view only + ingest += stage_rows(d, "cold-ledgers-view.csv", m, "cold", "view", {"write": "ledgers.write"}) + ingest += stage_rows(d, "cold-txhash-view.csv", m, "cold", "view", {"extract": "txhash.extract"}) + ingest += stage_rows(d, "cold-events-view.csv", m, "cold", "view", { + "extract": "events.extract", "term_index": "events.term_index", "cold_append": "events.cold_append"}) + ingest += stage_rows(d, "cold-driver-view.csv", m, "cold", "view", { + "read_blocked": "driver.read_blocked", "fan_out_per_ledger": "driver.fan_out"}) + cw = next((r for r in rc(os.path.join(d, "cold-driver-view.csv")) if r["stage"] == "chunk_wall"), None) + if cw: + # chunk_wall total_ns sums the per-chunk walls (count = n chunks). + # Effective e2e wall ≈ that sum / chunk-workers. These synthetic runs + # used chunk-workers == chunk-count (n), so divide by n; override with + # COLD_CHUNK_WORKERS for a run that used a different worker count. + n = int(cw["n"]) + workers = int(os.environ.get("COLD_CHUNK_WORKERS", n)) or 1 + secs = (int(cw["total_ns"]) / 1e9) / workers + throughput.append({"machine": m, "tier": "cold", "mode": "view", "ledgers_per_s": round((n * 10000) / secs, 0)}) + bi = rc(os.path.join(d, "build-txhash-index.csv")) + if bi: + r = bi[0] + keys = int(r["total_keys"]); secs = (int(r["feed_ns"]) + int(r["finish_ns"])) / 1e9 + buildidx.append({"machine": m, "keys_per_s": round(keys / secs), "idx_mb": round(int(r["index_bytes"]) / 1e6)}) + return {"machines": machines, "queries": queries, "ingest": ingest, + "throughput": throughput, "build_index": buildidx} + +HTML = r""" + + +__TITLE__ + + +

__TITLE__

__SOURCE__ · interactive explorer · all data embedded (offline-capable). Toggle tier / decode-path / percentiles / machines / concurrency; click a column header to sort.
+
Queries
Ingest
Ingest totals
+
+
+
+
+ chart metric:
+
+

Line graph — one line per series (dataset · tier · workload · path), x-axis = concurrency. Filter to one workload/path/tier to overlay datasets; click a legend entry to hide/show a line.

+

Bar chart — one bar per visible row.

+
+
+ + +
+ +""" + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("run_root") + ap.add_argument("out") + ap.add_argument("--title", default="stellar-rpc full-history bench explorer") + ap.add_argument("--source", default="") + a = ap.parse_args() + data = build(a.run_root) + html = (HTML.replace("__TITLE__", a.title) + .replace("__SOURCE__", a.source or a.run_root) + .replace("__DATA__", json.dumps(data, separators=(",", ":")))) + with open(a.out, "w") as f: + f.write(html) + print(f"wrote {a.out}: {len(data['queries'])} query rows, {len(data['ingest'])} ingest rows, " + f"{len(data['machines'])} machines, {len(html)} bytes") + +if __name__ == "__main__": + main() diff --git a/cmd/stellar-rpc/scripts/bench-fullhistory/results/rocksdb-config.md b/cmd/stellar-rpc/scripts/bench-fullhistory/results/rocksdb-config.md new file mode 100644 index 000000000..02c5cfde2 --- /dev/null +++ b/cmd/stellar-rpc/scripts/bench-fullhistory/results/rocksdb-config.md @@ -0,0 +1,637 @@ +# RocksDB config (full-history hot stores) + +Effective config from the `OPTIONS` files RocksDB writes at open (fully resolved). +Set in `internal/fullhistory/pkg/rocksdb/{rocksdb.go,tuning.go}`; grocksdb v1.10.7 → RocksDB 10.9.1. +Store: sac hot store, `bench-results/all3-20260608T181106Z` (c6id.8xlarge). + +## Key knobs per column family + +| knob | events | ledgers | txhash | +|---|---|---|---| +| write_buffer_size | 67108864 | 67108864 | 67108864 | +| max_write_buffer_number | 2 | 2 | 2 | +| min_write_buffer_number_to_merge | 1 | 1 | 1 | +| level0_file_num_compaction_trigger | 4 | 4 | 999 | +| level0_slowdown_writes_trigger | 20 | 20 | 999 | +| level0_stop_writes_trigger | 36 | 36 | 999 | +| target_file_size_base | 67108864 | 67108864 | 67108864 | +| max_bytes_for_level_base | 268435456 | 268435456 | 268435456 | +| max_bytes_for_level_multiplier | 10.000000 | 10.000000 | 10.000000 | +| num_levels | 7 | 7 | 7 | +| disable_auto_compactions | false | false | true | +| compression | kNoCompression | kNoCompression | kNoCompression | +| max_background_jobs (DB) | 2 | 2 | 8 | +| max_open_files (DB) | -1 | -1 | 10000 | + +_bytes: 67108864 = 64 MiB, 268435456 = 256 MiB._ + +## Why this explains the p99 ingest tail + +- **`max_background_jobs=2`** — only 2 threads for all flushes + compactions per store. At ~6k events/ledger the events CF fills its 64 MiB memtable fast; 2 background threads can't always keep flush + L0→L1 compaction up. +- **`level0_slowdown_writes_trigger=20`, `level0_stop_writes_trigger=36`** — at 20 L0 files writes are throttled, at 36 they stall. Compaction debt on the events CF periodically hits these → the p99/max spikes while p50 stays low. +- **64 MiB memtable × `max_write_buffer_number=2`** — a third fill back-pressures writers until a flush drains. +- Mostly RocksDB defaults — the wrapper leaves a knob at default unless a non-zero `Tuning` value is passed; **not** tuned for the synthetic dense-write worst case. + +## Full OPTIONS — events CF (verbatim, the tail driver) +```ini +# This is a RocksDB option file. +# +# For detailed file format spec, please refer to the example file +# in examples/rocksdb_option_file_example.ini +# + +[Version] + rocksdb_version=10.9.1 + options_file_version=1.1 + +[DBOptions] + max_manifest_space_amp_pct=500 + manifest_preallocation_size=4194304 + max_manifest_file_size=1073741824 + compaction_readahead_size=2097152 + strict_bytes_per_sync=false + bytes_per_sync=0 + max_background_jobs=2 + avoid_flush_during_shutdown=false + max_background_flushes=-1 + delayed_write_rate=16777216 + max_open_files=-1 + max_subcompactions=1 + writable_file_max_buffer_size=1048576 + wal_bytes_per_sync=0 + max_background_compactions=-1 + max_total_wal_size=0 + delete_obsolete_files_period_micros=21600000000 + stats_dump_period_sec=600 + stats_history_buffer_size=1048576 + stats_persist_period_sec=600 + follower_refresh_catchup_period_ms=10000 + enforce_single_del_contracts=true + lowest_used_cache_tier=kNonVolatileBlockTier + bgerror_resume_retry_interval=1000000 + metadata_write_temperature=kUnknown + best_efforts_recovery=false + log_readahead_size=0 + write_identity_file=true + write_dbid_to_manifest=true + prefix_seek_opt_in_only=false + wal_compression=kNoCompression + manual_wal_flush=false + db_host_id=__hostname__ + two_write_queues=false + skip_checking_sst_file_sizes_on_db_open=false + flush_verify_memtable_count=true + atomic_flush=false + verify_sst_unique_id_in_manifest=true + skip_stats_update_on_db_open=false + track_and_verify_wals=false + track_and_verify_wals_in_manifest=false + compaction_verify_record_count=true + paranoid_checks=true + create_if_missing=true + max_write_batch_group_size_bytes=1048576 + follower_catchup_retry_count=10 + avoid_flush_during_recovery=false + file_checksum_gen_factory=nullptr + enable_thread_tracking=false + allow_fallocate=true + allow_data_in_errors=false + error_if_exists=false + use_direct_io_for_flush_and_compaction=false + background_close_inactive_wals=false + create_missing_column_families=true + WAL_size_limit_MB=0 + use_direct_reads=false + persist_stats_to_disk=false + allow_2pc=false + max_log_file_size=0 + is_fd_close_on_exec=true + avoid_unnecessary_blocking_io=false + max_file_opening_threads=16 + wal_filter=nullptr + wal_write_temperature=kUnknown + follower_catchup_retry_wait_ms=100 + allow_mmap_reads=false + allow_mmap_writes=false + use_adaptive_mutex=false + use_fsync=false + table_cache_numshardbits=6 + dump_malloc_stats=false + db_write_buffer_size=0 + allow_ingest_behind=false + keep_log_file_num=1000 + max_bgerror_resume_count=2147483647 + allow_concurrent_memtable_write=true + recycle_log_file_num=0 + log_file_time_to_roll=0 + WAL_ttl_seconds=0 + enable_pipelined_write=false + write_thread_slow_yield_usec=3 + unordered_write=false + wal_recovery_mode=kPointInTimeRecovery + enable_write_thread_adaptive_yield=true + write_thread_max_yield_usec=100 + advise_random_on_open=true + info_log_level=INFO_LEVEL + + +[CFOptions "default"] + memtable_max_range_deletions=0 + compression_manager=nullptr + compression_opts={checksum=false;max_dict_buffer_bytes=0;enabled=false;max_dict_bytes=0;max_compressed_bytes_per_kb=896;parallel_threads=1;zstd_max_train_bytes=0;level=32767;use_zstd_dict_trainer=true;strategy=0;window_bits=-14;} + paranoid_memory_checks=false + memtable_avg_op_scan_flush_trigger=0 + block_protection_bytes_per_key=0 + uncache_aggressiveness=0 + bottommost_file_compaction_delay=0 + memtable_protection_bytes_per_key=0 + bottommost_compression=kDisableCompressionOption + sample_for_compression=0 + prepopulate_blob_cache=kDisable + blob_file_starting_level=0 + blob_compaction_readahead_size=0 + blob_garbage_collection_force_threshold=1.000000 + blob_garbage_collection_age_cutoff=0.250000 + table_factory=BlockBasedTable + max_successive_merges=0 + max_write_buffer_number=2 + prefix_extractor=nullptr + memtable_huge_page_size=0 + write_buffer_size=67108864 + strict_max_successive_merges=false + arena_block_size=1048576 + memtable_op_scan_flush_trigger=0 + level0_file_num_compaction_trigger=4 + report_bg_io_stats=false + inplace_update_num_locks=10000 + memtable_prefix_bloom_size_ratio=0.000000 + level0_stop_writes_trigger=36 + blob_compression_type=kNoCompression + level0_slowdown_writes_trigger=20 + hard_pending_compaction_bytes_limit=274877906944 + target_file_size_multiplier=1 + paranoid_file_checks=false + min_blob_size=0 + max_compaction_bytes=1677721600 + disable_auto_compactions=false + experimental_mempurge_threshold=0.000000 + verify_output_flags=0 + last_level_temperature=kUnknown + preserve_internal_time_seconds=0 + memtable_veirfy_per_key_checksum_on_seek=false + soft_pending_compaction_bytes_limit=68719476736 + target_file_size_base=67108864 + enable_blob_files=false + bottommost_compression_opts={checksum=false;max_dict_buffer_bytes=0;enabled=false;max_dict_bytes=0;max_compressed_bytes_per_kb=896;parallel_threads=1;zstd_max_train_bytes=0;level=32767;use_zstd_dict_trainer=true;strategy=0;window_bits=-14;} + memtable_whole_key_filtering=false + target_file_size_is_upper_bound=false + max_bytes_for_level_base=268435456 + compaction_options_fifo={trivial_copy_buffer_size=4096;allow_trivial_copy_when_change_temperature=false;file_temperature_age_thresholds=;allow_compaction=false;age_for_warm=0;max_table_files_size=1073741824;} + max_bytes_for_level_multiplier=10.000000 + max_bytes_for_level_multiplier_additional=1:1:1:1:1:1:1 + max_sequential_skip_in_iterations=8 + compression=kNoCompression + default_write_temperature=kUnknown + compaction_options_universal={reduce_file_locking=false;incremental=false;compression_size_percent=-1;allow_trivial_move=false;max_size_amplification_percent=200;max_merge_width=4294967295;stop_style=kCompactionStopStyleTotalSize;min_merge_width=2;max_read_amp=-1;size_ratio=1;} + ttl=2592000 + periodic_compaction_seconds=0 + preclude_last_level_data_seconds=0 + blob_file_size=268435456 + enable_blob_garbage_collection=false + cf_allow_ingest_behind=false + min_write_buffer_number_to_merge=1 + sst_partitioner_factory=nullptr + num_levels=7 + disallow_memtable_writes=false + force_consistency_checks=true + memtable_insert_with_hint_prefix_extractor=nullptr + memtable_factory=SkipListFactory + optimize_filters_for_hits=false + level_compaction_dynamic_level_bytes=true + compaction_style=kCompactionStyleLevel + compaction_filter=nullptr + default_temperature=kUnknown + inplace_update_support=false + merge_operator=nullptr + bloom_locality=0 + comparator=leveldb.BytewiseComparator + compaction_filter_factory=nullptr + max_write_buffer_size_to_maintain=0 + compaction_pri=kMinOverlappingRatio + persist_user_defined_timestamps=true + +[TableOptions/BlockBasedTable "default"] + fail_if_no_udi_on_open=false + initial_auto_readahead_size=8192 + max_auto_readahead_size=262144 + metadata_cache_options={unpartitioned_pinning=kFallback;partition_pinning=kFallback;top_level_index_pinning=kFallback;} + block_align=false + read_amp_bytes_per_bit=0 + verify_compression=false + detect_filter_construct_corruption=false + whole_key_filtering=true + user_defined_index_factory=nullptr + filter_policy=nullptr + super_block_alignment_space_overhead_ratio=128 + use_delta_encoding=true + optimize_filters_for_memory=true + partition_filters=false + prepopulate_block_cache=kDisable + pin_top_level_index_and_filter=true + index_block_restart_interval=1 + block_size_deviation=10 + num_file_reads_for_auto_readahead=2 + format_version=6 + decouple_partitioned_filters=true + checksum=kXXH3 + block_size=4096 + data_block_hash_table_util_ratio=0.750000 + index_shortening=kShortenSeparators + block_restart_interval=16 + data_block_index_type=kDataBlockBinarySearch + index_type=kBinarySearch + super_block_alignment_size=0 + metadata_block_size=4096 + pin_l0_filter_and_index_blocks_in_cache=false + no_block_cache=false + cache_index_and_filter_blocks_with_high_priority=true + cache_index_and_filter_blocks=false + enable_index_compression=true + flush_block_policy_factory=FlushBlockBySizePolicyFactory + + +[CFOptions "events_data"] + memtable_max_range_deletions=0 + compression_manager=nullptr + compression_opts={checksum=false;max_dict_buffer_bytes=0;enabled=false;max_dict_bytes=0;max_compressed_bytes_per_kb=896;parallel_threads=1;zstd_max_train_bytes=0;level=32767;use_zstd_dict_trainer=true;strategy=0;window_bits=-14;} + paranoid_memory_checks=false + memtable_avg_op_scan_flush_trigger=0 + block_protection_bytes_per_key=0 + uncache_aggressiveness=0 + bottommost_file_compaction_delay=0 + memtable_protection_bytes_per_key=0 + bottommost_compression=kDisableCompressionOption + sample_for_compression=0 + prepopulate_blob_cache=kDisable + blob_file_starting_level=0 + blob_compaction_readahead_size=0 + blob_garbage_collection_force_threshold=1.000000 + blob_garbage_collection_age_cutoff=0.250000 + table_factory=BlockBasedTable + max_successive_merges=0 + max_write_buffer_number=2 + prefix_extractor=nullptr + memtable_huge_page_size=0 + write_buffer_size=67108864 + strict_max_successive_merges=false + arena_block_size=1048576 + memtable_op_scan_flush_trigger=0 + level0_file_num_compaction_trigger=4 + report_bg_io_stats=false + inplace_update_num_locks=10000 + memtable_prefix_bloom_size_ratio=0.000000 + level0_stop_writes_trigger=36 + blob_compression_type=kNoCompression + level0_slowdown_writes_trigger=20 + hard_pending_compaction_bytes_limit=274877906944 + target_file_size_multiplier=1 + paranoid_file_checks=false + min_blob_size=0 + max_compaction_bytes=1677721600 + disable_auto_compactions=false + experimental_mempurge_threshold=0.000000 + verify_output_flags=0 + last_level_temperature=kUnknown + preserve_internal_time_seconds=0 + memtable_veirfy_per_key_checksum_on_seek=false + soft_pending_compaction_bytes_limit=68719476736 + target_file_size_base=67108864 + enable_blob_files=false + bottommost_compression_opts={checksum=false;max_dict_buffer_bytes=0;enabled=false;max_dict_bytes=0;max_compressed_bytes_per_kb=896;parallel_threads=1;zstd_max_train_bytes=0;level=32767;use_zstd_dict_trainer=true;strategy=0;window_bits=-14;} + memtable_whole_key_filtering=false + target_file_size_is_upper_bound=false + max_bytes_for_level_base=268435456 + compaction_options_fifo={trivial_copy_buffer_size=4096;allow_trivial_copy_when_change_temperature=false;file_temperature_age_thresholds=;allow_compaction=false;age_for_warm=0;max_table_files_size=1073741824;} + max_bytes_for_level_multiplier=10.000000 + max_bytes_for_level_multiplier_additional=1:1:1:1:1:1:1 + max_sequential_skip_in_iterations=8 + compression=kZSTD + default_write_temperature=kUnknown + compaction_options_universal={reduce_file_locking=false;incremental=false;compression_size_percent=-1;allow_trivial_move=false;max_size_amplification_percent=200;max_merge_width=4294967295;stop_style=kCompactionStopStyleTotalSize;min_merge_width=2;max_read_amp=-1;size_ratio=1;} + ttl=2592000 + periodic_compaction_seconds=0 + preclude_last_level_data_seconds=0 + blob_file_size=268435456 + enable_blob_garbage_collection=false + cf_allow_ingest_behind=false + min_write_buffer_number_to_merge=1 + sst_partitioner_factory=nullptr + num_levels=7 + disallow_memtable_writes=false + force_consistency_checks=true + memtable_insert_with_hint_prefix_extractor=nullptr + memtable_factory=SkipListFactory + optimize_filters_for_hits=false + level_compaction_dynamic_level_bytes=true + compaction_style=kCompactionStyleLevel + compaction_filter=nullptr + default_temperature=kUnknown + inplace_update_support=false + merge_operator=nullptr + bloom_locality=0 + comparator=leveldb.BytewiseComparator + compaction_filter_factory=nullptr + max_write_buffer_size_to_maintain=0 + compaction_pri=kMinOverlappingRatio + persist_user_defined_timestamps=true + +[TableOptions/BlockBasedTable "events_data"] + fail_if_no_udi_on_open=false + initial_auto_readahead_size=8192 + max_auto_readahead_size=262144 + metadata_cache_options={unpartitioned_pinning=kFallback;partition_pinning=kFallback;top_level_index_pinning=kFallback;} + block_align=false + read_amp_bytes_per_bit=0 + verify_compression=false + detect_filter_construct_corruption=false + whole_key_filtering=true + user_defined_index_factory=nullptr + filter_policy=nullptr + super_block_alignment_space_overhead_ratio=128 + use_delta_encoding=true + optimize_filters_for_memory=true + partition_filters=false + prepopulate_block_cache=kDisable + pin_top_level_index_and_filter=true + index_block_restart_interval=1 + block_size_deviation=10 + num_file_reads_for_auto_readahead=2 + format_version=6 + decouple_partitioned_filters=true + checksum=kXXH3 + block_size=32768 + data_block_hash_table_util_ratio=0.750000 + index_shortening=kShortenSeparators + block_restart_interval=16 + data_block_index_type=kDataBlockBinarySearch + index_type=kBinarySearch + super_block_alignment_size=0 + metadata_block_size=4096 + pin_l0_filter_and_index_blocks_in_cache=false + no_block_cache=false + cache_index_and_filter_blocks_with_high_priority=true + cache_index_and_filter_blocks=false + enable_index_compression=true + flush_block_policy_factory=FlushBlockBySizePolicyFactory + + +[CFOptions "events_index"] + memtable_max_range_deletions=0 + compression_manager=nullptr + compression_opts={checksum=false;max_dict_buffer_bytes=0;enabled=false;max_dict_bytes=0;max_compressed_bytes_per_kb=896;parallel_threads=1;zstd_max_train_bytes=0;level=32767;use_zstd_dict_trainer=true;strategy=0;window_bits=-14;} + paranoid_memory_checks=false + memtable_avg_op_scan_flush_trigger=0 + block_protection_bytes_per_key=0 + uncache_aggressiveness=0 + bottommost_file_compaction_delay=0 + memtable_protection_bytes_per_key=0 + bottommost_compression=kDisableCompressionOption + sample_for_compression=0 + prepopulate_blob_cache=kDisable + blob_file_starting_level=0 + blob_compaction_readahead_size=0 + blob_garbage_collection_force_threshold=1.000000 + blob_garbage_collection_age_cutoff=0.250000 + table_factory=BlockBasedTable + max_successive_merges=0 + max_write_buffer_number=2 + prefix_extractor=nullptr + memtable_huge_page_size=0 + write_buffer_size=67108864 + strict_max_successive_merges=false + arena_block_size=1048576 + memtable_op_scan_flush_trigger=0 + level0_file_num_compaction_trigger=4 + report_bg_io_stats=false + inplace_update_num_locks=10000 + memtable_prefix_bloom_size_ratio=0.000000 + level0_stop_writes_trigger=36 + blob_compression_type=kNoCompression + level0_slowdown_writes_trigger=20 + hard_pending_compaction_bytes_limit=274877906944 + target_file_size_multiplier=1 + paranoid_file_checks=false + min_blob_size=0 + max_compaction_bytes=1677721600 + disable_auto_compactions=false + experimental_mempurge_threshold=0.000000 + verify_output_flags=0 + last_level_temperature=kUnknown + preserve_internal_time_seconds=0 + memtable_veirfy_per_key_checksum_on_seek=false + soft_pending_compaction_bytes_limit=68719476736 + target_file_size_base=67108864 + enable_blob_files=false + bottommost_compression_opts={checksum=false;max_dict_buffer_bytes=0;enabled=false;max_dict_bytes=0;max_compressed_bytes_per_kb=896;parallel_threads=1;zstd_max_train_bytes=0;level=32767;use_zstd_dict_trainer=true;strategy=0;window_bits=-14;} + memtable_whole_key_filtering=false + target_file_size_is_upper_bound=false + max_bytes_for_level_base=268435456 + compaction_options_fifo={trivial_copy_buffer_size=4096;allow_trivial_copy_when_change_temperature=false;file_temperature_age_thresholds=;allow_compaction=false;age_for_warm=0;max_table_files_size=1073741824;} + max_bytes_for_level_multiplier=10.000000 + max_bytes_for_level_multiplier_additional=1:1:1:1:1:1:1 + max_sequential_skip_in_iterations=8 + compression=kNoCompression + default_write_temperature=kUnknown + compaction_options_universal={reduce_file_locking=false;incremental=false;compression_size_percent=-1;allow_trivial_move=false;max_size_amplification_percent=200;max_merge_width=4294967295;stop_style=kCompactionStopStyleTotalSize;min_merge_width=2;max_read_amp=-1;size_ratio=1;} + ttl=2592000 + periodic_compaction_seconds=0 + preclude_last_level_data_seconds=0 + blob_file_size=268435456 + enable_blob_garbage_collection=false + cf_allow_ingest_behind=false + min_write_buffer_number_to_merge=1 + sst_partitioner_factory=nullptr + num_levels=7 + disallow_memtable_writes=false + force_consistency_checks=true + memtable_insert_with_hint_prefix_extractor=nullptr + memtable_factory=SkipListFactory + optimize_filters_for_hits=false + level_compaction_dynamic_level_bytes=true + compaction_style=kCompactionStyleLevel + compaction_filter=nullptr + default_temperature=kUnknown + inplace_update_support=false + merge_operator=nullptr + bloom_locality=0 + comparator=leveldb.BytewiseComparator + compaction_filter_factory=nullptr + max_write_buffer_size_to_maintain=0 + compaction_pri=kMinOverlappingRatio + persist_user_defined_timestamps=true + +[TableOptions/BlockBasedTable "events_index"] + fail_if_no_udi_on_open=false + initial_auto_readahead_size=8192 + max_auto_readahead_size=262144 + metadata_cache_options={unpartitioned_pinning=kFallback;partition_pinning=kFallback;top_level_index_pinning=kFallback;} + block_align=false + read_amp_bytes_per_bit=0 + verify_compression=false + detect_filter_construct_corruption=false + whole_key_filtering=true + user_defined_index_factory=nullptr + filter_policy=nullptr + super_block_alignment_space_overhead_ratio=128 + use_delta_encoding=true + optimize_filters_for_memory=true + partition_filters=false + prepopulate_block_cache=kDisable + pin_top_level_index_and_filter=true + index_block_restart_interval=1 + block_size_deviation=10 + num_file_reads_for_auto_readahead=2 + format_version=6 + decouple_partitioned_filters=true + checksum=kXXH3 + block_size=4096 + data_block_hash_table_util_ratio=0.750000 + index_shortening=kShortenSeparators + block_restart_interval=16 + data_block_index_type=kDataBlockBinarySearch + index_type=kBinarySearch + super_block_alignment_size=0 + metadata_block_size=4096 + pin_l0_filter_and_index_blocks_in_cache=false + no_block_cache=false + cache_index_and_filter_blocks_with_high_priority=true + cache_index_and_filter_blocks=false + enable_index_compression=true + flush_block_policy_factory=FlushBlockBySizePolicyFactory + + +[CFOptions "events_offsets"] + memtable_max_range_deletions=0 + compression_manager=nullptr + compression_opts={checksum=false;max_dict_buffer_bytes=0;enabled=false;max_dict_bytes=0;max_compressed_bytes_per_kb=896;parallel_threads=1;zstd_max_train_bytes=0;level=32767;use_zstd_dict_trainer=true;strategy=0;window_bits=-14;} + paranoid_memory_checks=false + memtable_avg_op_scan_flush_trigger=0 + block_protection_bytes_per_key=0 + uncache_aggressiveness=0 + bottommost_file_compaction_delay=0 + memtable_protection_bytes_per_key=0 + bottommost_compression=kDisableCompressionOption + sample_for_compression=0 + prepopulate_blob_cache=kDisable + blob_file_starting_level=0 + blob_compaction_readahead_size=0 + blob_garbage_collection_force_threshold=1.000000 + blob_garbage_collection_age_cutoff=0.250000 + table_factory=BlockBasedTable + max_successive_merges=0 + max_write_buffer_number=2 + prefix_extractor=nullptr + memtable_huge_page_size=0 + write_buffer_size=67108864 + strict_max_successive_merges=false + arena_block_size=1048576 + memtable_op_scan_flush_trigger=0 + level0_file_num_compaction_trigger=4 + report_bg_io_stats=false + inplace_update_num_locks=10000 + memtable_prefix_bloom_size_ratio=0.000000 + level0_stop_writes_trigger=36 + blob_compression_type=kNoCompression + level0_slowdown_writes_trigger=20 + hard_pending_compaction_bytes_limit=274877906944 + target_file_size_multiplier=1 + paranoid_file_checks=false + min_blob_size=0 + max_compaction_bytes=1677721600 + disable_auto_compactions=false + experimental_mempurge_threshold=0.000000 + verify_output_flags=0 + last_level_temperature=kUnknown + preserve_internal_time_seconds=0 + memtable_veirfy_per_key_checksum_on_seek=false + soft_pending_compaction_bytes_limit=68719476736 + target_file_size_base=67108864 + enable_blob_files=false + bottommost_compression_opts={checksum=false;max_dict_buffer_bytes=0;enabled=false;max_dict_bytes=0;max_compressed_bytes_per_kb=896;parallel_threads=1;zstd_max_train_bytes=0;level=32767;use_zstd_dict_trainer=true;strategy=0;window_bits=-14;} + memtable_whole_key_filtering=false + target_file_size_is_upper_bound=false + max_bytes_for_level_base=268435456 + compaction_options_fifo={trivial_copy_buffer_size=4096;allow_trivial_copy_when_change_temperature=false;file_temperature_age_thresholds=;allow_compaction=false;age_for_warm=0;max_table_files_size=1073741824;} + max_bytes_for_level_multiplier=10.000000 + max_bytes_for_level_multiplier_additional=1:1:1:1:1:1:1 + max_sequential_skip_in_iterations=8 + compression=kNoCompression + default_write_temperature=kUnknown + compaction_options_universal={reduce_file_locking=false;incremental=false;compression_size_percent=-1;allow_trivial_move=false;max_size_amplification_percent=200;max_merge_width=4294967295;stop_style=kCompactionStopStyleTotalSize;min_merge_width=2;max_read_amp=-1;size_ratio=1;} + ttl=2592000 + periodic_compaction_seconds=0 + preclude_last_level_data_seconds=0 + blob_file_size=268435456 + enable_blob_garbage_collection=false + cf_allow_ingest_behind=false + min_write_buffer_number_to_merge=1 + sst_partitioner_factory=nullptr + num_levels=7 + disallow_memtable_writes=false + force_consistency_checks=true + memtable_insert_with_hint_prefix_extractor=nullptr + memtable_factory=SkipListFactory + optimize_filters_for_hits=false + level_compaction_dynamic_level_bytes=true + compaction_style=kCompactionStyleLevel + compaction_filter=nullptr + default_temperature=kUnknown + inplace_update_support=false + merge_operator=nullptr + bloom_locality=0 + comparator=leveldb.BytewiseComparator + compaction_filter_factory=nullptr + max_write_buffer_size_to_maintain=0 + compaction_pri=kMinOverlappingRatio + persist_user_defined_timestamps=true + +[TableOptions/BlockBasedTable "events_offsets"] + fail_if_no_udi_on_open=false + initial_auto_readahead_size=8192 + max_auto_readahead_size=262144 + metadata_cache_options={unpartitioned_pinning=kFallback;partition_pinning=kFallback;top_level_index_pinning=kFallback;} + block_align=false + read_amp_bytes_per_bit=0 + verify_compression=false + detect_filter_construct_corruption=false + whole_key_filtering=true + user_defined_index_factory=nullptr + filter_policy=nullptr + super_block_alignment_space_overhead_ratio=128 + use_delta_encoding=true + optimize_filters_for_memory=true + partition_filters=false + prepopulate_block_cache=kDisable + pin_top_level_index_and_filter=true + index_block_restart_interval=1 + block_size_deviation=10 + num_file_reads_for_auto_readahead=2 + format_version=6 + decouple_partitioned_filters=true + checksum=kXXH3 + block_size=4096 + data_block_hash_table_util_ratio=0.750000 + index_shortening=kShortenSeparators + block_restart_interval=16 + data_block_index_type=kDataBlockBinarySearch + index_type=kBinarySearch + super_block_alignment_size=0 + metadata_block_size=4096 + pin_l0_filter_and_index_blocks_in_cache=false + no_block_cache=false + cache_index_and_filter_blocks_with_high_priority=true + cache_index_and_filter_blocks=false + enable_index_compression=true + flush_block_policy_factory=FlushBlockBySizePolicyFactory + +``` diff --git a/cmd/stellar-rpc/scripts/bench-fullhistory/sources.go b/cmd/stellar-rpc/scripts/bench-fullhistory/sources.go index 974be32a4..985f05c02 100644 --- a/cmd/stellar-rpc/scripts/bench-fullhistory/sources.go +++ b/cmd/stellar-rpc/scripts/bench-fullhistory/sources.go @@ -11,12 +11,15 @@ import ( "context" "errors" "fmt" + "io" "iter" "os" "time" "github.com/stellar/go-stellar-sdk/ingest/ledgerbackend" "github.com/stellar/go-stellar-sdk/support/datastore" + supportlog "github.com/stellar/go-stellar-sdk/support/log" + "github.com/stellar/go-stellar-sdk/xdr" chunkPkg "github.com/stellar/stellar-rpc/cmd/stellar-rpc/internal/fullhistory/pkg/chunk" "github.com/stellar/stellar-rpc/cmd/stellar-rpc/internal/fullhistory/pkg/stores/ledger" @@ -26,8 +29,38 @@ import ( const ( sourcePack = "pack" sourceBSB = "bsb" + sourceLCM = "lcm" ) +// lcmOpts configures the --source=lcm reader, which ingests a framed-XDR +// LedgerCloseMeta stream produced by stellar-core's `apply-load` +// (METADATA_OUTPUT_STREAM). It lets the synthetic-ledger driver feed +// apply-load output straight into the existing chunked ingest path. +// +// apply-load emits a run of setup ledgers (genesis + test-account creation, +// up to the "pre-benchmark checkpoint") followed by the dense benchmark +// ledgers. Checkpoint is the last setup ledger sequence: frames with +// seq <= Checkpoint are skipped so block 0 is the first benchmark ledger. +// BaseChunk is the chunk ID that maps to benchmark block 0 (i.e. the +// ingest's --chunk); chunk C then reads benchmark ledgers +// [(C-BaseChunk)*LedgersPerChunk, +LedgersPerChunk). +type lcmOpts struct { + file string + checkpoint uint32 + baseChunk chunkPkg.ID + // fixTxHashes repairs apply-load's tx-hash/envelope mismatch so the + // roundtrip ingest reader can consume the meta (see lcm_fixup.go). + fixTxHashes bool + // passphrase is the network passphrase used to recompute tx hashes during + // the fixup; it must match the bench reader's passphrase. + passphrase string + // allowPartial lets the final chunk be short (fewer than LedgersPerChunk): + // when the framed file ends before `want` ledgers, stop cleanly instead of + // erroring. This supports small synthetic runs sized to a TPS target rather + // than a full 10k-ledger chunk. + allowPartial bool +} + // packStream is a ledgerbackend.LedgerStream backed by a single cold packfile. // Like NewBufferedStorageStream it owns its lifecycle: each RawLedgers call // opens the chunk's ColdReader, yields each ledger's bytes, and closes the @@ -76,6 +109,194 @@ func (p *packStream) RawLedgers(_ context.Context, r ledgerbackend.Range) iter.S } } +// lcmStream is a ledgerbackend.LedgerStream backed by one framed-XDR +// LedgerCloseMeta file (apply-load's METADATA_OUTPUT_STREAM). Each chunk +// worker opens its own file handle, skips the setup ledgers and the chunks +// before it, then yields exactly LedgersPerChunk raw payloads for its block. +// +// The big inter-chunk skip is decode-free (read the 4-byte frame length, seek +// past the payload); only the small leading setup region is decoded, to find +// the first benchmark ledger by sequence. Yielded slices are borrowed (valid +// until the next iteration step), matching packStream's contract — the ingest +// driver copies what it retains. +type lcmStream struct { + opts lcmOpts + chunkID chunkPkg.ID + logger *supportlog.Entry +} + +// applyFixup runs the apply-load tx-hash fixup on one raw payload when enabled, +// accumulating stats into st. On any decode/encode error it returns the input +// unchanged (the ingester will surface the underlying problem downstream). +func (p *lcmStream) applyFixup(raw []byte, st *fixupStats) []byte { + if !p.opts.fixTxHashes { + return raw + } + out, s, err := fixupModelTxHashes(raw, p.opts.passphrase) + if err != nil { + if p.logger != nil { + p.logger.Warnf("lcm fixup decode failed (passing through): %v", err) + } + return raw + } + st.add(s) + return out +} + +var _ ledgerbackend.LedgerStream = (*lcmStream)(nil) + +func (p *lcmStream) RawLedgers(_ context.Context, r ledgerbackend.Range) iter.Seq2[[]byte, error] { + return func(yield func([]byte, error) bool) { + f, err := os.Open(p.opts.file) + if err != nil { + yield(nil, fmt.Errorf("open lcm file %s: %w", p.opts.file, err)) + return + } + defer func() { _ = f.Close() }() + + want := int(chunkPkg.LedgersPerChunk) + if r.Bounded() { + want = int(r.To() - r.From() + 1) + } + block := uint32(p.chunkID) - uint32(p.opts.baseChunk) + + // Position at the first benchmark ledger (first frame with + // seq > checkpoint), then frame-skip the blocks before this one. + first, firstPayload, ferr := p.seekFirstBenchmark(f) + if ferr != nil { + yield(nil, ferr) + return + } + // firstPayload holds benchmark index 0. Skip to block*want. + toSkip := int(block) * want + var buf []byte + switch { + case toSkip == 0: + buf = firstPayload // index 0 is the first ledger we yield + default: + // Discard index 0's payload; skip the remaining toSkip-1 + // frames decode-free, leaving the file at index toSkip. + _ = firstPayload + if serr := skipFrames(f, toSkip-1); serr != nil { + yield(nil, p.shortErr(serr, block)) + return + } + } + + var fx fixupStats + yielded := 0 + for i := 0; i < want; i++ { + var payload []byte + if i == 0 && buf != nil { + payload = buf + } else { + raw, rerr := readFrame(f, &buf) + if rerr != nil { + // End of the framed file. For the final/only chunk this is + // expected when the synthetic run was sized below a full + // chunk: yield what we have (if allowed) rather than error. + if p.opts.allowPartial && isEnd(rerr) { + break + } + yield(nil, p.shortErr(rerr, block)) + return + } + payload = raw + } + if !yield(p.applyFixup(payload, &fx), nil) { + return + } + yielded++ + } + if p.logger != nil { + if yielded < want { + p.logger.Infof("lcm chunk %d: short chunk — yielded %d of %d ledgers (file ended; sized below a full chunk)", + uint32(p.chunkID), yielded, want) + } + if p.opts.fixTxHashes { + p.logger.Infof("lcm chunk %d: tx-hash fixup — ledgers=%d txs=%d fixed=%d skipped=%d", + uint32(p.chunkID), fx.ledgers, fx.txs, fx.fixed, fx.skipped) + } + } + _ = first + } +} + +// isEnd reports whether err signals a clean end of the framed-XDR file. +func isEnd(err error) bool { + return errors.Is(err, io.EOF) || errors.Is(err, io.ErrUnexpectedEOF) +} + +// seekFirstBenchmark advances f past the setup ledgers (seq <= checkpoint) +// and returns the first benchmark ledger's sequence and its (already-read) +// payload. Only the setup region plus the first benchmark frame are decoded. +func (p *lcmStream) seekFirstBenchmark(f *os.File) (uint32, []byte, error) { + var buf []byte + for { + payload, err := readFrame(f, &buf) + if err != nil { + return 0, nil, fmt.Errorf("lcm %s: reached end before any benchmark ledger (checkpoint=%d): %w", + p.opts.file, p.opts.checkpoint, err) + } + var lcm xdr.LedgerCloseMeta + if uerr := lcm.UnmarshalBinary(payload); uerr != nil { + return 0, nil, fmt.Errorf("lcm %s: decode ledger header: %w", p.opts.file, uerr) + } + seq := lcm.LedgerSequence() + if seq > p.opts.checkpoint { + // First benchmark ledger. Copy the payload since buf is reused. + out := make([]byte, len(payload)) + copy(out, payload) + return seq, out, nil + } + } +} + +func (p *lcmStream) shortErr(err error, block uint32) error { + if errors.Is(err, io.EOF) || errors.Is(err, io.ErrUnexpectedEOF) { + return fmt.Errorf("lcm %s: not enough benchmark ledgers for chunk %d (block %d): each chunk needs %d full ledgers; "+ + "generate more apply-load ledgers (raise APPLY_LOAD_NUM_LEDGERS) or ingest fewer chunks: %w", + p.opts.file, uint32(p.chunkID), block, chunkPkg.LedgersPerChunk, err) + } + return fmt.Errorf("lcm %s chunk %d: %w", p.opts.file, uint32(p.chunkID), err) +} + +// readFrame reads one framed-XDR record (4-byte length prefix + payload) and +// returns the payload, reusing *bufp across calls. The returned slice is valid +// until the next readFrame call. +func readFrame(f *os.File, bufp *[]byte) ([]byte, error) { + n, err := xdr.ReadFrameLength(f) + if err != nil { + return nil, err + } + if cap(*bufp) < int(n) { + *bufp = make([]byte, n) + } + buf := (*bufp)[:n] + if _, err := io.ReadFull(f, buf); err != nil { + if errors.Is(err, io.EOF) { + err = io.ErrUnexpectedEOF + } + return nil, err + } + return buf, nil +} + +// skipFrames advances f past k framed-XDR records without decoding payloads +// (read the length prefix, seek past the payload). +func skipFrames(f *os.File, k int) error { + for range k { + n, err := xdr.ReadFrameLength(f) + if err != nil { + return err + } + if _, err := f.Seek(int64(n), io.SeekCurrent); err != nil { + return err + } + } + return nil +} + // BSBOpts is the per-stream BufferedStorageStream tuning, shared by the hot // driver (one stream) and each cold chunk worker (one stream per chunk). type BSBOpts struct { @@ -91,8 +312,19 @@ type BSBOpts struct { // buffered-storage stream opens/closes its datastore + backend per iteration. // Each call yields an INDEPENDENT stream, so concurrent chunk workers run fully // in parallel (independent ColdReaders / GCS prefetch pipelines). -func openChunkStream(source, coldDir, bucketPath string, opts BSBOpts, chunkID chunkPkg.ID) (ledgerbackend.LedgerStream, error) { +func openChunkStream(logger *supportlog.Entry, source, coldDir, bucketPath string, opts BSBOpts, lcm lcmOpts, chunkID chunkPkg.ID) (ledgerbackend.LedgerStream, error) { switch source { + case sourceLCM: + if lcm.file == "" { + return nil, errors.New("--lcm-file is required when --source=lcm") + } + if uint32(chunkID) < uint32(lcm.baseChunk) { + return nil, fmt.Errorf("--source=lcm: chunk %d is below base chunk %d", uint32(chunkID), uint32(lcm.baseChunk)) + } + if _, err := os.Stat(lcm.file); err != nil { + return nil, fmt.Errorf("lcm file missing: %s: %w", lcm.file, err) + } + return &lcmStream{opts: lcm, chunkID: chunkID, logger: logger}, nil case sourcePack: if coldDir == "" { return nil, errors.New("--cold-dir is required when --source=pack") @@ -118,6 +350,6 @@ func openChunkStream(source, coldDir, bucketPath string, opts BSBOpts, chunkID c } return ledgerbackend.NewBufferedStorageStream(cfg, dsConfig, nil), nil default: - return nil, fmt.Errorf("--source=%s; expected pack|bsb", source) + return nil, fmt.Errorf("--source=%s; expected pack|bsb|lcm", source) } } diff --git a/cmd/stellar-rpc/scripts/bench-fullhistory/synthetic-run.sh b/cmd/stellar-rpc/scripts/bench-fullhistory/synthetic-run.sh new file mode 100755 index 000000000..a7681fa97 --- /dev/null +++ b/cmd/stellar-rpc/scripts/bench-fullhistory/synthetic-run.sh @@ -0,0 +1,97 @@ +#!/usr/bin/env bash +# +# synthetic-run.sh — end-to-end driver: generate synthetic cold stores for one +# or more apply-load profiles, then run the read bench suite on them. +# +# for each PROFILE: apply-load-gen.sh (apply-load -> meta -> cold packfiles) +# then: bench-suite.sh (cold-* / hot-* read benches -> CSVs) +# +# Profiles run SEQUENTIALLY by default. This matters: each dense apply-load holds +# in-memory soroban state that GROWS with ledger count (~8.5 MB/ledger at +# 6000 SAC tx/ledger), so running profiles in parallel multiplies peak RAM and +# can OOM. See MEMORY below. +# +# ---- MEMORY (read this before picking NUM_LEDGERS) -------------------------- +# Peak RSS ≈ density(tx/ledger) × NUM_LEDGERS × ~1.4 KB/tx of live state. +# Measured on c6id.8xlarge (61 GB): sac @6000 tx/ledger hit ~32 GB at 3,760 +# ledgers and projected ~85 GB at 10,000 -> OOM. Rough guidance per profile: +# RAM 61 GB -> sac/token ~6,000 ledgers; soroswap ~20,000 (it's 1,500 tx/ledger) +# RAM 128 GB -> sac/token ~14,000; soroswap full chunks easily +# RAM 256 GB -> sac/token ~28,000 (3 chunks); etc. +# A full 10k-ledger chunk of 10k-TPS SAC needs ~96-128 GB RAM. Size NUM_LEDGERS +# to your box, or run PARALLEL=1 only when total peak RSS fits in RAM. +# +# ---- USAGE ----------------------------------------------------------------- +# CORE_BIN=/usr/bin/stellar-core \ # a BUILD_TESTS (~buildtests) core +# OUT_ROOT=/mnt/nvme/synth \ # work + cold output (use fast local disk) +# PROFILES="sac token soroswap" \ +# NUM_LEDGERS=6000 \ # per profile; size to RAM (see above) +# PARALLEL=0 \ # 1 = all profiles at once (watch RAM!) +# GCS_DEST=gs://bucket/path \ # optional: upload cold stores + results +# ./synthetic-run.sh +# +# BENCH_BIN is auto-built from this dir if unset (needs Go + the RocksDB cgo deps; +# see SYNTHETIC-LEDGERS.md). On Linux, export LD_LIBRARY_PATH to the RocksDB .so. +set -uo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +CORE_BIN="${CORE_BIN:-$(command -v stellar-core || true)}" +OUT_ROOT="${OUT_ROOT:-./synthetic-out}" +PROFILES="${PROFILES:-sac token soroswap}" +NUM_LEDGERS="${NUM_LEDGERS:-6000}" +CLUSTERS="${CLUSTERS:-8}" +PARALLEL="${PARALLEL:-0}" +RUN_BENCH="${RUN_BENCH:-1}" +GCS_DEST="${GCS_DEST:-}" +export CLOSE_TIME_MS="${CLOSE_TIME_MS:-600}" +export KEEP_META="${KEEP_META:-0}" + +log(){ printf '\033[1;36m[synthetic-run]\033[0m %s\n' "$*" >&2; } +die(){ printf '\033[1;31m[synthetic-run] ERROR:\033[0m %s\n' "$*" >&2; exit 1; } + +[ -n "$CORE_BIN" ] || die "stellar-core not found; set CORE_BIN (must be a ~buildtests build with apply-load)" +mkdir -p "$OUT_ROOT"; OUT_ROOT="$(cd "$OUT_ROOT" && pwd)" + +# Build the bench binary once if not provided. +if [ -z "${BENCH_BIN:-}" ]; then + BENCH_BIN="$SCRIPT_DIR/bench-fullhistory" + log "building bench-fullhistory -> $BENCH_BIN" + ( cd "$SCRIPT_DIR" && go build -o "$BENCH_BIN" . ) || die "go build failed (see SYNTHETIC-LEDGERS.md for cgo/RocksDB setup)" +fi +export CORE_BIN BENCH_BIN OUT_ROOT CLUSTERS NUM_LEDGERS + +gen_one(){ + local P="$1" + log "generate $P (num_ledgers=$NUM_LEDGERS clusters=$CLUSTERS close_ms=$CLOSE_TIME_MS)" + PROFILE="$P" "$SCRIPT_DIR/apply-load-gen.sh" > "$OUT_ROOT/$P.gen.log" 2>&1 + log "$P generate exit=$? (log: $OUT_ROOT/$P.gen.log)" +} + +log "START $(date -u +%FT%TZ) profiles='$PROFILES' parallel=$PARALLEL out=$OUT_ROOT mem-free=$(free -g 2>/dev/null|awk '/Mem/{print $4}')G" +if [ "$PARALLEL" = "1" ]; then + log "PARALLEL=1: ensure combined peak RSS fits in RAM (see MEMORY note)" + pids=(); for P in $PROFILES; do gen_one "$P" & pids+=($!); done + wait "${pids[@]}" +else + for P in $PROFILES; do gen_one "$P"; done +fi +log "generation done $(date -u +%FT%TZ)" + +if [ "$RUN_BENCH" = "1" ]; then + RESULTS="${RESULTS:-$OUT_ROOT/bench-results/run-$(date -u +%Y%m%dT%H%M%SZ)}" + log "bench suite -> $RESULTS" + ROOT="$OUT_ROOT" RESULTS="$RESULTS" PROFILES="$PROFILES" BENCH_BIN="$BENCH_BIN" \ + bash "$SCRIPT_DIR/bench-suite.sh" || log "bench-suite returned nonzero" +fi + +if [ -n "$GCS_DEST" ]; then + command -v gsutil >/dev/null || die "GCS_DEST set but gsutil not found" + log "uploading cold stores + results to $GCS_DEST" + for P in $PROFILES; do + [ -d "$OUT_ROOT/$P/cold" ] && gsutil -m cp -r "$OUT_ROOT/$P/cold" "$GCS_DEST/$P/cold" + [ -f "$OUT_ROOT/$P/work/apply-load.cfg" ] && gsutil cp "$OUT_ROOT/$P/work/apply-load.cfg" "$GCS_DEST/$P/apply-load.cfg" + done + [ -d "${RESULTS:-}" ] && gsutil -m cp -r "$RESULTS" "$GCS_DEST/bench-results" + log "upload done -> $GCS_DEST" +fi +log "DONE $(date -u +%FT%TZ)" diff --git a/cmd/stellar-rpc/scripts/bench-fullhistory/tx_hash_helpers.go b/cmd/stellar-rpc/scripts/bench-fullhistory/tx_hash_helpers.go index 0faf12d8f..4aa6c6b5d 100644 --- a/cmd/stellar-rpc/scripts/bench-fullhistory/tx_hash_helpers.go +++ b/cmd/stellar-rpc/scripts/bench-fullhistory/tx_hash_helpers.go @@ -44,7 +44,21 @@ func sampleHashesFromCold( } defer r.Close() + // Clamp the requested [first,last] to what the pack actually holds. A chunk + // generated from a synthetic run sized below LedgersPerChunk is a partial + // chunk: its real ledgers occupy only the start of the nominal chunk range, + // so sampling a random seq across the full nominal span would hit holes. + if fs, ferr := r.FirstSeq(); ferr == nil && fs > first { + first = fs + } + if ls, lerr := r.LastSeq(); lerr == nil && ls < last { + last = ls + } + span := int(last - first + 1) + if span < 1 { + return nil, nil + } if nLedgers > span { nLedgers = span }