Skip to content
Draft
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -182,4 +182,5 @@ __pycache__/
sentinel.toml
sentinel_state.json
gravity_e2e/cluster_test_cases/bridge/mock_anvil_metadata.json
gravity_e2e/cluster_test_cases/oracle_live_soak/artifacts/
bench_reports/
163 changes: 163 additions & 0 deletions gravity_e2e/cluster_test_cases/oracle_live_soak/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,163 @@
# Wave 7 Binance Price Feed Live Soak

This manual, non-gating suite runs four equal-power Gravity validators against
three continuous Binance Futures testnet closed index-price kline feeds:

- `NVDAUSDT` (`sourceType=3`, feed ID `1001`);
- `BTCUSDT` (`sourceType=3`, feed ID `1002`);
- `ETHUSDT` (`sourceType=3`, feed ID `1003`).

The suite submits one Gravity governance proposal that registers the three
price tasks and the `PriceFeedResolver` callback. The same proposal changes the
epoch interval from the 60-second bootstrap value to two hours for `E+1`. It
proves that observers remain absent in proposal epoch `E`, start in `E+1`, and
reach a three-of-four JWK quorum before the soak timer begins.

This is the initial Binance-only launch gate tracked by gravity-audit #1093.
The suite does not enable source type 6, deploy or register a Polymarket
resolver, create a Polymarket task or callback, require Polygon credentials, or
write a `gravity://6/...` validator mapping. The merged deterministic
Polymarket transport suite remains separate regression coverage for a possible
future activation.

The short bootstrap epoch keeps dynamic task activation fast. The two-hour soak
epoch intentionally follows the repository's regular four-validator topology,
so a long Oracle soak does not also become a minute-by-minute reconfiguration
stress test.

## What Every Heartbeat Checks

- all four Gravity RPC replicas expose the same block hash at the common
confirmed height `min(latest node heights) - 16`;
- each replica returns the same NativeOracle progress and PriceFeedResolver
state at that exact EIP-1898 canonical block hash;
- bounded whole-snapshot retries tolerate a transient execution-view lag, while
a state difference that persists across the retry window fails the soak;
- every price snapshot uses a progress/resolver/progress seqlock read;
- each feed's delivery nonce, source position, and resolver round never regress;
- each latest resolver value maps to the pair's exact closed one-minute bucket;
- all three feeds and every Gravity node stay within the stall budget;
- at least three relayers have checkpointed each committed Binance nonce;
- the governance-pending two-hour epoch interval is active before timing starts.

At completion, the suite fetches each exact final Binance bucket again and
compares its close with the corresponding onchain value. Each feed's
NativeOracle callback count must equal its independent final delivery nonce.
Observed price changes remain informational because timely identical index
closes are valid Oracle updates.

For runs of at least one hour, `node4` becomes eligible to restart halfway
through by default. A focused run can configure multiple restart times. The
runner defers each restart while the chain is within five minutes of an epoch
boundary, then requires RPC, block height, and all three relayer checkpoints to
catch up without nonce regression.

## Prerequisites

- Current quick-release `gravity_node` and `gravity_cli` binaries.
- Foundry `forge` for the PriceFeedResolver artifact.
- Approved outbound access to Binance Futures testnet.

The Binance Futures testnet `indexPriceKlines` endpoint is public and does not
require an API key. Each validator independently fetches all three pairs. The
values are testnet index data and must be labeled that way in any demo. The
generated relayer mapping and source metadata live under the ignored suite
`artifacts/` directory and are removed during normal teardown.

The runner performs a global local `gravity_node` cleanup before and after the
suite. Do not run it beside another local Gravity cluster that must stay alive.

## Build

From the SDK repository root:

```bash
make MODE=quick-release gravity_node gravity_cli
export PATH="$HOME/.foundry/bin:$PWD/target/quick-release:$PATH"
```

## Quick Burn-In

Use a short no-restart run after changing task activation, payload handling, or
the price resolver:

```bash
ORACLE_SOAK_DURATION_SECONDS=1800 \
ORACLE_SOAK_POLL_SECONDS=15 \
ORACLE_SOAK_STALL_TIMEOUT_SECONDS=900 \
ORACLE_SOAK_MIN_ADVANCES=24 \
ORACLE_SOAK_RESTART_AFTER_SECONDS=0 \
./gravity_e2e/run_test.sh \
oracle_live_soak \
--force-init \
--log-cli-level=INFO
```

## Four-Hour Two-Restart Gate

This is the focused release-profile stability run. It schedules node4 restarts
one and three hours into the monitored four-hour period, away from the expected
two-hour epoch boundaries:

```bash
ORACLE_SOAK_DURATION_SECONDS=14400 \
ORACLE_SOAK_POLL_SECONDS=15 \
ORACLE_SOAK_STALL_TIMEOUT_SECONDS=900 \
ORACLE_SOAK_MIN_ADVANCES=192 \
ORACLE_SOAK_RESTART_SCHEDULE_SECONDS=3600,10800 \
./gravity_e2e/run_test.sh \
oracle_live_soak \
--force-init \
--log-cli-level=INFO
```

The summary must contain two entries in `restartRecoveries`; the final snapshot
must still show all four replicas, every relayer checkpoint caught up, and every
price callback count equal to its source nonce.

## Optional 24-Hour Soak

Run from a terminal multiplexer or another session that will stay alive:

```bash
ORACLE_SOAK_DURATION_SECONDS=86400 \
ORACLE_SOAK_POLL_SECONDS=15 \
ORACLE_SOAK_STALL_TIMEOUT_SECONDS=900 \
./gravity_e2e/run_test.sh \
oracle_live_soak \
--force-init \
--log-cli-level=INFO
```

With the defaults, `node4` becomes eligible to restart after 12 hours and runs
as soon as the five-minute epoch-transition guard is open. The pytest process
is the monitor, performs final assertions, writes the report, and tears down the
cluster. A long-lived node can spend more than the harness's usual 30 seconds
reopening its persisted databases, so this suite gives restart RPC recovery a
three-minute window while still failing if the process exits.

## Controls

| Variable | Default | Meaning |
| --- | ---: | --- |
| `ORACLE_SOAK_DURATION_SECONDS` | `86400` | Monitored soak duration |
| `ORACLE_SOAK_POLL_SECONDS` | `15` | Onchain heartbeat interval |
| `ORACLE_SOAK_STALL_TIMEOUT_SECONDS` | `900` | Maximum feed or chain stall |
| `ORACLE_SOAK_MIN_ADVANCES` | 80% of expected minutes | Required nonce advances for each Binance feed |
| `ORACLE_SOAK_RESTART_AFTER_SECONDS` | Halfway for runs >= 1 hour | Earliest restart time; `0` disables |
| `ORACLE_SOAK_RESTART_SCHEDULE_SECONDS` | unset | Strictly increasing comma-separated restart times; mutually exclusive with `ORACLE_SOAK_RESTART_AFTER_SECONDS` |
| `ORACLE_SOAK_RESTART_NODE` | `node4` | Validator selected for restart |
| `ORACLE_SOAK_RESTART_RPC_TIMEOUT_SECONDS` | `180` | RPC recovery window for the restarted validator |
| `BINANCE_PRICE_FEED_BASE_URL` | `https://testnet.binancefuture.com` | Binance Futures testnet base URL |
| `BINANCE_PRICE_FEED_GRACE_MS` | `120000` | Closed-bucket safety delay |

## Evidence

Runtime evidence remains local and ignored by Git:

- `artifacts/oracle_live_soak_heartbeat.jsonl`: one bounded checkpoint per poll;
- `artifacts/oracle_live_soak_summary.json`: final PASS/FAIL report and restart
recovery times.

The summary is the acceptance artifact. A process that is still running has not
yet passed the soak.
73 changes: 73 additions & 0 deletions gravity_e2e/cluster_test_cases/oracle_live_soak/cluster.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,73 @@
# Four-validator live Binance price-feed soak test.
# Keep fixed listeners below Linux's default ephemeral source-port range.

[cluster]
name = "gravity-devnet-oracle-live-soak"
base_dir = "/tmp/gravity-cluster-oracle-live-soak"

[genesis_source]
genesis_path = "./artifacts/genesis.json"
waypoint_path = "./artifacts/waypoint.txt"

[[nodes]]
id = "node1"
role = "genesis"
source = { project_path = "../" }
host = "127.0.0.1"
validator_port = 26680
vfn_port = 26690
rpc_port = 26700
api_port = 26710
metrics_port = 26720
inspection_port = 26730
https_port = 26740
authrpc_port = 26750
reth_p2p_port = 26760

[[nodes]]
id = "node2"
role = "genesis"
source = { project_path = "../" }
host = "127.0.0.1"
validator_port = 26681
vfn_port = 26691
rpc_port = 26701
api_port = 26711
metrics_port = 26721
inspection_port = 26731
https_port = 26741
authrpc_port = 26751
reth_p2p_port = 26761

[[nodes]]
id = "node3"
role = "genesis"
source = { project_path = "../" }
host = "127.0.0.1"
validator_port = 26682
vfn_port = 26692
rpc_port = 26702
api_port = 26712
metrics_port = 26722
inspection_port = 26732
https_port = 26742
authrpc_port = 26752
reth_p2p_port = 26762

[[nodes]]
id = "node4"
role = "genesis"
source = { project_path = "../" }
host = "127.0.0.1"
validator_port = 26683
vfn_port = 26693
rpc_port = 26703
api_port = 26713
metrics_port = 26723
inspection_port = 26733
https_port = 26743
authrpc_port = 26753
reth_p2p_port = 26763

[faucet_init]
num_accounts = 0
105 changes: 105 additions & 0 deletions gravity_e2e/cluster_test_cases/oracle_live_soak/genesis.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,105 @@
# Source type 3 is enabled at genesis, while its concrete price-feed tasks are
# registered by governance during the test. Source type 6 stays disabled for
# the initial Binance-only launch profile.

[dependencies.genesis_contracts]
repo = "https://github.com/Galxe/gravity_chain_core_contracts.git"
ref = "f447687bf6f6f9efabd92626b588aeda94148f88"

# Equal voting power means a JWK QC requires matching signatures from at least
# three of the four validators.
[[genesis_validators]]
id = "node1"
address = "0xf39Fd6e51aad88F6F4ce6aB8827279cffFb92266"
host = "127.0.0.1"
validator_port = 26680
vfn_port = 26690
stake_amount = "2000000000000000000"
voting_power = "2000000000000000000"
consensus_pop = "0x000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000"

[[genesis_validators]]
id = "node2"
address = "0x7b254Bd44F6CE45e00a912b2460D47F3Be56fAD7"
host = "127.0.0.1"
validator_port = 26681
vfn_port = 26691
stake_amount = "2000000000000000000"
voting_power = "2000000000000000000"
consensus_pop = "0x000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000"

[[genesis_validators]]
id = "node3"
address = "0x9B2C25E77a97d3e84DC0Cb7F83fb676ddC4F24b9"
host = "127.0.0.1"
validator_port = 26682
vfn_port = 26692
stake_amount = "2000000000000000000"
voting_power = "2000000000000000000"
consensus_pop = "0x000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000"

[[genesis_validators]]
id = "node4"
address = "0x18c23753385ce7A60B15d171302E48b6AFf0BDC5"
host = "127.0.0.1"
validator_port = 26683
vfn_port = 26693
stake_amount = "2000000000000000000"
voting_power = "2000000000000000000"
consensus_pop = "0x000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000"

[genesis]
chain_id = 1337
epoch_interval_micros = 60000000
major_version = 1
consensus_config = "0x0301010a00000000000000280000000000000001010000000a000000000000000100010200000000000000000020000000000000"
execution_config = "0x00"
initial_locked_until_micros = 1798848000000000
governance_owner = "0xf39Fd6e51aad88F6F4ce6aB8827279cffFb92266"

[genesis.faucet]
address = "0xf39Fd6e51aad88F6F4ce6aB8827279cffFb92266"
balance = "0x2000000000000000000000000000000000000000000000000000000000000000"

[genesis.validator_config]
minimum_bond = "1000000000000000000"
maximum_bond = "1000000000000000000000000"
unbonding_delay_micros = 604800000000
allow_validator_set_change = true
voting_power_increase_limit_pct = 20
max_validator_set_size = "100"
auto_evict_enabled = false
auto_evict_threshold_pct = 0

[genesis.staking_config]
minimum_stake = "1000000000000000000"
lockup_duration_micros = 86400000000
unbonding_delay_micros = 86400000000

[genesis.governance_config]
min_voting_threshold = "1000000000000000000"
required_proposer_stake = "1000000000000000000"
voting_duration_micros = 5000000

[genesis.randomness_config]
variant = 1
secrecy_threshold = 9223372036854775808
reconstruction_threshold = 12297829382473033728
fast_path_secrecy_threshold = 12297829382473033728

[genesis.oracle_config]
source_types = [1, 3]
callbacks = [
"0x00000000000000000000000000000001625F4001",
"0x0000000000000000000000000000000000000000",
]

[genesis.jwk_config]
issuers = ["0x68747470733a2f2f6163636f756e74732e676f6f676c652e636f6d"]

[[genesis.jwk_config.jwks]]
kid = "f5f4c0ae6e6090a65ab0a694d6ba6f19d5d0b4e6"
kty = "RSA"
alg = "RS256"
e = "AQAB"
n = "2K7epoJWl_aBoYGpXmDBBiEnwQ0QdVRU1gsbGXNrEbrZEQdY5KjH5P5gZMq3d3KvT1j5KsD2tF_9jFMDLqV4VWDNJRLgSNJxhJuO_oLO2BXUSL9a7fLHxnZCUfJvT2K-O8AXjT3_ZM8UuL8d4jBn_fZLzdEI4MHrZLVSaHDvvKqL_mExQo6cFD-qyLZ-T6aHv2x8R7L_3X7E1nGMjKVVZMveQ_HMeXvnGxKf5yfEP0hIQlC_kFm4L_1kV1S0UPmMptZL2qI4VnXqmqI6TZJyE-3VXHgNn1Z1O_9QZlPC0fF0spLHf2S3nNqI0v3k2E7q3DkqxVf5xvn7q_X-gPqzVE9Jw"
Loading
Loading