From d754abe37047d6faa068b7faffcc5f5a01c4babd Mon Sep 17 00:00:00 2001 From: marioevz Date: Wed, 5 Aug 2026 14:59:55 -0600 Subject: [PATCH 1/3] Calibration pass 1 --- complexity_assessments/calibration/README.md | 212 ++++++++ .../calibration/amsterdam-dataset.json | 405 +++++++++++++++ .../calibration/calibrate.py | 478 ++++++++++++++++++ .../calibration/proposed-anchors.md | 266 ++++++++++ .../calibration/proposed-anchors.py | 184 +++++++ 5 files changed, 1545 insertions(+) create mode 100644 complexity_assessments/calibration/README.md create mode 100644 complexity_assessments/calibration/amsterdam-dataset.json create mode 100644 complexity_assessments/calibration/calibrate.py create mode 100644 complexity_assessments/calibration/proposed-anchors.md create mode 100644 complexity_assessments/calibration/proposed-anchors.py diff --git a/complexity_assessments/calibration/README.md b/complexity_assessments/calibration/README.md new file mode 100644 index 0000000..27307e1 --- /dev/null +++ b/complexity_assessments/calibration/README.md @@ -0,0 +1,212 @@ +# Complexity Score Calibration — Test Effort Units (TEU) + +Links the **complexity assessment score** of an EIP to the **measured testing work** it +generated in [`ethereum/execution-specs`](https://github.com/ethereum/execution-specs). + +Calibrated against Amsterdam, 2025-10-21 (the EEST → execution-specs weld) through +2026-08-05. 917 squash-merged commits on `forks/amsterdam`, 15 EIP test suites, +12 of which have a complexity assessment on file. + +--- + +## 1. The metric + +**Test Effort Units (TEU)** — the observable cost of testing one EIP: + +``` +TEU = merged_PRs + (review_submissions + review_threads) / 10 +``` + +One TEU ≈ one merged PR's worth of work. The review term adds the PRs that +were expensive to land, which PR count alone misses. + +Both components come from PRs *owned* by the EIP (see [attribution](#4-attribution-model)). +Neither depends on how tests happen to be written, which is why they beat the +obvious alternatives — see [§5](#5-why-not-the-obvious-measures). + +## 2. The calibration + +``` +TEU = 0.38 × S^1.62 +``` + +where `S` is the assessment's Total Score. + +| | | +|---|---| +| Fit sample | n = 9 (EIPs ≥ 10 weeks in flight) | +| Pearson r | **0.920** | +| Leave-one-out median error | **21 %** | +| Same fit, linear model | 148 % LOO error — the power law is **7× more accurate** | +| Fork-level closure | Σ per-EIP predictions = **320 TEU** vs **328 measured** (2.5 % off) | + +### The headline: cost is super-linear in score + +**Doubling the complexity score triples the work** (2^1.62 ≈ 3.1×). + +Every measure of work independently shows an exponent above 1: + +| Work measure | Exponent | +|---|---:| +| Review load (submissions + threads) | 2.02 | +| Diff churn (lines added + deleted) | 1.88 | +| **TEU** | **1.62** | +| Test functions written | 1.48 | +| Merged PRs | 1.46 | +| Final test-suite LoC | 1.32 | + +Review attention scales worst of all. Test-writing volume grows roughly with +`S^1.5`; the *arguing about it* grows with `S^2`. + +**What the data actually shows is a cliff at the red tier, not a smooth curve.** +Fit only the seven mature 🟢/🟡 EIPs (scores 5–13) and the exponent collapses to +**0.78** — inside that band, cost is flat to slightly sub-linear in score. The +1.62 exponent is produced entirely by the jump to EIP-7928 (29) and EIP-8037 (28). +So read the power law as a smooth interpolation across a step, not as evidence +that each additional point costs more than the last. Both readings give the same +planning numbers at the anchors; they disagree in the 16–25 gap, where Amsterdam +has no observations at all. + +## 3. Planning table + +Fitted on mature Amsterdam EIPs. Read across for the workload a score implies. + +| Score | Tier | TEU | PRs | Review items | Test funcs | Suite LoC | Churn | vs. a 5-pt EIP | +|---:|:--:|---:|---:|---:|---:|---:|---:|---:| +| 3 | 🟢 | 2.3 | 1.8 | 4 | 6 | 600 | 630 | 0.4× | +| 5 | 🟢 | 5.2 | 3.7 | 12 | 12 | 1 180 | 1 660 | 1.0× | +| 8 | 🟢 | 11.1 | 7.3 | 32 | 24 | 2 200 | 4 010 | 2.1× | +| 10 | 🟡 | 16.0 | 10.2 | 50 | 34 | 2 960 | 6 100 | 3.1× | +| 13 | 🟡 | 24.5 | 14.9 | 85 | 50 | 4 180 | 9 990 | 4.7× | +| 16 | 🟡 | 34.2 | 20.2 | 129 | 68 | 5 510 | 14 750 | 6.6× | +| 20 | 🔴 | 49.2 | 28.0 | 203 | 95 | 7 400 | 22 440 | 9.4× | +| 25 | 🔴 | 70.6 | 38.8 | 319 | 132 | 9 930 | 34 130 | 13.6× | +| 30 | 🔴 | 95.0 | 50.7 | 462 | 172 | 12 640 | 48 090 | 18.2× | +| 36 | 🔴 | 127.4 | 66.2 | 668 | 226 | 16 090 | 67 740 | 24.5× | + +**A single 🔴 EIP at the top of the range costs more than the entire 🟢 tier of a +fork combined.** In Amsterdam, EIP-7928 and EIP-8037 were 39 % of the score +total but **70 % of the measured TEU**. + +### Using it for a fork + +Apply the curve **per EIP, then sum**. Never apply it to a summed score — the +super-linearity is a property of one EIP's internal interactions, not of a fork's +EIP list. Amsterdam: per-EIP sum gives 320 TEU (correct); the summed-score +shortcut gives 1 217 TEU (~4× too high). + +## 4. Attribution model + +Each squash-merged commit on `forks/amsterdam` is one work unit, assigned to +owner EIPs: + +``` +owners = EIPs named in the commit subject + else EIPs whose tests/amsterdam/eipNNNN_* directory the unit touched +``` + +Title takes precedence on purpose. PR #2901 (*"merge EIP-8037 to +`forks/amsterdam`"*, 910 files, 159 review submissions) touched 8 EIPs' test +directories because a gas-cost change breaks everyone's tests. It is 8037's +cost, not a cost smeared across its victims. Multi-owner units split `1/n`. + +Three churn classes are tracked separately: + +- **direct** — inside the EIP's own `tests/amsterdam/eipNNNN_*` directory +- **ripple** — anywhere else, from units the EIP owns (EELS, framework, other forks' tests) +- **generated** — `tests/ported_static/`, `tests/static/`; machine-ported, 2.3 M lines of + unattributed churn that would otherwise swamp every real signal + +Ripple churn matters: EIP-7928 wrote 19 k lines inside its own directory and +caused **68 k lines** of change everywhere else. + +## 5. Why not the obvious measures + +| Measure | r (mature) | Verdict | +|---|---:|---| +| Review load alone | 0.969 | Best single predictor, but noisy on quiet EIPs | +| Final suite LoC | 0.961 | Excellent, but rewards verbose test code | +| Test functions | 0.932 | Style-dependent (see below) | +| **TEU (PRs + review/10)** | **0.920** | **Chosen** — lowest LOO error, style-independent | +| Diff churn | 0.882 | Highest churn-side LOO error (59 %); rebases and moves inflate it | +| **Filled test cases** | **0.674** | **Do not use** — 72 % LOO error, exponent 0.80 | + +**Filled test cases are actively misleading.** Spearman ρ = 0.36 — barely +better than chance. EIP-8246 (trivial, 2 test functions, no assessment) +generates 1 302 cases; EIP-7778 (score 10) generates 54. One +`@pytest.mark.parametrize` decorator can produce a thousand vectors for free, +so vector count measures parametrization style, not work. + +Test-function counts have a milder version of the same problem — EIP-8024 wrote +68 functions on a score of 6, EIP-7976 wrote 10 on a score of 5 while generating +1 598 cases. TEU avoids both by counting process events instead of code. + +## 6. Which anchors are actually earning their keep + +Per-anchor scores parsed from all 12 assessments and correlated against measured +work. *The total beats every individual row and every thematic group* — the +checklist as a whole is the signal, which is the strongest possible endorsement +of its current design. + +| Anchor / group | r vs churn | r vs test funcs | n>0 | +|---|---:|---:|---:| +| **TOTAL (all rows)** | **0.83** | **0.91** | 12 | +| Edge/boundary conditions | 0.63 | 0.77 | 9 | +| Cross-EIP interactions | 0.73 | 0.75 | 3 | +| Patterns affecting pre-existing tests | 0.36 | 0.76 | 9 | +| EVM Gas rule changes | 0.46 | 0.77 | 7 | +| *group:* risk & edges | 0.70 | 0.56 | — | +| *group:* blast radius | 0.54 | 0.85 | — | +| *group:* gas & metering | 0.31 | 0.68 | — | + +Notes: + +- **"Patterns affecting pre-existing tests" is a volume driver, not a churn + driver** (0.76 vs test functions, 0.36 vs churn). Breaking existing tests means + writing many more cases, not rewriting more lines than usual. +- **"Edge/boundary conditions"** is the best-populated high-signal row and the + single most broadly useful anchor. +- Six anchors were **never scored above 0** in Amsterdam: added precompiles, + modified precompiles, cryptography-related testing, new transaction types, + encoding changes (RLP/SSZ), Engine API encoding changes. No evidence either + way — Amsterdam simply has no precompile or encoding work. Do not prune them + on this data. +- Rows with `n>0 ≤ 2` (block syncing at r = 0.93, modified system contracts, + new header fields) are single-observation artifacts. Ignore the numbers. + +## 7. Caveats + +1. **n = 12, one fork, and two points carry the exponent.** Every number here is + Amsterdam-shaped. Drop EIP-7928 and EIP-8037 and the exponent falls from 1.62 + to **0.78**. The power law is the better *model* by a wide margin (7× better + LOO error than linear), but that verdict rests on two observations at the top + of the range. Treat the curve as reliable *at* the tier anchors and + provisional *between* them until a second fork is measured. +2. **Discrimination is poor within a tier.** Score vs PRs is r = 0.80 over the + full range but only **r = 0.36** once the two reds are removed. The assessment + reliably separates 🔴 from 🟢; it does not reliably rank a 5 against a 7. Use + the tier, not the point value, for anything below 20. +3. **Truncation.** EIP-8038 (score 20, 4.7 weeks in flight), EIP-8070 (7, + 0.9 weeks) and EIP-7997 (5, 4.9 weeks) are excluded from the fit — their work + has barely started, and they are exactly the three largest + under-predictions. Re-run after Amsterdam ships. +4. **Score inflation risk.** These are the *original* assessment scores. If + scores are ever revised after test work begins, the correlation becomes + self-fulfilling. Freeze the score at CFI time and record the frozen value. +5. **TEU is not engineer-hours.** Squash merges collapse each PR to one commit, + so per-author-day counts degenerate to PR counts and no honest + effort-in-hours figure is recoverable from git. TEU is a workload *index*, + comparable across EIPs but not convertible to headcount without time tracking. +6. **Three Amsterdam suites have no assessment on file** — EIP-7954, EIP-8282, + EIP-8246. Their measured TEU back-solves to scores of ~6, ~5 and ~4. + +## 8. Reproducing + +```sh +# from a clone of ethereum/execution-specs with `upstream` pointing at the repo +python3 calibrate.py --specs-repo ~/path/to/execution-specs --fork amsterdam +``` + +`calibrate.py` regenerates `amsterdam-dataset.json` (per-EIP raw measures) and +every table above. It needs `gh` authenticated for PR metadata, and the +`execution-specs` venv only if you pass `--collect` to recount filled cases. diff --git a/complexity_assessments/calibration/amsterdam-dataset.json b/complexity_assessments/calibration/amsterdam-dataset.json new file mode 100644 index 0000000..2b26daf --- /dev/null +++ b/complexity_assessments/calibration/amsterdam-dataset.json @@ -0,0 +1,405 @@ +{ + "eips": [ + { + "eip": "2780", + "dir": "eip2780_reduce_intrinsic_tx_gas", + "prs": 10, + "authors": 5, + "active_days": 10, + "first": "2026-02-17", + "last": "2026-08-05", + "weeks": 24.1, + "direct_churn": 6786, + "ripple_churn": 5561, + "ripple_eels": 681, + "ripple_framework": 612, + "ripple_existing_tests": 480, + "generated": 1670, + "review_load": 86.2, + "changed_files": 177, + "median_pr_days": 3.5, + "loc": 5625, + "test_funcs": 50, + "py_files": 12, + "cases": 641, + "score": 13, + "churn": 12347, + "TEU": 18.6 + }, + { + "eip": "7708", + "dir": "eip7708_eth_transfer_logs", + "prs": 4, + "authors": 3, + "active_days": 4, + "first": "2026-05-13", + "last": "2026-07-24", + "weeks": 12.0, + "direct_churn": 4228, + "ripple_churn": 1507, + "ripple_eels": 134, + "ripple_framework": 169, + "ripple_existing_tests": 1015, + "generated": 869, + "review_load": 33.9, + "changed_files": 113, + "median_pr_days": 3.0, + "loc": 1830, + "test_funcs": 40, + "py_files": 5, + "cases": 224, + "score": 9, + "churn": 5735, + "TEU": 7.4 + }, + { + "eip": "7778", + "dir": "eip7778_block_gas_accounting_without_refunds", + "prs": 10, + "authors": 8, + "active_days": 9, + "first": "2026-05-13", + "last": "2026-07-30", + "weeks": 12.0, + "direct_churn": 1232, + "ripple_churn": 155, + "ripple_eels": 33, + "ripple_framework": 85, + "ripple_existing_tests": 12, + "generated": 0, + "review_load": 35.2, + "changed_files": 33, + "median_pr_days": 0.5, + "loc": 829, + "test_funcs": 6, + "py_files": 3, + "cases": 54, + "score": 10, + "churn": 1387, + "TEU": 13.5 + }, + { + "eip": "7843", + "dir": "eip7843_slotnum", + "prs": 5, + "authors": 3, + "active_days": 5, + "first": "2026-05-14", + "last": "2026-07-24", + "weeks": 11.9, + "direct_churn": 879, + "ripple_churn": 915, + "ripple_eels": 48, + "ripple_framework": 374, + "ripple_existing_tests": 488, + "generated": 869, + "review_load": 28.9, + "changed_files": 130, + "median_pr_days": 4, + "loc": 639, + "test_funcs": 13, + "py_files": 5, + "cases": 65, + "score": 7, + "churn": 1794, + "TEU": 7.9 + }, + { + "eip": "7928", + "dir": "eip7928_block_level_access_lists", + "prs": 95, + "authors": 20, + "active_days": 72, + "first": "2025-10-22", + "last": "2026-07-30", + "weeks": 41.0, + "direct_churn": 19416, + "ripple_churn": 67839, + "ripple_eels": 23026, + "ripple_framework": 21352, + "ripple_existing_tests": 20509, + "generated": 2177, + "review_load": 552.9, + "changed_files": 2858, + "median_pr_days": 2.5, + "loc": 16469, + "test_funcs": 181, + "py_files": 15, + "cases": 2082, + "score": 29, + "churn": 87255, + "TEU": 150.3 + }, + { + "eip": "7954", + "dir": "eip7954_increase_max_contract_size", + "prs": 5, + "authors": 4, + "active_days": 5, + "first": "2026-05-14", + "last": "2026-06-18", + "weeks": 11.9, + "direct_churn": 1391, + "ripple_churn": 684, + "ripple_eels": 6, + "ripple_framework": 191, + "ripple_existing_tests": 486, + "generated": 885, + "review_load": 17.7, + "changed_files": 117, + "median_pr_days": 1, + "loc": 1236, + "test_funcs": 21, + "py_files": 7, + "cases": 106, + "score": null, + "churn": 2075, + "TEU": 6.8 + }, + { + "eip": "7976", + "dir": "eip7976_increase_calldata_floor_cost", + "prs": 7, + "authors": 5, + "active_days": 7, + "first": "2026-02-23", + "last": "2026-07-30", + "weeks": 23.3, + "direct_churn": 3112, + "ripple_churn": 93, + "ripple_eels": 8, + "ripple_framework": 60, + "ripple_existing_tests": 25, + "generated": 18, + "review_load": 29.2, + "changed_files": 35, + "median_pr_days": 1, + "loc": 2849, + "test_funcs": 10, + "py_files": 11, + "cases": 1598, + "score": 5, + "churn": 3205, + "TEU": 9.9 + }, + { + "eip": "7981", + "dir": "eip7981_increase_access_list_cost", + "prs": 6, + "authors": 5, + "active_days": 6, + "first": "2026-04-16", + "last": "2026-07-28", + "weeks": 15.9, + "direct_churn": 1721, + "ripple_churn": 176, + "ripple_eels": 20, + "ripple_framework": 144, + "ripple_existing_tests": 11, + "generated": 36, + "review_load": 4.5, + "changed_files": 31, + "median_pr_days": 0.5, + "loc": 1655, + "test_funcs": 17, + "py_files": 9, + "cases": 367, + "score": 6, + "churn": 1897, + "TEU": 6.5 + }, + { + "eip": "7997", + "dir": "eip7997_deterministic_factory_predeploy", + "prs": 3, + "authors": 3, + "active_days": 3, + "first": "2026-07-02", + "last": "2026-07-28", + "weeks": 4.9, + "direct_churn": 1022, + "ripple_churn": 71, + "ripple_eels": 5, + "ripple_framework": 58, + "ripple_existing_tests": 8, + "generated": 0, + "review_load": 5.2, + "changed_files": 12, + "median_pr_days": 0, + "loc": 1018, + "test_funcs": 16, + "py_files": 5, + "cases": 56, + "score": 5, + "churn": 1093, + "TEU": 3.5 + }, + { + "eip": "8024", + "dir": "eip8024_dupn_swapn_exchange", + "prs": 5, + "authors": 4, + "active_days": 5, + "first": "2026-05-19", + "last": "2026-07-30", + "weeks": 11.1, + "direct_churn": 3765, + "ripple_churn": 1254, + "ripple_eels": 223, + "ripple_framework": 457, + "ripple_existing_tests": 573, + "generated": 869, + "review_load": 32.9, + "changed_files": 122, + "median_pr_days": 5, + "loc": 3463, + "test_funcs": 68, + "py_files": 11, + "cases": 828, + "score": 6, + "churn": 5019, + "TEU": 8.3 + }, + { + "eip": "8037", + "dir": "eip8037_state_creation_gas_cost_increase", + "prs": 38, + "authors": 11, + "active_days": 20, + "first": "2026-04-20", + "last": "2026-07-30", + "weeks": 15.3, + "direct_churn": 22006, + "ripple_churn": 12829, + "ripple_eels": 6428, + "ripple_framework": 1592, + "ripple_existing_tests": 2496, + "generated": 9801, + "review_load": 396.7, + "changed_files": 1177, + "median_pr_days": 1, + "loc": 14914, + "test_funcs": 234, + "py_files": 16, + "cases": 1608, + "score": 28, + "churn": 34835, + "TEU": 77.7 + }, + { + "eip": "8038", + "dir": "eip8038_state_access_gas_cost_increase", + "prs": 6, + "authors": 4, + "active_days": 6, + "first": "2026-07-03", + "last": "2026-08-05", + "weeks": 4.7, + "direct_churn": 7211, + "ripple_churn": 1210, + "ripple_eels": 222, + "ripple_framework": 348, + "ripple_existing_tests": 154, + "generated": 1612, + "review_load": 17.2, + "changed_files": 103, + "median_pr_days": 0.0, + "loc": 5279, + "test_funcs": 70, + "py_files": 16, + "cases": 590, + "score": 20, + "churn": 8421, + "TEU": 7.7 + }, + { + "eip": "8070", + "dir": "eip8070_sparse_blobpool", + "prs": 1, + "authors": 1, + "active_days": 1, + "first": "2026-07-30", + "last": "2026-07-30", + "weeks": 0.9, + "direct_churn": 592, + "ripple_churn": 295, + "ripple_eels": 0, + "ripple_framework": 295, + "ripple_existing_tests": 0, + "generated": 0, + "review_load": 1.0, + "changed_files": 11, + "median_pr_days": 57, + "loc": 592, + "test_funcs": 7, + "py_files": 5, + "cases": 0, + "score": 7, + "churn": 887, + "TEU": 1.1 + }, + { + "eip": "8246", + "dir": "eip8246_selfdestruct_no_burn", + "prs": 2, + "authors": 2, + "active_days": 2, + "first": "2026-07-02", + "last": "2026-07-03", + "weeks": 4.9, + "direct_churn": 289, + "ripple_churn": 1420, + "ripple_eels": 108, + "ripple_framework": 13, + "ripple_existing_tests": 219, + "generated": 36, + "review_load": 10.0, + "changed_files": 22, + "median_pr_days": 0.5, + "loc": 287, + "test_funcs": 2, + "py_files": 3, + "cases": 1302, + "score": null, + "churn": 1709, + "TEU": 3.0 + }, + { + "eip": "8282", + "dir": "eip8282_builder_execution_requests", + "prs": 4, + "authors": 3, + "active_days": 4, + "first": "2026-07-02", + "last": "2026-07-31", + "weeks": 4.9, + "direct_churn": 1616, + "ripple_churn": 1812, + "ripple_eels": 62, + "ripple_framework": 554, + "ripple_existing_tests": 579, + "generated": 0, + "review_load": 13.2, + "changed_files": 58, + "median_pr_days": 1.0, + "loc": 1474, + "test_funcs": 12, + "py_files": 13, + "cases": 120, + "score": null, + "churn": 3428, + "TEU": 5.3 + } + ], + "baseline": { + "meta": 24308, + "eels": 153930, + "framework": 95167, + "generated": 2307161, + "existing_tests": 92202 + }, + "n_units": 917, + "n_commits": 917, + "generated_on": "2026-08-05", + "fork": "amsterdam" +} diff --git a/complexity_assessments/calibration/calibrate.py b/complexity_assessments/calibration/calibrate.py new file mode 100644 index 0000000..f3a9159 --- /dev/null +++ b/complexity_assessments/calibration/calibrate.py @@ -0,0 +1,478 @@ +#!/usr/bin/env python3 +"""Calibrate EIP complexity-assessment scores against measured execution-specs work. + +Measures, per EIP, the testing work that landed in ethereum/execution-specs, then +fits it against the Total Score from complexity_assessments/EIPs/EIP-.md. + + python3 calibrate.py --specs-repo ~/src/execution-specs + +Needs `gh` authenticated (PR metadata). Pass --collect to recount filled test +cases via the execution-specs venv; otherwise cached counts in the dataset are +reused. See README.md for the model and its caveats. Stdlib only. +""" + +from __future__ import annotations + +import argparse +import json +import math +import re +import statistics +import subprocess +import sys +from collections import defaultdict +from datetime import date, datetime +from pathlib import Path + +HERE = Path(__file__).resolve().parent +ASSESSMENTS = HERE.parent / "EIPs" + +SCORE_RE = re.compile(r"\|\s*\*\*Total Score\*\*\s*\|[^|]*\|\s*\**`?(\d+)`?\**\s*\|") +ROW_RE = re.compile(r"^\|\s*\*\*(.+?)\*\*\s*\|([^|]*)\|", re.M) +PR_RE = re.compile(r"\(#(\d+)\)\s*$") +EIP_TEXT_RE = re.compile(r"eip[-_ ]?(\d{4})", re.IGNORECASE) +GIT_TS = "%Y-%m-%dT%H:%M:%SZ" + +# Machine-ported trees. ~2.3M lines of churn in Amsterdam alone; including them +# would swamp every hand-written signal. +GENERATED_PREFIXES = ("tests/ported_static/", "tests/static/") +# The EEST -> execution-specs weld. Before this there is no `tests/` tree here. +WELD_DATE = "2025-10-21" +# EIPs whose work started fewer than this many weeks ago are measured but held +# out of the fit: their totals are truncated, not small. +MATURE_WEEKS = 10.0 + + +# --------------------------------------------------------------- assessments +def read_assessments() -> tuple[dict[str, int], dict[str, dict[str, int]]]: + """Return {eip: total_score} and {eip: {anchor: score}} from the markdown.""" + scores: dict[str, int] = {} + anchors: dict[str, dict[str, int]] = {} + for path in sorted(ASSESSMENTS.glob("EIP-*.md")): + eip = path.stem.split("-")[1] + text = path.read_text() + if m := SCORE_RE.search(text): + scores[eip] = int(m.group(1)) + start = text.find("### Checklist") + if start < 0: + continue + end = text.find("#### Special", start) + rows = {} + for name, value in ROW_RE.findall(text[start : end if end > 0 else len(text)]): + # Cells read "3 + 3 + 3" when an anchor is counted more than once. + if nums := [int(x) for x in re.findall(r"\d+", value)]: + rows[name.strip()] = sum(nums) + if rows: + anchors[eip] = rows + return scores, anchors + + +# ------------------------------------------------------------------ git side +def git_log(repo: Path, branch: str) -> list[dict]: + """One entry per squash-merged commit, with numstat per file.""" + out = subprocess.run( + ["git", "-C", str(repo), "log", branch, f"--since={WELD_DATE}", "--numstat", + "--no-merges", "--format=__C__%H|%ad|%an|%s", "--date=short"], + capture_output=True, text=True, check=True, + ).stdout + commits: list[dict] = [] + cur: dict | None = None + for line in out.splitlines(): + if line.startswith("__C__"): + sha, day, author, subject = line[5:].split("|", 3) + cur = {"sha": sha, "date": day, "author": author, "subject": subject, "files": []} + commits.append(cur) + elif cur is not None and "\t" in line: + parts = line.split("\t") + if len(parts) == 3: + add, dele, path = parts + cur["files"].append((int(add) if add.isdigit() else 0, + int(dele) if dele.isdigit() else 0, path)) + return commits + + +def classify(path: str, fork: str) -> tuple[str, str | None]: + """Bucket a changed path, returning (kind, owning_eip_or_None).""" + if m := re.search(rf"tests/{fork}/eip(\d+)_", path): + return "own", m.group(1) + if path.startswith(GENERATED_PREFIXES): + return "generated", None + if path.startswith("src/ethereum/forks/"): + return "eels", None + # The test framework lives in packages/testing/ post-reorg; src/ethereum_test* + # no longer exists. src/ethereum_spec_tools is the t8n/b11r tooling. + if path.startswith(("packages/testing", "packages/tests", "src/ethereum_spec_tools", + "src/ethereum_optimized", "scripts/")): + return "framework", None + if path.startswith("tests/"): + return "existing_tests", None + return "meta", None + + +def group_units(commits: list[dict]) -> dict[tuple[str, object], list[dict]]: + """Squash merges make one commit per PR; anything else is its own unit.""" + units: dict[tuple[str, object], list[dict]] = defaultdict(list) + for c in commits: + m = PR_RE.search(c["subject"]) + units[("pr", int(m.group(1))) if m else ("sha", c["sha"])].append(c) + return units + + +def owners_of(unit: list[dict], fork: str) -> set[str]: + """EIPs named in the subject win over EIPs whose directory was touched. + + A PR titled "merge EIP-8037 to forks/amsterdam" repairs eight other EIPs' + tests as collateral. That is 8037's cost, not a cost shared eight ways. + """ + titled: set[str] = set() + touched: set[str] = set() + for c in unit: + titled |= {m.group(1) for m in EIP_TEXT_RE.finditer(c["subject"])} + for _, _, path in c["files"]: + kind, eip = classify(path, fork) + if kind == "own": + touched.add(eip) + return titled or touched + + +# ------------------------------------------------------------- github side +def fetch_prs(numbers: list[int], repo: str, batch: int = 25) -> dict[int, dict]: + """Batch PR metadata over GraphQL aliases. Requires an authenticated `gh`.""" + owner, name = repo.split("/") + meta: dict[int, dict] = {} + for i in range(0, len(numbers), batch): + chunk = numbers[i : i + batch] + fields = ("number createdAt mergedAt additions deletions changedFiles " + "comments{totalCount} reviews{totalCount} reviewThreads{totalCount}") + aliases = " ".join(f"p{n}: pullRequest(number:{n}) {{ {fields} }}" for n in chunk) + query = f'query {{ repository(owner:"{owner}", name:"{name}") {{ {aliases} }} }}' + res = subprocess.run( + ["gh", "api", "graphql", "-f", f"query={query}"], + capture_output=True, text=True, + ) + if res.returncode != 0: + sys.exit(f"gh failed: {res.stderr.strip()[:400]}") + for pr in json.loads(res.stdout)["data"]["repository"].values(): + if pr: + meta[pr["number"]] = pr + print(f" fetched {min(i + batch, len(numbers))}/{len(numbers)} PRs", file=sys.stderr) + return meta + + +# ------------------------------------------------------------------ measure +def measure(repo: Path, fork: str, branch: str, gh_repo: str, today: date, + collect: bool, cached_cases: dict[str, int]) -> dict: + commits = git_log(repo, branch) + units = group_units(commits) + + pr_numbers = sorted({n for kind, n in units if kind == "pr" + and owners_of(units[(kind, n)], fork)}) + print(f"{len(commits)} commits, {len(units)} work units, " + f"{len(pr_numbers)} EIP-owned PRs", file=sys.stderr) + pr_meta = fetch_prs(pr_numbers, gh_repo) + + acc: dict[str, dict[str, float]] = defaultdict(lambda: defaultdict(float)) + owned: dict[str, set] = defaultdict(set) + authors: dict[str, set] = defaultdict(set) + days: dict[str, set] = defaultdict(set) + lifetimes: dict[str, list[float]] = defaultdict(list) + baseline: dict[str, float] = defaultdict(float) + + for key, unit in units.items(): + # Direct churn belongs to the directory's EIP regardless of who wrote it. + for c in unit: + for add, dele, path in c["files"]: + kind, eip = classify(path, fork) + if kind == "own": + acc[eip]["direct_churn"] += add + dele + + owners = owners_of(unit, fork) + if not owners: + for c in unit: + for add, dele, path in c["files"]: + baseline[classify(path, fork)[0]] += add + dele + continue + + n = len(owners) + for eip in owners: + owned[eip].add(key) + for c in unit: + authors[eip].add(c["author"]) + days[eip].add(c["date"]) + + for c in unit: + for add, dele, path in c["files"]: + kind, eip = classify(path, fork) + churn = add + dele + if kind == "own" and eip in owners: + continue # already booked above + bucket = "generated" if kind == "generated" else f"ripple_{kind}" + for o in owners: + acc[o][bucket] += churn / n + if kind != "generated": + acc[o]["ripple_churn"] += churn / n + + if key[0] == "pr" and (pr := pr_meta.get(key[1])): + load = (pr["reviews"]["totalCount"] + pr["reviewThreads"]["totalCount"]) / n + for eip in owners: + acc[eip]["review_load"] += load + acc[eip]["changed_files"] += pr["changedFiles"] / n + if pr.get("mergedAt"): + span = (datetime.strptime(pr["mergedAt"], GIT_TS) + - datetime.strptime(pr["createdAt"], GIT_TS)).days + for eip in owners: + lifetimes[eip].append(span) + + cases = collect_cases(repo, fork) if collect else dict(cached_cases) + + rows = [] + for suite in sorted((repo / "tests" / fork).glob("eip*")): + eip = re.match(r"eip(\d+)_", suite.name).group(1) + py = list(suite.rglob("*.py")) + sources = [f.read_text(errors="replace") for f in py] + a = acc[eip] + first = min(days[eip]) if days[eip] else None + rows.append({ + "eip": eip, + "dir": suite.name, + "prs": len(owned[eip]), + "authors": len(authors[eip]), + "active_days": len(days[eip]), + "first": first, + "last": max(days[eip]) if days[eip] else None, + "weeks": round((today - date(*map(int, first.split("-")))).days / 7, 1) + if first else 0.0, + "direct_churn": int(a["direct_churn"]), + "ripple_churn": int(a["ripple_churn"]), + "ripple_eels": int(a["ripple_eels"]), + "ripple_framework": int(a["ripple_framework"]), + "ripple_existing_tests": int(a["ripple_existing_tests"]), + "generated": int(a["generated"]), + "review_load": round(a["review_load"], 1), + "changed_files": int(a["changed_files"]), + "median_pr_days": statistics.median(lifetimes[eip]) if lifetimes[eip] else None, + "loc": sum(len(s.splitlines()) for s in sources), + "test_funcs": sum(len(re.findall(r"^def test_", s, re.M)) for s in sources), + "py_files": len(py), + "cases": cases.get(eip, 0), + }) + return {"eips": rows, "baseline": {k: int(v) for k, v in baseline.items()}, + "n_units": len(units), "n_commits": len(commits)} + + +def collect_cases(repo: Path, fork: str) -> dict[str, int]: + """Count filled test cases per EIP. Reported for completeness only -- see + README §5: vector counts track parametrization style, not work.""" + fill = repo / ".venv/bin/fill" + if not fill.exists(): + print(f" no {fill}; skipping case collection", file=sys.stderr) + return {} + out = subprocess.run( + [str(fill), "--collect-only", "-q", f"--until={fork.capitalize()}", f"tests/{fork}"], + cwd=repo, capture_output=True, text=True, + ).stdout + counts: dict[str, int] = defaultdict(int) + for line in out.splitlines(): + if m := re.match(rf"tests/{fork}/eip(\d+)_", line): + counts[m.group(1)] += 1 + return dict(counts) + + +# ---------------------------------------------------------------------- stats +def pearson(xs, ys) -> float: + n = len(xs) + if n < 3: + return float("nan") + mx, my = sum(xs) / n, sum(ys) / n + num = sum((a - mx) * (b - my) for a, b in zip(xs, ys)) + dx = math.sqrt(sum((a - mx) ** 2 for a in xs)) + dy = math.sqrt(sum((b - my) ** 2 for b in ys)) + return num / (dx * dy) if dx and dy else float("nan") + + +def ols(xs, ys) -> tuple[float, float]: + n = len(xs) + mx, my = sum(xs) / n, sum(ys) / n + slope = (sum((a - mx) * (b - my) for a, b in zip(xs, ys)) + / sum((a - mx) ** 2 for a in xs)) + return my - slope * mx, slope + + +def power_fit(rows, key) -> tuple[float, float]: + """effort = A * score^k, via OLS in log space.""" + a, k = ols([math.log(r["score"]) for r in rows], + [math.log(max(r[key], 0.5)) for r in rows]) + return math.exp(a), k + + +def loo_error(rows, key, power=True) -> float: + """Median leave-one-out |% error|, the only honest accuracy figure at n<15.""" + errs = [] + for held in rows: + train = [r for r in rows if r is not held] + actual = max(held[key], 0.5) + if power: + A, k = power_fit(train, key) + pred = A * held["score"] ** k + else: + b = (sum(r["score"] * r[key] for r in train) + / sum(r["score"] ** 2 for r in train)) + pred = b * held["score"] + errs.append(abs(pred - actual) / actual * 100) + return statistics.median(errs) + + +def teu(row) -> float: + return round(row["prs"] + row["review_load"] / 10, 1) + + +# --------------------------------------------------------------------- report +MEASURES = [("TEU", "TEU"), ("merged PRs", "prs"), ("review load", "review_load"), + ("test functions", "test_funcs"), ("suite LoC", "loc"), + ("total churn", "churn"), ("filled cases", "cases")] + + +def report(data: dict, scores: dict, anchors: dict) -> None: + for r in data["eips"]: + r["score"] = scores.get(r["eip"]) + r["churn"] = r["direct_churn"] + r["ripple_churn"] + r["TEU"] = teu(r) + + scored = [r for r in data["eips"] if r["score"]] + mature = [r for r in scored if r["weeks"] >= MATURE_WEEKS] + if len(mature) < 4: + sys.exit("not enough mature EIPs to fit") + + print(f"\n{'=' * 72}\nMEASURED WORK PER EIP\n{'=' * 72}\n") + head = (f"{'EIP':>6}{'score':>6}{'TEU':>7}{'PRs':>5}{'revw':>6}{'funcs':>6}" + f"{'LoC':>7}{'direct':>8}{'ripple':>8}{'gener':>7}{'wks':>5}") + print(head + "\n" + "-" * len(head)) + for r in sorted(data["eips"], key=lambda r: -r["TEU"]): + flag = "" if r["weeks"] >= MATURE_WEEKS else " in flight" + print(f"{r['eip']:>6}{str(r['score'] or '-'):>6}{r['TEU']:>7}{r['prs']:>5}" + f"{r['review_load']:>6.0f}{r['test_funcs']:>6}{r['loc']:>7}" + f"{r['direct_churn']:>8}{r['ripple_churn']:>8}{r['generated']:>7}" + f"{r['weeks']:>5.0f}{flag}") + print(f"\nunattributed baseline churn: {data['baseline']}") + + print(f"\n{'=' * 72}\nWHICH WORK MEASURE TRACKS THE SCORE?\n{'=' * 72}") + print(f"(mature n={len(mature)}, all scored n={len(scored)})\n") + print(f"{'measure':18}{'r(all)':>9}{'r(mature)':>11}{'LOO%':>7}{'exponent k':>12}") + print("-" * 57) + for label, key in MEASURES: + print(f"{label:18}" + f"{pearson([r['score'] for r in scored], [r[key] for r in scored]):>9.3f}" + f"{pearson([r['score'] for r in mature], [r[key] for r in mature]):>11.3f}" + f"{loo_error(mature, key):>7.0f}{power_fit(mature, key)[1]:>12.2f}") + + A, K = power_fit(mature, "TEU") + lin = loo_error(mature, "TEU", power=False) + print(f"\n{'=' * 72}\nCALIBRATION\n{'=' * 72}\n") + print(" TEU = merged_PRs + (review_submissions + review_threads) / 10\n") + print(f" TEU = {A:.2f} * S^{K:.2f}" + f" r = {pearson([r['score'] for r in mature], [r['TEU'] for r in mature]):.3f}" + f" LOO = {loo_error(mature, 'TEU'):.0f}%") + print(f" linear model for comparison: LOO = {lin:.0f}% " + f"(power law {lin / loo_error(mature, 'TEU'):.1f}x better)") + print(f"\n doubling the score multiplies the work by 2^{K:.2f} = {2 ** K:.1f}x") + + print(f"\n{'EIP':>6}{'score':>6}{'TEU':>7}{'predicted':>11}{'ratio':>7} note") + for r in sorted(scored, key=lambda r: -r["score"]): + pred = A * r["score"] ** K + note = "" if r["weeks"] >= MATURE_WEEKS else f"held out ({r['weeks']:.0f}w in flight)" + print(f"{r['eip']:>6}{r['score']:>6}{r['TEU']:>7}{pred:>11.1f}" + f"{r['TEU'] / pred:>7.2f} {note}") + for r in sorted(data["eips"], key=lambda r: -r["TEU"]): + if not r["score"]: + print(f"{r['eip']:>6}{'-':>6}{r['TEU']:>7}{'':>11}{'':>7} " + f"no assessment; back-solved S ~ {(r['TEU'] / A) ** (1 / K):.0f}") + + print(f"\n{'=' * 72}\nPLANNING TABLE\n{'=' * 72}\n") + fits = {k: power_fit(mature, k) for _, k in MEASURES} + cols = [("TEU", "TEU"), ("PRs", "prs"), ("review", "review_load"), + ("tests", "test_funcs"), ("LoC", "loc"), ("churn", "churn")] + print(f"{'S':>4}{'tier':>5}" + "".join(f"{h:>9}" for h, _ in cols) + f"{'vs S=5':>9}") + print("-" * 68) + base = fits["TEU"][0] * 5 ** fits["TEU"][1] + for s in (3, 5, 8, 10, 13, 16, 20, 25, 30, 36): + tier = "🟢" if s < 10 else ("🟡" if s < 20 else "🔴") + vals = "".join(f"{fits[k][0] * s ** fits[k][1]:>9.0f}" for _, k in cols) + print(f"{s:>4}{tier:>5}{vals}" + f"{fits['TEU'][0] * s ** fits['TEU'][1] / base:>8.1f}x") + + total = sum(r["TEU"] for r in data["eips"]) + per_eip = sum(A * (r["score"] or 0) ** K for r in data["eips"]) + summed = A * sum(r["score"] or 0 for r in data["eips"]) ** K + print(f"\nfork total: {total:.0f} TEU measured across {len(data['eips'])} suites") + print(f" sum of per-EIP predictions: {per_eip:.0f} TEU <- apply the curve per EIP") + print(f" curve applied to summed score: {summed:.0f} TEU <- wrong, {summed / per_eip:.1f}x too high") + + if anchors: + print(f"\n{'=' * 72}\nPER-ANCHOR SIGNAL\n{'=' * 72}\n") + eips = [r["eip"] for r in scored if r["eip"] in anchors] + by_eip = {r["eip"]: r for r in scored} + names = sorted({k for e in eips for k in anchors[e]}) + print(f"{'anchor':46}{'n>0':>5}{'r:churn':>9}{'r:funcs':>9}") + print("-" * 69) + table = [] + for name in names: + xs = [anchors[e].get(name, 0) for e in eips] + if not any(xs): + continue + table.append(( + sum(1 for v in xs if v > 0), name, + pearson(xs, [by_eip[e]["churn"] for e in eips]), + pearson(xs, [by_eip[e]["test_funcs"] for e in eips]), + )) + for nz, name, rc, rf in sorted(table, key=lambda t: -t[2]): + warn = " (single observation)" if nz <= 2 else "" + print(f"{name[:45]:46}{nz:>5}{rc:>9.2f}{rf:>9.2f}{warn}") + xs = [sum(anchors[e].values()) for e in eips] + print(f"{'TOTAL (all rows)':46}{len(eips):>5}" + f"{pearson(xs, [by_eip[e]['churn'] for e in eips]):>9.2f}" + f"{pearson(xs, [by_eip[e]['test_funcs'] for e in eips]):>9.2f}") + unused = [n for n in names if not any(anchors[e].get(n, 0) for e in eips)] + if unused: + print(f"\nnever scored above 0 in this fork (no evidence either way):") + for n in unused: + print(f" - {n}") + + +def main() -> None: + ap = argparse.ArgumentParser(description=__doc__, + formatter_class=argparse.RawDescriptionHelpFormatter) + ap.add_argument("--specs-repo", type=Path, required=True, + help="clone of ethereum/execution-specs") + ap.add_argument("--fork", default="amsterdam", help="fork name under tests/") + ap.add_argument("--branch", default=None, help="default: upstream/forks/") + ap.add_argument("--gh-repo", default="ethereum/execution-specs") + ap.add_argument("--collect", action="store_true", + help="recount filled test cases via the execution-specs venv") + ap.add_argument("--today", default=None, help="YYYY-MM-DD, for reproducible week counts") + ap.add_argument("--out", type=Path, default=None, + help="default: -dataset.json beside this script") + args = ap.parse_args() + + branch = args.branch or f"upstream/forks/{args.fork}" + out = args.out or HERE / f"{args.fork}-dataset.json" + today = date.fromisoformat(args.today) if args.today else date.today() + + cached: dict[str, int] = {} + if out.exists() and not args.collect: + cached = {r["eip"]: r.get("cases", 0) + for r in json.loads(out.read_text()).get("eips", [])} + + scores, anchors = read_assessments() + print(f"{len(scores)} assessments with a Total Score", file=sys.stderr) + + data = measure(args.specs_repo.expanduser(), args.fork, branch, args.gh_repo, + today, args.collect, cached) + data["generated_on"] = today.isoformat() + data["fork"] = args.fork + report(data, scores, anchors) + out.write_text(json.dumps(data, indent=1) + "\n") + print(f"\nwrote {out}", file=sys.stderr) + + +if __name__ == "__main__": + main() diff --git a/complexity_assessments/calibration/proposed-anchors.md b/complexity_assessments/calibration/proposed-anchors.md new file mode 100644 index 0000000..6ce8aae --- /dev/null +++ b/complexity_assessments/calibration/proposed-anchors.md @@ -0,0 +1,266 @@ +# Proposed Checklist Anchors + +Derived from the [Amsterdam calibration](README.md). EIP-7928 is the only mature +Amsterdam EIP the current checklist materially mis-scores, and the reason is +coverage and *structure*, not evaluator error. + +--- + +## 1. The evidence that this is a coverage gap + +Back-solving each EIP's score from its measured work (`TEU = 0.38 × S^1.62`): + +| EIP | Scored | Implied by measured work | Gap | +|---|---:|---:|---:| +| **7928** | **29** | **40** | **+11** | +| 8037 | 28 | 27 | −1 | +| 2780 | 13 | 11 | −2 | +| 7778 | 10 | 9 | −1 | +| 7843 | 7 | 6 | −1 | +| 7981 | 6 | 6 | 0 | +| 8024 | 6 | 7 | +1 | +| 7976 | 5 | 7 | +2 | +| 7708 | 9 | 6 | −3 | + +Every mature EIP lands within ±3 of its score. EIP-7928 is off by +11. That is +not noise — it is one EIP whose dominant cost has no row to be written on. + +**7928 was already at the ceiling of what the checklist can express.** It scored +3 on seven rows and 2 on three more: 29 out of a maximum 33 across the ten +applicable rows. The remaining rows are genuinely N/A (no precompiles, no new +opcodes, no cryptography, no new transaction types). There was no room left. + +### The smoking gun + +7928's assessment scored **Modified opcodes: 0**, with this rationale: + +> Final EVM semantics (gas charged, state changes, reverts) are unchanged. Only +> the observation order for BAL inclusion is constrained, which is a +> spec-framework concern, not a behavior change in the anchor's sense. + +That is the checklist applied *correctly*, and it assigns zero to the single most +expensive part of the work. Whether an opcode runs out of gas **before or after** +it touches state became consensus-observable, and clients had to agree on where +that line falls for every state-accessing opcode. In the repo this shows up as: + +- `fix(spec-specs): Calculate all gas we can before accessing state` +- `fix(spec-specs): Early static check for SSTORE before any reads` +- `feat(test-tests): Expand BAL CALL opcode OOG boundary test cases` +- 16 spec-side `fix`/`refactor` iterations — 3.2× the next highest EIP (8037, 5) +- 23 026 lines of EELS churn — 3.6× the next highest + +## 2. The structural problem: the checklist is additive, the cost is a product + +7928's test surface is a Cartesian product: + +``` +(state-accessing opcodes) × (cold / warm) + × (OOG-before-access / OOG-after-access / success−1 / success) + × (static / non-static) × (7702-delegated / direct) +``` + +An additive 0–3 checklist cannot express a product. Adding rows does not fix it. +Holding the new anchor's *score* fixed and changing only how it enters the total: + +| How the same anchor enters the total | S(7928) | 7928 predicted out-of-sample | vs 150.3 actual | +|---|---:|---:|---:| +| not scored at all (today) | 29 | 61 | 0.40× | +| additive 0–3 row | 32 | 67 | 0.45× | +| additive row at 3× weight | 38 | 79 | 0.52× | +| additive row at 5× weight | 44 | 89 | 0.59× | +| **multiplier `× 2^(score/2)`** | **82** | **154** | **1.03×** | + +*(Fitted on the other eight mature EIPs only, then asked to predict 7928. The +multiplier is the only structure that lands it.)* + +Even at 5× weight — 15 additive points, more than a fifth of the whole 72-point +scale — an additive row gets to 0.59×. This is not a tuning problem. + +The evaluators already knew this: EIP-8037's checklist is written as `2 + 2 + 3 + 1` +and `3 + 3 + 3`, hand-rolling multiplication because the form doesn't support it. +Formalise what they were already doing. + +## 3. Proposed anchors + +### 3.1 Sub-opcode failure-point observability — **MULTIPLIER** + +> Makes it consensus-observable *where inside an opcode's execution* a failure +> occurred — in particular whether gas ran out before or after a state access. +> Previously only the final state mattered. + +- **0.** Failure points inside an opcode are not observable; only final state matters. +- **1.** A single opcode or narrow family gains an observable internal failure point, with one gas boundary to settle. +- **2.** Multiple opcodes gain observable internal failure points, but the same rule fixes the boundary in each case. +- **3.** Every state-accessing opcode gains an observable internal failure point, and the boundary must be settled per-opcode because the gas-charge site differs. + +**This row multiplies the additive subtotal by `2^(score/2)`** rather than adding +to it. Justification: each such boundary must be re-tested across every other +dimension that can change the answer, so it scales the existing matrix instead of +extending it. + +> Do not confuse this with **Modified opcodes**. That row asks whether an +> opcode's *result* changed. This row asks whether the *path to the result* +> became observable. 7928 is 0 on the former and 3 on the latter. + +### 3.2 State gas accounting changes — additive + +Amsterdam introduced a second metering dimension via EIP-8037 — the cost of +*writing* state, distinct from accessing it: `StateGasCosts` +(`STORAGE_SET`, `NEW_ACCOUNT`, `AUTH_BASE`), `COST_PER_STATE_BYTE`, +`state_gas_reservoir`, block-level `block_state_gas_used`, and a `state_gas_spilled` +spill path into execution gas. The checklist has a row for blob gas and none for +this. + +- **0.** No state gas accounting changes. +- **1.** An existing state gas cost or `STATE_BYTES_PER_*` rate is adjusted. +- **2.** A new state-gas-charging site is introduced, or the block-level state gas budget or reservoir allocation is modified. +- **3.** A new state gas charging mechanism is introduced, or the spill interaction between state gas and execution gas is modified, affecting existing gas tests. + +Worse to test than blob gas: the spill path means state gas is **not** +independently meterable, and `NEW_ACCOUNT` is state-dependent. + +### 3.3 Specification underdetermination — additive + +> The EIP text does not determine the answer; clients must agree on a previously +> unspecified detail before tests can be baselined. The cost is coordination and +> re-baselining, not typing. + +- **0.** The EIP text determines the answer for every case a test could construct. +- **1.** A few details are unspecified but have an obvious intended reading. +- **2.** Details require client agreement before tests can be written, but are localized. +- **3.** A previously unspecified *and previously unobservable* behaviour becomes consensus-critical; expect test re-baselining on each round of EIP amendment. + +Evidence: 7928's `uint32 → uint64 → uint32` index-width reversal, BAL moved out of +the payload, `max bal item` check added late. Each round re-baselined tests. + +### 3.4 Test-framework primitives required — additive + +- **0.** Existing test primitives suffice. +- **1.** Existing primitives need minor extension. +- **2.** New expectation or modifier primitives are required, reusable within this EIP's suite. +- **3.** New framework-level primitives are required that become a permanent part of the framework and are used by other EIPs' tests. + +Evidence: 7928 added ~2.4 KLOC of `BlockAccessListExpectation`, +`BalAccountExpectation`, `BalAccountAbsentValues` and a modifier DSL — 9 framework +units and 9 801 lines of framework churn, 6.8× the next highest. + +### 3.5 Retroactive expectation obligation — additive + +> Existing tests must *carry a new validated artifact*. Distinct from +> **Patterns affecting pre-existing tests**, which asks whether they must be +> reworked. + +- **0.** Existing tests are unaffected. +- **1.** A contrived subset of existing tests must carry a new expectation. +- **2.** A broad category must carry a new expectation, mechanically. +- **3.** Every test for the fork carries a new validated artifact, and pre-fork vectors need re-derivation. + +7928 scored only 2 on the existing "pre-existing tests" row — correctly, since +little was *reworked*. But every Amsterdam-filled test now generates and validates +a BAL, and Osaka's BAL checks had to be moved. That is a different cost. + +### 3.6 Uncap Cross-EIP interactions + +The existing row saturates at 3. 7928's rationale names **11** interacting EIPs +(2929, 2930, 1559, 6780, 7702, 4895, 2935, 4788, 7002, 7251, 1153, 214); 8038 +names 6; everyone else names 0. Let level 3 scale with the count — e.g. `3 + 1` +per additional 4 EIPs beyond the first 3 — in the same style the 8037 assessment +already uses. + +## 4. Validation + +Scoring all 12 Amsterdam EIPs on the proposals and refitting: + +| Model | r | exponent k | LOO error | 7928 ratio | 8037 ratio | +|---|---:|---:|---:|---:|---:| +| current checklist | 0.920 | 1.62 | 21 % | 1.68 | 0.92 | +| + the four additive rows only (U, P, R, G) | 0.916 | 1.55 | 31 % | 1.66 | 0.86 | +| + sub-opcode multiplier | 0.996 | 1.12 | 13 % | 0.97 | 1.20 | +| **+ multiplier + state gas row** | **0.998** | **1.11** | **13 %** | **1.01** | **1.12** | + +Note the second row: **the four additive anchors on their own make the fit +slightly worse** (r 0.920 → 0.916, LOO 21 % → 31 %). They add points to 7928, but +they add points to almost everything else too, so the refitted curve moves with +them and 7928's ratio barely budges (1.68 → 1.66). Only the multiplier changes +the shape. The additive rows are worth adding for coverage — they let evaluators +record costs that currently have nowhere to go — but they are not what fixes +7928. + +``` +S_eff = (base + underdetermination + framework + retro + state_gas) × 2^(sub_opcode / 2) +TEU = 0.83 × S_eff^1.11 +``` + +Per EIP, with `add` = additive subtotal and `O` = the multiplier row: + +| EIP | base | U | P | R | G | add | O | S_eff | TEU | pred | ratio | +|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:| +| 7928 | 29 | 3 | 3 | 3 | 0 | 38 | 3 | 107.5 | 150.3 | 149.3 | **1.01** | +| 8037 | 28 | 2 | 2 | 3 | 3 | 38 | 1 | 53.7 | 77.7 | 69.2 | 1.12 | +| 8038 | 20 | 2 | 0 | 2 | 2 | 26 | 1 | 36.8 | 7.7 | 45.4 | 0.17 † | +| 2780 | 13 | 1 | 0 | 2 | 0 | 16 | 0 | 16.0 | 18.6 | 18.0 | 1.03 | +| 7778 | 10 | 0 | 1 | 0 | 0 | 11 | 0 | 11.0 | 13.5 | 11.9 | 1.14 | +| 7708 | 9 | 0 | 1 | 1 | 0 | 11 | 0 | 11.0 | 7.4 | 11.9 | 0.62 | +| 7843 | 7 | 0 | 1 | 1 | 0 | 9 | 0 | 9.0 | 7.9 | 9.5 | 0.83 | +| 8070 | 7 | 0 | 2 | 0 | 0 | 9 | 0 | 9.0 | 1.1 | 9.5 | 0.12 † | +| 8024 | 6 | 0 | 1 | 1 | 0 | 8 | 0 | 8.0 | 8.3 | 8.4 | 0.99 | +| 7981 | 6 | 0 | 1 | 0 | 0 | 7 | 0 | 7.0 | 6.5 | 7.2 | 0.90 | +| 7976 | 5 | 0 | 1 | 0 | 0 | 6 | 0 | 6.0 | 9.9 | 6.1 | 1.63 | +| 7997 | 5 | 0 | 0 | 0 | 0 | 5 | 0 | 5.0 | 3.5 | 5.0 | 0.71 | + +† still in flight; held out of the fit. + +### The exponent collapses to ~1.1 + +This is the most interesting result and the best argument that the proposals +describe something real. The 1.62 exponent in the current calibration was **the +power law compensating for a missing multiplicative term**. Express the product +structure explicitly and the residual super-linearity nearly vanishes — cost +becomes close to linear in the effort-weighted score. "Complexity is mysteriously +super-linear" was the wrong conclusion; "the score was missing a multiplier" is +the right one. + +## 5. Tier rescaling + +The additive ceiling rises from 72 to 84 (four new 0–3 rows), and `S_eff` can +reach `84 × 2.83 = 238`. + +| Tier | On `S_eff` | Amsterdam members | +|---|---|---| +| 🟢 Low | < 12 | 7997, 7976, 7981, 8024, 7843, 8070, 7708, 7778 | +| 🟡 Medium | 12 – 33 | 2780 | +| 🔴 High | ≥ 34 | 8038, 8037, 7928 | + +Thresholds chosen to preserve the current tier populations. Note 7928 at 107 is +now visibly 3× the next-worst EIP rather than sitting one point above it — which +matches what actually happened. + +## 6. Caveats + +1. **`r = 0.998` is not a credible accuracy claim.** Nine points and a + hand-assigned four-level variable can fit almost anything. The credible + evidence is the out-of-sample test in §2 (fit without 7928, predict 7928: + 0.40× → 1.03×), the LOO error dropping 21 % → 13 %, and the exponent + collapsing to ~1.1. +2. **The multiplier's calibration rests on one observation.** 7928 is the only + EIP scoring 3 on it. `2^(score/2)` is a plausible shape, not a measured one — + only the endpoint is measured. A second fork with a 2 or 3 on this row is + needed before the intermediate levels mean anything. +3. **All proposed scores are assigned in hindsight**, by reading the repo. The + real test is whether an evaluator can score these rows *at CFI time*, before + the work. Rows 3.2–3.5 look assessable a priori. Row 3.1 is the risky one: + recognising that an EIP makes an execution detail newly observable is exactly + the insight 7928's original evaluation missed. +4. **The state gas row changes nothing retroactively** except 8037, which + introduced the mechanism. Its value is forward-looking: any Amsterdam+ EIP + touching state gas costs now has a row to be scored on. +5. **Independent evidence for the additive rows is thin.** 3.3 (spec + underdetermination) is scored above 0 for only four EIPs, 3.5 (retro + expectations) for six. They improve the fit but each rests on a handful of + observations. + +## 7. Reproducing + +The scoring tables and every figure above come from `proposed-anchors.py`, which +reads `amsterdam-dataset.json`. Re-run after Amsterdam ships — 8038, 8070 and +7997 are still truncated and will change the picture. diff --git a/complexity_assessments/calibration/proposed-anchors.py b/complexity_assessments/calibration/proposed-anchors.py new file mode 100644 index 0000000..b2b9ede --- /dev/null +++ b/complexity_assessments/calibration/proposed-anchors.py @@ -0,0 +1,184 @@ +#!/usr/bin/env python3 +"""Evaluate proposed checklist anchors against measured Amsterdam work. + +Reads amsterdam-dataset.json (produced by calibrate.py) and reproduces every +figure in proposed-anchors.md. Stdlib only. + + python3 proposed-anchors.py + +The proposed scores below are assigned in hindsight from repo evidence -- see +proposed-anchors.md §6 for why that limits what the fit statistics can claim. +""" + +from __future__ import annotations + +import json +import math +import re +import statistics +from pathlib import Path + +HERE = Path(__file__).resolve().parent +SCORE_RE = re.compile(r"\|\s*\*\*Total Score\*\*\s*\|[^|]*\|\s*\**`?(\d+)`?\**\s*\|") +MATURE_WEEKS = 10.0 + +# --------------------------------------------------------------------------- +# Proposed anchor scores. Evidence for each is cited in proposed-anchors.md §3. +# +# O sub-opcode failure-point observability -- MULTIPLIER, x 2^(O/2) +# G state gas accounting changes -- additive +# U specification underdetermination -- additive +# P test-framework primitives required -- additive +# R retroactive expectation obligation -- additive +# --------------------------------------------------------------------------- +O = {"7928": 3, "8037": 1, "8038": 1} +G = {"8037": 3, "8038": 2} +U = {"7928": 3, "8037": 2, "8038": 2, "2780": 1} +P = {"7928": 3, "8037": 2, "8070": 2, "7778": 1, "7708": 1, "7843": 1, + "8024": 1, "7976": 1, "7981": 1} +R = {"7928": 3, "8037": 3, "8038": 2, "2780": 2, "7708": 1, "7843": 1, "8024": 1} + +MULT_BASE = 2.0 # multiplier is MULT_BASE ** (O / 2) + + +def g(d: dict, eip: str) -> int: + return d.get(eip, 0) + + +# ------------------------------------------------------------------ load data +data = json.loads((HERE / "amsterdam-dataset.json").read_text()) +scores: dict[str, int] = {} +for f in sorted((HERE.parent / "EIPs").glob("EIP-*.md")): + if m := SCORE_RE.search(f.read_text()): + scores[f.stem.split("-")[1]] = int(m.group(1)) + +for r in data["eips"]: + r["TEU"] = round(r["prs"] + r["review_load"] / 10, 1) + r["base"] = scores.get(r["eip"]) +rows = [r for r in data["eips"] if r["base"]] +mature = [r for r in rows if r["weeks"] >= MATURE_WEEKS] +BASE = {r["eip"]: r["base"] for r in rows} + + +# ---------------------------------------------------------------------- stats +def ols(xs, ys): + n = len(xs) + mx, my = sum(xs) / n, sum(ys) / n + b = sum((a - mx) * (c - my) for a, c in zip(xs, ys)) / sum((a - mx) ** 2 for a in xs) + return my - b * mx, b + + +def pearson(xs, ys): + n = len(xs) + mx, my = sum(xs) / n, sum(ys) / n + return (sum((a - mx) * (b - my) for a, b in zip(xs, ys)) + / (math.sqrt(sum((a - mx) ** 2 for a in xs)) + * math.sqrt(sum((b - my) ** 2 for b in ys)))) + + +def fit(subset, score_of): + a, k = ols([math.log(score_of(r["eip"])) for r in subset], + [math.log(r["TEU"]) for r in subset]) + return math.exp(a), k + + +def loo(subset, score_of): + errs = [] + for held in subset: + A, k = fit([r for r in subset if r is not held], score_of) + errs.append(abs(A * score_of(held["eip"]) ** k - held["TEU"]) / held["TEU"] * 100) + return statistics.median(errs) + + +def out_of_sample(subset, score_of, eip="7928"): + """Fit without `eip`, then predict it. The honest test of a proposal.""" + A, k = fit([r for r in subset if r["eip"] != eip], score_of) + actual = next(r["TEU"] for r in subset if r["eip"] == eip) + return A * score_of(eip) ** k, actual + + +# --------------------------------------------------------------- §1 back-solve +print("=" * 74) +print("1. WHAT SCORE WOULD EACH EIP NEED TO LAND ON THE MEASURED CURVE?") +print("=" * 74) +A0, K0 = fit(mature, lambda e: BASE[e]) +print(f"\ncurrent calibration: TEU = {A0:.2f} * S^{K0:.2f}\n") +print(f"{'EIP':>6}{'scored':>8}{'implied':>9}{'gap':>7} note") +for r in sorted(rows, key=lambda r: -r["TEU"]): + imp = (r["TEU"] / A0) ** (1 / K0) + note = "" if r["weeks"] >= MATURE_WEEKS else "in flight, ignore" + star = " <--" if abs(imp - r["base"]) > 5 and r["weeks"] >= MATURE_WEEKS else "" + print(f"{r['eip']:>6}{r['base']:>8}{imp:>9.0f}{imp - r['base']:>+7.0f} {note}{star}") + +# ------------------------------------------------- §2 additive vs multiplier +print("\n" + "=" * 74) +print("2. THE SAME ANCHOR, ADDED vs MULTIPLIED") +print("=" * 74) +print("\nO scored identically in every row; only how it enters the total changes.") +print("Fit on the 8 other mature EIPs, then predict 7928.\n") +print(f"{'structure':40}{'S(7928)':>9}{'predicted':>11}{'vs actual':>12}{'LOO':>7}") +print("-" * 79) +STRUCTURES = [ + ("not scored at all (today)", lambda e: BASE[e]), + ("additive 0-3 row", lambda e: BASE[e] + g(O, e)), + ("additive row at 3x weight", lambda e: BASE[e] + 3 * g(O, e)), + ("additive row at 5x weight", lambda e: BASE[e] + 5 * g(O, e)), + ("multiplier x 2^(O/2)", lambda e: BASE[e] * MULT_BASE ** (g(O, e) / 2)), +] +for label, fn in STRUCTURES: + pred, actual = out_of_sample(mature, fn) + print(f"{label:40}{fn('7928'):>9.0f}{pred:>11.0f}{pred / actual:>11.2f}x" + f"{loo(mature, fn):>6.0f}%") + +# ------------------------------------------------------------- §4 validation +print("\n" + "=" * 74) +print("4. VALIDATION OF THE FULL PROPOSAL") +print("=" * 74 + "\n") + +MODELS = [ + ("current checklist", lambda e: BASE[e]), + ("+ four additive rows only", + lambda e: BASE[e] + g(U, e) + g(P, e) + g(R, e) + g(G, e)), + ("+ sub-opcode multiplier", + lambda e: (BASE[e] + g(U, e) + g(P, e) + g(R, e)) * MULT_BASE ** (g(O, e) / 2)), + ("+ multiplier + state gas row", + lambda e: (BASE[e] + g(U, e) + g(P, e) + g(R, e) + g(G, e)) + * MULT_BASE ** (g(O, e) / 2)), +] +print(f"{'model':32}{'r':>7}{'k':>6}{'LOO':>7}{'7928':>8}{'8037':>8}") +print("-" * 68) +for label, fn in MODELS: + A, k = fit(mature, fn) + ratios = {r["eip"]: r["TEU"] / (A * fn(r["eip"]) ** k) for r in mature} + print(f"{label:32}{pearson([fn(r['eip']) for r in mature], [r['TEU'] for r in mature]):>7.3f}" + f"{k:>6.2f}{loo(mature, fn):>6.0f}%{ratios['7928']:>8.2f}{ratios['8037']:>8.2f}") + +final = MODELS[-1][1] +A, k = fit(mature, final) +print(f"\nS_eff = (base + U + P + R + G) x {MULT_BASE:.0f}^(O/2)") +print(f"TEU = {A:.2f} * S_eff^{k:.2f}\n") +print(f"{'EIP':>6}{'base':>6}{'U':>3}{'P':>3}{'R':>3}{'G':>3}{'add':>6}{'O':>3}" + f"{'S_eff':>8}{'TEU':>8}{'pred':>8}{'ratio':>7}{'tier':>6}") +for r in sorted(rows, key=lambda r: -final(r["eip"])): + e = r["eip"] + add = BASE[e] + g(U, e) + g(P, e) + g(R, e) + g(G, e) + se = final(e) + tier = "🔴" if se >= 34 else ("🟡" if se >= 12 else "🟢") + flag = "" if r["weeks"] >= MATURE_WEEKS else " in flight" + print(f"{e:>6}{BASE[e]:>6}{g(U, e):>3}{g(P, e):>3}{g(R, e):>3}{g(G, e):>3}{add:>6}" + f"{g(O, e):>3}{se:>8.1f}{r['TEU']:>8}{A * se ** k:>8.1f}" + f"{r['TEU'] / (A * se ** k):>7.2f}{tier:>5}{flag}") + +print(f"\nexponent: {K0:.2f} (current) -> {k:.2f} (with the multiplier)") +print("The 1.6 exponent was the power law compensating for a missing product term.") + +# ------------------------------------------------------------------- §5 tiers +print("\n" + "=" * 74) +print("5. TIER MEMBERSHIP UNDER S_eff") +print("=" * 74 + "\n") +for tier, lo, hi in (("🟢 Low", 0, 12), ("🟡 Medium", 12, 34), ("🔴 High", 34, 1e9)): + members = sorted((r["eip"] for r in rows if lo <= final(r["eip"]) < hi), + key=lambda e: final(e)) + print(f"{tier:12} S_eff {lo:>3}-{hi if hi < 1e9 else '':<4} {' '.join(members)}") +print(f"\nadditive ceiling 72 -> 84 (four new rows); S_eff ceiling " + f"84 * {MULT_BASE ** 1.5:.2f} = {84 * MULT_BASE ** 1.5:.0f}") From f94d86e23f92fbaa3d0034947057a01d74546a64 Mon Sep 17 00:00:00 2001 From: marioevz Date: Wed, 5 Aug 2026 16:10:34 -0600 Subject: [PATCH 2/3] Calibration anchors 2 --- .../calibration/proposed-anchors.md | 242 +++++++++++++----- .../calibration/proposed-anchors.py | 67 ++++- 2 files changed, 244 insertions(+), 65 deletions(-) diff --git a/complexity_assessments/calibration/proposed-anchors.md b/complexity_assessments/calibration/proposed-anchors.md index 6ce8aae..a001979 100644 --- a/complexity_assessments/calibration/proposed-anchors.md +++ b/complexity_assessments/calibration/proposed-anchors.md @@ -70,37 +70,128 @@ Holding the new anchor's *score* fixed and changing only how it enters the total | additive row at 5× weight | 44 | 89 | 0.59× | | **multiplier `× 2^(score/2)`** | **82** | **154** | **1.03×** | -*(Fitted on the other eight mature EIPs only, then asked to predict 7928. The -multiplier is the only structure that lands it.)* +*(Fitted on the other eight mature EIPs only, then asked to predict 7928.)* Even at 5× weight — 15 additive points, more than a fifth of the whole 72-point -scale — an additive row gets to 0.59×. This is not a tuning problem. +scale — an additive row gets to 0.59×. This is not a tuning problem: a sum cannot +approximate a product over this range. The evaluators already knew this: EIP-8037's checklist is written as `2 + 2 + 3 + 1` and `3 + 3 + 3`, hand-rolling multiplication because the form doesn't support it. -Formalise what they were already doing. + +**The arithmetic above is sound. What it does *not* establish is that the +state-access ordering row is the multiplier — see the next section.** + +## 2b. Why the data cannot identify *which* row is the multiplier + +Running the same test with every existing anchor as the multiplier +(`S_eff = (base − row) × 2^(row/2)`, so the row isn't double-counted): + +| Anchor used as multiplier | 7928's score | Other EIPs scoring >0 | 7928 out-of-sample | +|---|---:|---:|---:| +| Security risks | 3 | 0 | 1.40× | +| Engine API changes | 3 | 1 | 1.35× | +| New block / header fields | 3 | 1 | **1.10×** | +| Block syncing changes | 2 | 0 | **0.93×** | +| Performance risks | 3 | 3 | 0.77× | +| Transition-tool interface | 2 | 2 | 0.75× | +| Cross-EIP interactions | 3 | 1 | 0.68× | +| Edge/boundary conditions | 3 | 7 | 0.38× | +| EVM Gas rule changes | 3 | 5 | 0.21× | +| Patterns affecting pre-existing tests | 2 | 6 | 0.16× | + +**Six existing rows work about as well as the proposed one, and they work for a +bad reason.** Look at the third column: the rows that "land" 7928 are exactly the +rows 7928 scores high on and nobody else scores at all. Made multiplicative, such +a row is a 7928 indicator variable — it identifies 7928 rather than explaining it. +Rows that many EIPs score (edge/boundary: 7 others; pre-existing tests: 6; gas +rules: 5) all fail, because multiplying them lifts everyone and the refit absorbs +the change. + +So the out-of-sample test in §2 is weaker evidence than it looks. It guards +against fitting 7928's *measured TEU*, but not against having chosen a variable +that happens to single 7928 out. With one high-cost EIP in the sample, **no +statistical test can distinguish a mechanism from a label.** + +### And you cannot simply make several rows multiplicative + +Independent multipliers compound: + +| EIP | base | product over 4 dimensional rows | S_eff | +|---|---:|---:|---:| +| 7928 | 29 | 5.7× | 164 | +| 8037 | 28 | **45.3×** | **1267** | +| 8038 | 20 | 8.0× | 160 | + +8037 explodes because its "pre-existing tests" cell is written `3 + 3 + 3` = 9, and +`2^4.5` = 22.6. Nor can the axis count be recovered by re-reading the existing +rows as dimensions: try it and almost every EIP saturates at the cap, because +those rows encode *severity*, not dimensionality. + +The resolution is arithmetic: axes **add in the exponent** +(`2^(a/2) × 2^(b/2) = 2^((a+b)/2)`). So the only stable form is **one** multiplier +whose score counts axes, scored independently of the severity rows — not N +individually multiplicative rows. ## 3. Proposed anchors -### 3.1 Sub-opcode failure-point observability — **MULTIPLIER** +### 3.1 State-access ordering within opcode execution — additive -> Makes it consensus-observable *where inside an opcode's execution* a failure -> occurred — in particular whether gas ran out before or after a state access. -> Previously only the final state mattered. +> Changes *where inside an opcode's execution* state is accessed, or where gas is +> charged relative to that access. -- **0.** Failure points inside an opcode are not observable; only final state matters. -- **1.** A single opcode or narrow family gains an observable internal failure point, with one gas boundary to settle. -- **2.** Multiple opcodes gain observable internal failure points, but the same rule fixes the boundary in each case. -- **3.** Every state-accessing opcode gains an observable internal failure point, and the boundary must be settled per-opcode because the gas-charge site differs. +**Worded for the world after EIP-7928, not for the transition into it.** Since the +BAL, every state access is consensus-observable — an address or slot appears only +if execution had enough gas to reach it — so the ordering of gas charges and state +accesses inside an opcode is consensus rather than a client implementation detail. +Introducing that observability was a one-time event that has already happened; what +recurs is EIPs *moving* the ordering. -**This row multiplies the additive subtotal by `2^(score/2)`** rather than adding -to it. Justification: each such boundary must be re-tested across every other -dimension that can change the answer, so it scales the existing matrix instead of -extending it. +- **0.** No change to where state is accessed, or to where gas is charged relative to a state access, within any opcode. +- **1.** A single opcode's state-access or gas-charge ordering changes. +- **2.** Multiple opcodes' ordering changes, or a new state-accessing operation is introduced whose position in the order must be settled. +- **3.** The ordering rule changes for a whole class of state-accessing opcodes at once, or what counts as a recordable state access is redefined — requiring existing BAL vectors to be re-derived across opcodes and forks. > Do not confuse this with **Modified opcodes**. That row asks whether an -> opcode's *result* changed. This row asks whether the *path to the result* -> became observable. 7928 is 0 on the former and 3 on the latter. +> opcode's *result* changed. This row asks about the *path to the result*, now +> observable even when the result is identical. 7928 is 0 on the former and 3 on +> the latter — which is why the most expensive part of the work scored zero. + +The reframing does not disturb the scores used below: 7928 remains a 3 (it +reordered gas-charge sites across an entire class of opcodes), 8037 and 8038 +remain 1 (each moves where gas is charged relative to a state write). + +This is the row whose absence is best evidenced (§1), independent of any argument +about multipliers. Add it additively now. + +### 3.1b Test-matrix dimensionality — **the multiplier**, if one is adopted + +Per §2b, a multiplier cannot be one of the severity rows, and there cannot be +several. If a multiplier is adopted it has to be a single row that asks directly +for the axis count, scored independently of everything else: + +> How many **independent axes** does this EIP add to the test matrix — dimensions +> across which every other case must be re-run to get a different answer? + +- **0.** No new axis. Cases are enumerable as a list. +- **1.** One new axis (e.g. every case must be run warm and cold). +- **2.** Two independent new axes. +- **3.** Three or more independent new axes. + +`S_eff = additive_subtotal × 2^(axes / 2)`. The base 2 and the `/2` are chosen so +three axes give ≈2.8× rather than 8×, on the assumption that axes are partly +redundant in practice. **Neither the base nor the exponent is measured** — only the +`axes = 3` endpoint is, from one EIP. + +For 7928 the three axes are: the intra-opcode gas boundary (§3.1), the 11-EIP cross +product, and the BAL expectation attaching to every existing test. For 8037, one: +the gas boundary re-run across the static suite. + +**Recommendation: do not adopt this yet.** Add rows 3.1–3.6 additively, and record +the axis count on assessments as an *unscored observation* for a fork or two. +When a second EIP scores 2 or 3 on it, there will be enough data to tell whether +the multiplier is real and what its base should be. Adopting it now means +hard-coding a curve fitted to a single point. ### 3.2 State gas accounting changes — additive @@ -144,20 +235,25 @@ Evidence: 7928 added ~2.4 KLOC of `BlockAccessListExpectation`, `BalAccountExpectation`, `BalAccountAbsentValues` and a modifier DSL — 9 framework units and 9 801 lines of framework churn, 6.8× the next highest. -### 3.5 Retroactive expectation obligation — additive +### 3.5 New invariant on pre-existing tests — additive -> Existing tests must *carry a new validated artifact*. Distinct from -> **Patterns affecting pre-existing tests**, which asks whether they must be -> reworked. +> Tests that are **not about this EIP** must nonetheless check something this EIP +> produces. Their logic doesn't change; they gain a new thing to assert. +> +> Deliberately named to sit alongside **Patterns affecting pre-existing tests**. +> That row asks whether existing tests must be *reworked*; this one asks whether +> they must *additionally assert something new*. 7928 is a 2 on the first and a 3 +> on the second. -- **0.** Existing tests are unaffected. -- **1.** A contrived subset of existing tests must carry a new expectation. -- **2.** A broad category must carry a new expectation, mechanically. -- **3.** Every test for the fork carries a new validated artifact, and pre-fork vectors need re-derivation. +- **0.** Pre-existing tests assert nothing new. +- **1.** A narrow, contrived category of pre-existing tests gains a new assertion. +- **2.** A broad category gains a new assertion, applied mechanically. +- **3.** Every test in the fork gains the assertion regardless of what it tests, and pre-fork vectors must be re-derived to satisfy it. -7928 scored only 2 on the existing "pre-existing tests" row — correctly, since -little was *reworked*. But every Amsterdam-filled test now generates and validates -a BAL, and Osaka's BAL checks had to be moved. That is a different cost. +7928 scored only 2 on the "pre-existing tests" row — correctly, since little was +*reworked*. But every Amsterdam-filled test now generates and validates a BAL +whether or not it has anything to do with access lists, and Osaka's BAL checks had +to be moved. That is a different cost, and it had nowhere to be recorded. ### 3.6 Uncap Cross-EIP interactions @@ -169,13 +265,16 @@ already uses. ## 4. Validation -Scoring all 12 Amsterdam EIPs on the proposals and refitting: +Scoring all 12 Amsterdam EIPs on the proposals and refitting. **Read rows 3 and 4 +of this table together with §2b** — the fit statistics do not discriminate between +the state-access ordering row and six other candidate multipliers, so they are shown to +document the arithmetic, not to validate the choice of row. | Model | r | exponent k | LOO error | 7928 ratio | 8037 ratio | |---|---:|---:|---:|---:|---:| | current checklist | 0.920 | 1.62 | 21 % | 1.68 | 0.92 | | + the four additive rows only (U, P, R, G) | 0.916 | 1.55 | 31 % | 1.66 | 0.86 | -| + sub-opcode multiplier | 0.996 | 1.12 | 13 % | 0.97 | 1.20 | +| + state-access ordering as multiplier | 0.996 | 1.12 | 13 % | 0.97 | 1.20 | | **+ multiplier + state gas row** | **0.998** | **1.11** | **13 %** | **1.01** | **1.12** | Note the second row: **the four additive anchors on their own make the fit @@ -183,11 +282,11 @@ slightly worse** (r 0.920 → 0.916, LOO 21 % → 31 %). They add points to 7928 they add points to almost everything else too, so the refitted curve moves with them and 7928's ratio barely budges (1.68 → 1.66). Only the multiplier changes the shape. The additive rows are worth adding for coverage — they let evaluators -record costs that currently have nowhere to go — but they are not what fixes -7928. +record costs that currently have nowhere to go — but they are not what closes the +gap on EIP-7928. ``` -S_eff = (base + underdetermination + framework + retro + state_gas) × 2^(sub_opcode / 2) +S_eff = (base + underdetermination + framework + new_invariant + state_gas) × 2^(state_access_ordering / 2) TEU = 0.83 × S_eff^1.11 ``` @@ -212,18 +311,36 @@ Per EIP, with `add` = additive subtotal and `O` = the multiplier row: ### The exponent collapses to ~1.1 -This is the most interesting result and the best argument that the proposals -describe something real. The 1.62 exponent in the current calibration was **the -power law compensating for a missing multiplicative term**. Express the product -structure explicitly and the residual super-linearity nearly vanishes — cost -becomes close to linear in the effort-weighted score. "Complexity is mysteriously -super-linear" was the wrong conclusion; "the score was missing a multiplier" is -the right one. +The 1.62 exponent in the current calibration behaves like **a power law +compensating for a missing multiplicative term**. Introduce a multiplicative term +of roughly the right size and the residual super-linearity nearly vanishes — cost +becomes close to linear in the effort-weighted score. + +This is suggestive rather than conclusive, and for the same reason as everything +else here: any of the six candidate rows from §2b produces a similar collapse +(exponents 1.07–1.36). What the collapse supports is the *shape* of the model — +that some multiplicative term is missing — not the identity of the row supplying +it. "Complexity is mysteriously super-linear" is probably the wrong reading; "the +score is missing a product term" is the better one, and which product is still +open. ## 5. Tier rescaling -The additive ceiling rises from 72 to 84 (four new 0–3 rows), and `S_eff` can -reach `84 × 2.83 = 238`. +**As adopted in the template** (additive rows only, no multiplier): the anchor set +goes 24 → 28 rows — five added, **Engine API encoding changes** removed — so the +nominal ceiling is 72 → 84, and **Cross-EIP interactions** is uncapped on top of +that. Thresholds scale by 84/72: 🟢 `<12`, 🟡 `12–22`, 🔴 `≥23`. Amsterdam tier +membership is unchanged except EIP-7778, which moves 🟡 → 🟢 at 11 points. + +> Engine API encoding changes described a JSON → RLP/SSZ migration at the Engine +> API layer. Those are coordinated outside the EIP process, so the row has no use +> in an EIP-scoped assessment. It was scored 0 or blank in all 28 assessments on +> the previous revision, and never had level definitions, so removing it changes +> no historical total. **Encoding changes (RLP/SSZ)** already covers the +> "interfaces level" if an EIP ever carries one. + +**If the deferred multiplier (§3.1b) were also adopted**, `S_eff` could reach +`84 × 2.83 = 238` and the thresholds would instead be: | Tier | On `S_eff` | Amsterdam members | |---|---|---| @@ -231,33 +348,38 @@ reach `84 × 2.83 = 238`. | 🟡 Medium | 12 – 33 | 2780 | | 🔴 High | ≥ 34 | 8038, 8037, 7928 | -Thresholds chosen to preserve the current tier populations. Note 7928 at 107 is -now visibly 3× the next-worst EIP rather than sitting one point above it — which -matches what actually happened. +Also chosen to preserve tier populations. Under this variant 7928 lands at 107 — +visibly 3× the next-worst EIP rather than sitting one point above it, which +matches what actually happened. This is the variant to revisit once a second fork +supplies a second observation on the axis-count row. ## 6. Caveats 1. **`r = 0.998` is not a credible accuracy claim.** Nine points and a - hand-assigned four-level variable can fit almost anything. The credible - evidence is the out-of-sample test in §2 (fit without 7928, predict 7928: - 0.40× → 1.03×), the LOO error dropping 21 % → 13 %, and the exponent - collapsing to ~1.1. -2. **The multiplier's calibration rests on one observation.** 7928 is the only - EIP scoring 3 on it. `2^(score/2)` is a plausible shape, not a measured one — - only the endpoint is measured. A second fork with a 2 or 3 on this row is - needed before the intermediate levels mean anything. -3. **All proposed scores are assigned in hindsight**, by reading the repo. The + hand-assigned four-level variable can fit almost anything. +2. **The out-of-sample test is weaker than it appears.** Fitting without 7928 and + predicting it (0.40× → 1.03×) rules out fitting to 7928's measured TEU. It does + *not* rule out having picked a variable that singles 7928 out — and §2b shows + six existing rows do just as well when made multiplicative, purely because + 7928 is the only EIP that scores them. With one high-cost EIP in the sample, no + statistical test can separate a mechanism from a label. +3. **The multiplier is therefore a design proposal, not a finding.** What the data + supports is the *negative* result: a sum cannot approximate a product, so no + additive row at any weight closes the gap. Which row multiplies, and with what + base, is unidentified. Hence the recommendation in §3.1b to record the axis + count unscored until a second fork provides a second observation. +4. **All proposed scores are assigned in hindsight**, by reading the repo. The real test is whether an evaluator can score these rows *at CFI time*, before the work. Rows 3.2–3.5 look assessable a priori. Row 3.1 is the risky one: recognising that an EIP makes an execution detail newly observable is exactly the insight 7928's original evaluation missed. -4. **The state gas row changes nothing retroactively** except 8037, which +5. **The state gas row changes nothing retroactively** except 8037, which introduced the mechanism. Its value is forward-looking: any Amsterdam+ EIP touching state gas costs now has a row to be scored on. -5. **Independent evidence for the additive rows is thin.** 3.3 (spec - underdetermination) is scored above 0 for only four EIPs, 3.5 (retro - expectations) for six. They improve the fit but each rests on a handful of - observations. +6. **Independent evidence for the additive rows is thin.** 3.3 (spec + underdetermination) is scored above 0 for only four EIPs, 3.5 (new invariant on + pre-existing tests) for six. They improve the fit but each rests on a handful + of observations. ## 7. Reproducing diff --git a/complexity_assessments/calibration/proposed-anchors.py b/complexity_assessments/calibration/proposed-anchors.py index b2b9ede..8963a64 100644 --- a/complexity_assessments/calibration/proposed-anchors.py +++ b/complexity_assessments/calibration/proposed-anchors.py @@ -25,11 +25,11 @@ # --------------------------------------------------------------------------- # Proposed anchor scores. Evidence for each is cited in proposed-anchors.md §3. # -# O sub-opcode failure-point observability -- MULTIPLIER, x 2^(O/2) +# O state-access ordering within opcodes -- additive (see note below) # G state gas accounting changes -- additive # U specification underdetermination -- additive # P test-framework primitives required -- additive -# R retroactive expectation obligation -- additive +# R new invariant on pre-existing tests -- additive # --------------------------------------------------------------------------- O = {"7928": 3, "8037": 1, "8038": 1} G = {"8037": 3, "8038": 2} @@ -46,11 +46,27 @@ def g(d: dict, eip: str) -> int: # ------------------------------------------------------------------ load data +ROW_RE = re.compile(r"^\|\s*\*\*(.+?)\*\*\s*\|([^|]*)\|", re.M) + data = json.loads((HERE / "amsterdam-dataset.json").read_text()) scores: dict[str, int] = {} +per_anchor: dict[str, dict[str, int]] = {} for f in sorted((HERE.parent / "EIPs").glob("EIP-*.md")): - if m := SCORE_RE.search(f.read_text()): - scores[f.stem.split("-")[1]] = int(m.group(1)) + eip = f.stem.split("-")[1] + txt = f.read_text() + if m := SCORE_RE.search(txt): + scores[eip] = int(m.group(1)) + i = txt.find("### Checklist") + if i < 0: + continue + j = txt.find("#### Special", i) + rows_ = {} + for name, val in ROW_RE.findall(txt[i: j if j > 0 else len(txt)]): + # cells read "3 + 3 + 3" when an anchor is counted more than once + if nums := [int(x) for x in re.findall(r"\d+", val)]: + rows_[name.strip()] = sum(nums) + if rows_: + per_anchor[eip] = rows_ for r in data["eips"]: r["TEU"] = round(r["prs"] + r["review_load"] / 10, 1) @@ -130,6 +146,47 @@ def out_of_sample(subset, score_of, eip="7928"): print(f"{label:40}{fn('7928'):>9.0f}{pred:>11.0f}{pred / actual:>11.2f}x" f"{loo(mature, fn):>6.0f}%") +# ------------------------------------- §3 can the data identify the multiplier? +print("\n" + "=" * 74) +print("3. TRY EVERY EXISTING ANCHOR AS THE MULTIPLIER") +print("=" * 74) +print("\nS_eff = (base - row) x 2^(row/2), so the row is not double-counted.") +print("If many rows work, the data cannot identify which row is multiplicative.\n") +print(f"{'anchor used as multiplier':44}{'7928':>6}{'others>0':>10}{'LOO':>7}{'OOS':>8}") +print("-" * 75) +live = [n for n in sorted({n for e in per_anchor for n in per_anchor[e]}) + if any(per_anchor[e].get(n, 0) for e in (r["eip"] for r in mature) + if e in per_anchor)] +sweep = [] +for n in live: + def sf(e, n=n): + row = per_anchor.get(e, {}).get(n, 0) + return max(BASE[e] - row, 1) * MULT_BASE ** (row / 2) + pred, actual = out_of_sample(mature, sf) + others = sum(1 for r in mature + if r["eip"] != "7928" and per_anchor.get(r["eip"], {}).get(n, 0)) + sweep.append((pred / actual, n, per_anchor.get("7928", {}).get(n, 0), + others, loo(mature, sf))) +for oos, n, s7, others, l in sorted(sweep, key=lambda t: -t[0]): + mark = " <-- lands it" if 0.85 <= oos <= 1.2 else "" + print(f"{n[:43]:44}{s7:>6}{others:>10}{l:>6.0f}%{oos:>7.2f}x{mark}") +print("\nThe rows that 'work' are the ones 7928 scores high and nobody else does") +print("(others>0 near zero). That is a 7928 indicator, not a mechanism. Rows many") +print("EIPs score -- edge/boundary, pre-existing tests, gas rules -- all fail,") +print("because multiplying them lifts everyone and the refit absorbs it.") + +print("\n--- and independent multipliers compound past usability ---\n") +DIM = ["Patterns affecting pre-existing tests", "Cross-EIP interactions", + "New transaction types", "Encoding changes (RLP/SSZ)"] +print(f"{'EIP':>6}{'base':>6}{'product of 2^(row/2) over 4 dimensional rows':>46}{'S_eff':>8}") +for e in sorted((r["eip"] for r in rows), key=lambda e: -BASE[e])[:4]: + prod = math.prod(MULT_BASE ** (per_anchor.get(e, {}).get(n, 0) / 2) for n in DIM) + print(f"{e:>6}{BASE[e]:>6}{prod:>46.1f}{BASE[e] * prod:>8.0f}") +print("\n8037's 'pre-existing tests' cell is written '3 + 3 + 3' = 9, so a") +print("multiplicative reading gives it 2^4.5 = 22.6x. Compounding is unusable:") +print("axes ADD in the exponent, so the stable form is ONE multiplier whose") +print("score counts axes -- not N independently multiplicative rows.") + # ------------------------------------------------------------- §4 validation print("\n" + "=" * 74) print("4. VALIDATION OF THE FULL PROPOSAL") @@ -139,7 +196,7 @@ def out_of_sample(subset, score_of, eip="7928"): ("current checklist", lambda e: BASE[e]), ("+ four additive rows only", lambda e: BASE[e] + g(U, e) + g(P, e) + g(R, e) + g(G, e)), - ("+ sub-opcode multiplier", + ("+ state-access ordering as mult", lambda e: (BASE[e] + g(U, e) + g(P, e) + g(R, e)) * MULT_BASE ** (g(O, e) / 2)), ("+ multiplier + state gas row", lambda e: (BASE[e] + g(U, e) + g(P, e) + g(R, e) + g(G, e)) From 5cb54539dc5b4bb7cb2b078306ea71444d09098b Mon Sep 17 00:00:00 2001 From: marioevz Date: Wed, 5 Aug 2026 16:12:50 -0600 Subject: [PATCH 3/3] EIP Complexity Assessment Revision 2 --- Templates/EIP-Complexity-Assessment.md | 147 ++++++++++++++++++++++++- 1 file changed, 141 insertions(+), 6 deletions(-) diff --git a/Templates/EIP-Complexity-Assessment.md b/Templates/EIP-Complexity-Assessment.md index f69abef..6fa315b 100644 --- a/Templates/EIP-Complexity-Assessment.md +++ b/Templates/EIP-Complexity-Assessment.md @@ -1,5 +1,7 @@ # EIP-NNNN: EIP Title +Checklist revision: **2** (28 anchors) — see [Revision Notes](#revision-notes) + Link: https://eips.ethereum.org/EIPS/eip-NNNN ## Execution Specs @@ -22,6 +24,21 @@ New EVM gas accounting rules - 2. A new gas accounting mechanism is introduced but it does not affect existing mechanisms nor does it affect existing tests. - 3. A new gas accounting mechanism is introduced and affects existing mechanisms which in turn affect existing tests. +##### State-access ordering within opcode execution + +Changes *where inside an opcode's execution* state is accessed, or where gas is charged relative to that access. Because a state access is recorded in the block-level access list only if execution had enough gas to reach it, this ordering is consensus-critical: moving it changes the BAL at every gas boundary of every affected opcode. + +- 0. No change to where state is accessed, or to where gas is charged relative to a state access, within any opcode. +- 1. A single opcode's state-access or gas-charge ordering changes. +- 2. Multiple opcodes' ordering changes, or a new state-accessing operation is introduced whose position in the order must be settled. +- 3. The ordering rule changes for a whole class of state-accessing opcodes at once, or what counts as a recordable state access is redefined — requiring existing BAL vectors to be re-derived across opcodes and forks. + +*Distinct from "Modified opcodes", which asks whether an opcode's **result** changed. This row asks about the **path to the result**, which is observable even when the result is identical. An EIP can be 0 on that row and 3 on this one. + +*Score changes **to** the ordering. Do not score the fact that state accesses are observable — they always are. + +*Each boundary must be re-tested against every other dimension that can change the answer (cold/warm, static/non-static, delegated/direct, revert/success), so the case count grows multiplicatively rather than additively. Note this explicitly under Special Considerations. + ##### Blob gas accounting changes New Blob gas accounting rules which potentially affect pre-existing tests @@ -31,6 +48,17 @@ New Blob gas accounting rules which potentially affect pre-existing tests - 2. A new blob gas accounting mechanism is introduced but it does not affect existing mechanisms nor does it affect existing tests. - 3. A new blob gas accounting mechanism is introduced and affects existing mechanisms which in turn affect existing tests. +##### State gas accounting changes + +New state gas accounting rules. State gas is the cost of *writing* state, as opposed to accessing or executing it: `StateGasCosts`, `COST_PER_STATE_BYTE`, the block-level state gas budget, and the spill path into execution gas. + +- 0. No state gas accounting changes. +- 1. An existing state gas cost or `STATE_BYTES_PER_*` rate is adjusted. +- 2. A new state-gas-charging site is introduced, or the block-level state gas budget or reservoir allocation is modified. +- 3. A new state gas charging mechanism is introduced, or the spill interaction between state gas and execution gas is modified, affecting existing gas tests. + +*Harder to test than blob gas: the spill path means state gas cannot be metered independently of execution gas, and some costs (e.g. `NEW_ACCOUNT`) are state-dependent. + ##### New EVM gas refund New gas-refund mechanism @@ -49,6 +77,17 @@ Implements a new validation mechanism or rule that translates in reworking pre-e - 2. Considerable subset of existing tests are affected by this change but involves only a contrived category of tests. - 3. Major subset of existing tests are affected, including diverse category of tests (benchmarks, static, multiple forks, etc.). +##### New invariant on pre-existing tests + +Tests that are **not about this EIP** must nonetheless assert something this EIP produces. Their logic does not change; they gain a new thing to check. + +- 0. Pre-existing tests assert nothing new. +- 1. A narrow, contrived category of pre-existing tests gains a new assertion. +- 2. A broad category gains a new assertion, applied mechanically. +- 3. Every test in the fork gains the assertion regardless of what it tests, and pre-fork vectors must be re-derived to satisfy it. + +*Paired with the row above, and easy to confuse with it. "Patterns affecting pre-existing tests" asks whether existing tests must be **reworked**; this row asks whether they must **additionally assert something new**. Score both — an EIP can be low on one and high on the other. + ##### Transition-tool interface changes Modifies or adds new fields to the transition tool interface. @@ -60,6 +99,15 @@ Modifies or adds new fields to the transition tool interface. *Special consideration must be paid to this section if the EIP introduces a mechanism that requires the state transition tool to be aware whether the block it is processing is the fork-activation block. +##### New test-framework primitives + +Requires new abstractions in the test framework itself — expectation types, modifiers, helpers — beyond writing test functions with what already exists. + +- 0. Existing test primitives suffice. +- 1. Existing primitives need minor extension. +- 2. New expectation or modifier primitives are required, reusable within this EIP's own test suite. +- 3. New framework-level primitives are required that become a permanent part of the framework and are used by other EIPs' tests. + ##### Cryptography Introduces new cryptography mechanisms or modifies existing functionality that involves cryptography @@ -159,6 +207,8 @@ Introduces encoding changes at the transaction/block/interfaces level - 0. No encoding changes are introduced at the transaction, block, or interfaces levels. - 3. An encoding change is introduced at transaction, block or interfaces level (e.g. RLP -> SSZ). +*"Interfaces level" includes the Engine API. Score an Engine API encoding change (e.g. JSON -> SSZ) here. + ##### New transaction types Introduces a new transaction type @@ -209,6 +259,17 @@ Introduces or modifies mechanisms that could compromise the security of the chai - 2. The introduced mechanisms interact with a limited number of existing components, slightly altering their security assumptions and requiring a targeted security review or fuzzing. - 3. The introduced mechanisms interact with multiple existing components, including critical ones, substantially altering their security assumptions and requiring an extensive security review and fuzzing. +##### Unspecified behavior requiring cross-client consensus + +The EIP text does not determine the answer for cases a test can construct. Clients must agree on a previously unspecified detail before tests can be baselined. The cost here is coordination and re-baselining, not test writing. + +- 0. The EIP text determines the answer for every case a test could construct. +- 1. A few details are unspecified but have an obvious intended reading. +- 2. Details require client agreement before tests can be written, but they are localized. +- 3. A previously unspecified *and previously unobservable* behavior becomes consensus-critical; expect tests to be re-baselined on each round of EIP amendment. + +*Score this from the EIP's state at assessment time: whether it has client implementations, whether it has been through a devnet, and how many open questions remain on its discussion thread. + ##### Cross-EIP interactions Introduces or modifies mechanisms that affect other EIPs in either the same or past forks. @@ -217,21 +278,27 @@ Introduces or modifies mechanisms that affect other EIPs in either the same or p - 1. The EIP interacts with one or more other EIPs in a non-critical and limited way but can be tested independently for the most part. - 2. The EIP depends on or modifies one or more other EIPs such that coordinated testing and consideration is required, but interactions are limited in scope and not complex. - 3. The EIP has strong interdependencies with multiple EIPs, requiring extensive coordinated cross-EIP testing as well as potential re-design of existing test vectors. +- **+1 for every 3 additional interacting EIPs beyond the first 3**, each of which requires its own coordinated test cases. List the EIPs in the rationale. + +*This row is intentionally uncapped, unlike every other anchor: each interacting EIP is another axis of the test matrix, so a ceiling would make a 12-EIP product indistinguishable from a 3-EIP one. ### Checklist | Anchor | Score (0–3) | Rationale | |---|---:|---| | **EVM Gas rule changes** | | | +| **State-access ordering within opcode execution** | | | | **Blob gas accounting changes** | | | +| **State gas accounting changes** | | | | **New EVM gas refund** | | | | **Patterns affecting pre-existing tests** | | | +| **New invariant on pre-existing tests** | | | | **Transition-tool interface changes** | | | +| **New test-framework primitives** | | | | **Cryptography-related testing** | | | | **Edge/boundary conditions** | | | | **Block syncing changes** | | | | **Engine API changes** | | | -| **Engine API encoding changes** | | | | **Added system contracts** | | | | **Modified system contracts** | | | | **Added opcodes** | | | @@ -245,7 +312,8 @@ Introduces or modifies mechanisms that affect other EIPs in either the same or p | **New fork activation mechanism** | | | | **Performance risks** | | | | **Security risks** | | | -| **Cross-EIP interactions** | | | +| **Unspecified behavior requiring cross-client consensus** | | | +| **Cross-EIP interactions** (uncapped) | | | **Total: X** @@ -261,16 +329,83 @@ Introduces or modifies mechanisms that affect other EIPs in either the same or p | Category | Description | Value | |-----------|--------------|:----:| -| **Total Score** | Sum of all anchor scores (0–72) | **`XX`** | +| **Total Score** | Sum of all anchor scores (0–84 nominal; **Cross-EIP interactions** is uncapped, so there is no hard maximum) | **`XX`** | | **Complexity Tier** | Computed from total score | 🟢 / 🟡 / 🔴 | ##### Tier Interpretation | Tier | Range | Meaning | |------|--------|----------| -| 🟢 **Low Complexity** | **<10** | Minor feature or localized change. Existing tests are largely unaffected. Does not require intensive cross-EIP testing. | -| 🟡 **Medium Complexity** | **>=10<20** | Moderate change affecting multiple components. Requires moderate cross-EIP testing. | -| 🔴 **High Complexity** | **>=20** | Broad or deep impact on protocol behavior; high regression risk; and/or requiring intensive cross-EIP testing. | +| 🟢 **Low Complexity** | **<12** | Minor feature or localized change. Existing tests are largely unaffected. Does not require intensive cross-EIP testing. | +| 🟡 **Medium Complexity** | **>=12<23** | Moderate change affecting multiple components. Requires moderate cross-EIP testing. | +| 🔴 **High Complexity** | **>=23** | Broad or deep impact on protocol behavior; high regression risk; and/or requiring intensive cross-EIP testing. | + +##### Revision Notes + +> Background only. Nothing here is needed to fill in the checklist — the anchor +> definitions above are self-contained. Record the revision a completed +> assessment was scored against at the top of the document; **scores are not +> comparable across revisions**, so re-score rather than compare. + +###### Revision 2 — 28 anchors, nominal 0–84 + +Five anchors were added, one was removed, and **Cross-EIP interactions** was +uncapped: 24 anchors become 28. + +Tier thresholds were 10/20 against revision 1's 24-anchor, 72-point scale. They +are scaled by 84/72 to 12/23 so that tier membership stays stable as the anchor +set grows, rather than every EIP drifting upward a tier. + +The revision comes from +[the Amsterdam calibration](../complexity_assessments/calibration/README.md), +which measured each Amsterdam EIP's score against the work it actually produced in +`ethereum/execution-specs`. Every mature Amsterdam EIP landed within ±3 of the +score its measured work implies, except EIP-7928, which was short by 11 points +with no rows left to score on — it had 29 of a possible 33 across the ten rows +that applied to it. The five new rows are where that work should have been +recorded. See +[proposed-anchors.md](../complexity_assessments/calibration/proposed-anchors.md) +for the evidence behind each one. + +Per-row notes: + +- **State-access ordering within opcode execution.** EIP-7928 made every state + access consensus-observable via the BAL. That was a one-time transition, so the + row is worded for the world after it: it scores EIPs that *move* the ordering, + not the introduction of observability. EIP-7928 itself scored 0 on **Modified + opcodes** — correctly, since final EVM semantics were unchanged — which is how + the most expensive part of its work scored zero under the previous revision. + Scored retroactively against this row it is a 3: it reordered gas-charge sites + across an entire class of opcodes. Any pre-Amsterdam EIP is a 0 regardless of + what it did internally, since intra-opcode ordering was not consensus then. +- **New invariant on pre-existing tests.** Split out from **Patterns affecting + pre-existing tests**, which only captures *rework*. EIP-7928 scored 2 there and + would score 3 here: every Amsterdam test now validates a BAL whether or not it + has anything to do with access lists. +- **State gas accounting changes.** State gas was introduced by EIP-8037. The + checklist had a row for blob gas and none for this. +- **Cross-EIP interactions.** The `+1 per 3 additional EIPs` increment is a + judgement call, not a calibrated figure. It exists because EIP-7928 interacted + with 11 EIPs and scored the same 3 as an EIP interacting with three. +- **Engine API encoding changes** was removed. It described a wire-format change + at the Engine API layer (JSON -> RLP/SSZ), but such migrations are coordinated + outside the EIP process, so an EIP-scoped assessment has no use for the row. It + was scored 0 or left blank in all 28 assessments written against revision 1, so + removing it changes no historical total, and it never had anchor level + definitions. **Encoding changes (RLP/SSZ)** covers the case if it ever arises — + that row already reads "transaction, block or interfaces level". + +One finding is deliberately **not** reflected in the scoring: some of this cost +grows multiplicatively rather than additively, and no additive row at any weight +reproduces EIP-7928's measured cost. A multiplier row was considered and deferred +— with only one high-cost EIP observed, the data cannot identify which row should +multiply. Until a second fork supplies a second observation, evaluators record +multiplicative test-matrix growth under **Special Considerations**, as the +State-access ordering anchor instructs. + +###### Revision 1 — 24 anchors, 0–72 + +Original version. ## Consensus Specs