Skip to content

Goalpost: v19 bounded-memory and CPU performance contract #758

Description

@flyingrobots

Objective

Make performance a versioned, reviewable v19 contract rather than an informal local observation.

git-warp must prove both:

  1. causal materialization and observation do not regress materially in CPU cost; and
  2. a graph whose logical size is substantially larger than the configured JavaScript heap can still be observed to completion with a bounded working set.

This goalpost extends #734 and #738. It is not complete merely because one optimized path is faster once.

Required benchmark surfaces

Execution modes

  • cold retained materialization from Git history;
  • warm reuse from git-cas materialization state;
  • incremental materialization after a bounded causal suffix;
  • streamed observation, including time to first Reading and completion;
  • cache-miss and cache-hit behavior reported separately.

Metrics

  • process user CPU, system CPU, and total CPU;
  • wall-clock duration as a diagnostic, not the only gate;
  • time to first Reading and steady-state throughput;
  • peak RSS and peak JavaScript heap;
  • logical corpus size, tick count, node/property/edge counts, and emitted Reading count;
  • Git command/object read/object write counts where instrumentation permits;
  • git-cas hit, miss, staged, promoted, and restored-byte evidence.

Oversized-graph proof

The corpus must be generated and consumed lazily. The benchmark subprocess must run with an explicit --max-old-space-size and operate on a logical graph at least several times larger than that cap. The proof fails if it silently creates a complete fixture array, materializes the full graph, omits expected Readings, exceeds the memory budget, or relies on an unbounded compatibility API.

Merge gates

  • correctness, expected cardinality, termination, and heap-cap compliance are hard deterministic gates;
  • base and head execute on the same runner and toolchain;
  • CPU comparisons use warmups plus repeated samples, medians, an absolute noise floor, and a reviewed relative-regression budget;
  • benchmark ordering must avoid systematically favoring base or head;
  • the initial calibration change may establish the baseline, but subsequent baseline movement requires explicit review and rationale;
  • shared-runner wall-clock noise must never be disguised as a precise CPU contract.

Publication

Every main-branch run publishes a machine-readable, versioned benchmark result and a human-readable summary. Pull requests show base/head deltas. Historical results are exposed without force-pushing generated Git branches; GitHub Pages artifacts or an equivalent append-safe publication path are preferred.

Acceptance criteria

  • A versioned corpus and result schema are committed.
  • Cold, warm, incremental, and streamed scenarios have reproducible runners.
  • The oversized-graph subprocess completes under its explicit heap cap.
  • Peak RSS/heap evidence and expected Reading cardinality are captured.
  • Base/head CPU comparison is calibrated and blocks material regressions.
  • Main publishes durable JSON and human-readable benchmark summaries.
  • Pull requests expose deltas and fail closed when required evidence is absent.
  • Baseline updates require an intentional reviewed change.
  • The v19 release witness cites benchmark runs and the active thresholds.

Non-goals

  • Claiming host-independent nanosecond precision.
  • Treating a microbenchmark as end-to-end proof.
  • Optimizing benchmark-only code paths.
  • Allowing performance thresholds to weaken correctness or causal evidence.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:testingPrimary work area: testing.priority:asapImmediate release pressure.status:activeSomeone is actively working this issue.type:goalpostRoadmap umbrella milestone issue.

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions