Skip to content

[Critical] Add latency benchmarks and SLA targets #55

@jessfortemnaturae8717

Description

@jessfortemnaturae8717

Problem

No performance benchmarks or SLA targets exist. We can't promise customers any latency guarantees.

Current state:

  • TraceBuilder tracks duration_ms but nothing consumes it
  • No vitest benchmarks
  • No CI gates for performance regression
  • No p95/p99 measurements

Risk

  • Can't set SLA for Pro/Enterprise tiers
  • Performance regressions go unnoticed
  • Customers experience unpredictable latency

Solution

Add comprehensive benchmarking:

// bench/evaluation.bench.ts
import { bench, describe } from 'vitest';

describe('evaluation latency', () => {
  bench('small plan (10 resources)', async () => { ... });
  bench('medium plan (100 resources)', async () => { ... });
  bench('large plan (1000 resources)', async () => { ... });
  bench('with state lookups (mocked)', async () => { ... });
});

Tasks

  • Add vitest bench configuration
  • Create benchmark suite for evaluation latency
  • Benchmark state lookup latency (mocked + real)
  • Add CI job to run benchmarks on PR
  • Set performance gates (fail if p95 > threshold)
  • Add latency histogram to /metrics endpoint
  • Document SLA targets in docs

SLA Targets (Proposed)

Scenario p50 p95 p99
Small plan (≤10 resources) < 50ms < 100ms < 200ms
Medium plan (≤100 resources) < 200ms < 500ms < 1s
Large plan (≤1000 resources) < 1s < 3s < 5s
With live state (per resource) +50ms +100ms +200ms

Priority: P1 — Required for Pro tier launch

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions