Skip to content

feat: record and replay mode for reproducible runs #55

Description

@JNK234

Roadmap priority: B6

Summary

Record every prompt/response pair in a run, and replay them offline for an identical re-run without API calls.

Why it matters

Three concrete uses:

  • Reproducibility — an LLM model is not currently reproducible even with a fixed random seed. Replay makes a published run exactly repeatable.
  • Demos without keys — present a model at a workshop or conference with no network and no API key.
  • Classroom — students explore a recorded run at zero cost.

mesa-llm has an equivalent (@record_model decorator, SimulationRecorder, plus a CLI replay viewer), and it is one of the genuinely good ideas in their codebase.

What needs to be done

  1. Recording: commands like llm:start-recording <file> / llm:stop-recording, capturing prompt, response, model, provider, timestamp, and agent identity.
  2. Replay: llm:load-recording <file> puts the extension into a mode where calls are served from the recording instead of the network.
  3. Decide the matching rule on replay — strict call-order sequence, or content-addressed lookup (which converges with caching, B5).
  4. Behaviour on a miss: error, fall through to a live call, or return a placeholder. Must be explicit.
  5. File format — JSONL is the obvious fit and is diffable.
  6. Interaction with clear-all and multiple agents; recordings must survive being shared alongside a .nlogox.

Open questions

  • Should recording be automatic under BehaviorSpace, given B7 (experiment logging) wants much the same data?
  • Does replay need to reproduce timing/latency, or return instantly? Instant is more useful; timing matters only for demo realism.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions