Skip to content

Export approved Studio revisions as reviewable eval candidates #283

Description

@anandpant

Goal

Turn an explicitly approved Studio revision and its source artifact into a portable, reviewable eval candidate without automatically promoting user content into maintained evals.

Current baseline

  • Maintained scenarios are checked into packages/diagram/scenarios and run through apps/eval-harness.
  • Studio has no approved-revision export contract today.
  • Code Mode artifacts retain a source-artifact relationship, but that alone does not define consent, review, or promotion into the maintained scenario suite.

Scope

  • Define an eval-candidate format that retains the before/after artifacts, source relationship, diagram type, and available generation metadata.
  • Add an explicit export path for a Studio revision only after both revision approval and a separate recorded consent decision; approval alone does not authorize export or reuse.
  • Define the human review/import boundary between candidates and maintained scenarios.
  • Document consent, access, retention, and deletion expectations before approved user content can be reused.

Non-goals

  • Automatically learning from every edit.
  • Automatically changing prompts, renderers, or models.
  • Treating approval as consent for public or indefinite reuse.

Acceptance criteria

  • An approved revision can be exported with a verifiable source/edited pair and provenance.
  • The export is portable and reviewable without depending on live Studio records.
  • Import into maintained evals is explicit and does not happen automatically.
  • Privacy, retention, access, and deletion behavior is documented and tested where enforceable.
  • At least one approved candidate can be reviewed and exercised through the current scenario/eval-harness workflow.

Current touchpoints

  • packages/studio/projects/
  • packages/diagram/agent/src/lib/code-mode-artifacts.ts
  • packages/diagram/scenarios/src/lib/scenarios.ts
  • apps/eval-harness/

This replaces the eval-loop portion of #63 and depends on durable Studio revision/approval in #282 plus the consent policy and record needed for candidate export.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions