Skip to content

Register --prompts soul variant reading from SOUL_MD_PATH #4

@EYH0602

Description

@EYH0602

Context

SOULFuzz is a benchmark-agnostic
prompt-optimization framework. It iteratively mutates a SOUL.md artifact (a
markdown file injected as system context) and evaluates each candidate against
a pluggable benchmark. KoraBench is the first adapter, used for the
child-safety case study.

To close the optimization loop, the SOULFuzz Python adapter needs KoraBench to
read the system-prompt body from a file path the adapter controls per
invocation. This is touch point #2 of the four-edit adapter contract
described in the SOULFuzz engineering plan §6 — minimal, additive, and
intended to be upstreamable.

Ask

Register a new --prompts soul variant in the run command, alongside the
existing default / child variants. Behavior:

  1. When invoked with --prompts soul, KoraBench reads the system-prompt body
    from process.env.SOUL_MD_PATH and uses that as the system prompt for the
    target model.
  2. If SOUL_MD_PATH is unset or the file is missing, fall back to
    data/souls/seed.md resolved relative to the KoraBench checkout root. If
    that file is also missing, exit with a clear error (not a silent fallback
    to default).
  3. Read-once-per-run: load the file when the prompt variant is resolved, not
    per-scenario. The file does not change during a single yarn kora run
    invocation.
  4. No changes to the JSON output schema, judge plumbing, or scenarios are
    required. This is strictly a new entry in the prompt-variant registry.

Acceptance criteria

  • yarn kora run <target> --prompts soul --limit 1 succeeds when
    SOUL_MD_PATH points at a readable markdown file, and the file content
    appears as the system prompt in the resulting transcript.
  • Same invocation with SOUL_MD_PATH unset and data/souls/seed.md
    present uses the seed file.
  • Same invocation with both unset and missing exits non-zero with a
    message naming SOUL_MD_PATH.
  • A unit test covers the three branches above (env set / env unset with
    seed / both missing).
  • No regression to --prompts default or --prompts child.

Out of scope (separate issues / PRs)

References

  • SOULFuzz repo: https://github.com/SecurityLab-UCD/SOULFuzz
  • Tracking PR-03 on SOULFuzz side: lands the Python adapter
    (Evaluator + Splits + Injector + seed) and bumps this fork's
    submodule pin once this issue is resolved.
  • The seed for data/souls/seed.md is a verbatim copy of the existing
    --prompts child body — the SOULFuzz adapter will produce it at bootstrap
    time, so no new content needs to be committed in this fork.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions