Context
SOULFuzz is a benchmark-agnostic
prompt-optimization framework. It iteratively mutates a SOUL.md artifact (a
markdown file injected as system context) and evaluates each candidate against
a pluggable benchmark. KoraBench is the first adapter, used for the
child-safety case study.
To close the optimization loop, the SOULFuzz Python adapter needs KoraBench to
read the system-prompt body from a file path the adapter controls per
invocation. This is touch point #2 of the four-edit adapter contract
described in the SOULFuzz engineering plan §6 — minimal, additive, and
intended to be upstreamable.
Ask
Register a new --prompts soul variant in the run command, alongside the
existing default / child variants. Behavior:
- When invoked with
--prompts soul, KoraBench reads the system-prompt body
from process.env.SOUL_MD_PATH and uses that as the system prompt for the
target model.
- If
SOUL_MD_PATH is unset or the file is missing, fall back to
data/souls/seed.md resolved relative to the KoraBench checkout root. If
that file is also missing, exit with a clear error (not a silent fallback
to default).
- Read-once-per-run: load the file when the prompt variant is resolved, not
per-scenario. The file does not change during a single yarn kora run
invocation.
- No changes to the JSON output schema, judge plumbing, or scenarios are
required. This is strictly a new entry in the prompt-variant registry.
Acceptance criteria
Out of scope (separate issues / PRs)
References
- SOULFuzz repo: https://github.com/SecurityLab-UCD/SOULFuzz
- Tracking PR-03 on SOULFuzz side: lands the Python adapter
(Evaluator + Splits + Injector + seed) and bumps this fork's
submodule pin once this issue is resolved.
- The seed for
data/souls/seed.md is a verbatim copy of the existing
--prompts child body — the SOULFuzz adapter will produce it at bootstrap
time, so no new content needs to be committed in this fork.
Context
SOULFuzz is a benchmark-agnostic
prompt-optimization framework. It iteratively mutates a
SOUL.mdartifact (amarkdown file injected as system context) and evaluates each candidate against
a pluggable benchmark. KoraBench is the first adapter, used for the
child-safety case study.
To close the optimization loop, the SOULFuzz Python adapter needs KoraBench to
read the system-prompt body from a file path the adapter controls per
invocation. This is touch point #2 of the four-edit adapter contract
described in the SOULFuzz engineering plan §6 — minimal, additive, and
intended to be upstreamable.
Ask
Register a new
--prompts soulvariant in theruncommand, alongside theexisting
default/childvariants. Behavior:--prompts soul, KoraBench reads the system-prompt bodyfrom
process.env.SOUL_MD_PATHand uses that as the system prompt for thetarget model.
SOUL_MD_PATHis unset or the file is missing, fall back todata/souls/seed.mdresolved relative to the KoraBench checkout root. Ifthat file is also missing, exit with a clear error (not a silent fallback
to
default).per-scenario. The file does not change during a single
yarn kora runinvocation.
required. This is strictly a new entry in the prompt-variant registry.
Acceptance criteria
yarn kora run <target> --prompts soul --limit 1succeeds whenSOUL_MD_PATHpoints at a readable markdown file, and the file contentappears as the system prompt in the resulting transcript.
SOUL_MD_PATHunset anddata/souls/seed.mdpresent uses the seed file.
message naming
SOUL_MD_PATH.seed / both missing).
--prompts defaultor--prompts child.Out of scope (separate issues / PRs)
--prompts soulvariant reading fromSOUL_MD_PATH#4 from the SOULFuzz engineering plan).custom-cfa-*model slugs and the CFA workspace (touch points docs: align README node requirement with .nvmrc (24 LTS) #1 + thepackages/cfa/workspace).intimacyCalibrationmechanism (touch point [feat] register openai as built-in openai-compatible provider #3).References
(
Evaluator+Splits+Injector+ seed) and bumps this fork'ssubmodule pin once this issue is resolved.
data/souls/seed.mdis a verbatim copy of the existing--prompts childbody — the SOULFuzz adapter will produce it at bootstraptime, so no new content needs to be committed in this fork.