Add audit taskflows v2: contested, reproduction-gated vulnerability discovery

[anticomputer]

What this adds

Audit taskflows v2: a five-stage vulnerability-discovery pipeline that lives alongside the existing v1 taskflows in its own audit_v2 namespace and does not change them.

The stages are survey (decompose the repo into components and map where untrusted data crosses a trust boundary), hunt (three model families hunt each component in parallel, then a dedup pass folds convergent findings), contest (a prosecutor argues a finding is real, a defender from a different model family argues it is not, and an adjudicator from a third family decides), reproduce (stand the target up in a container and dynamically show the flow reaches the sink), and report (write the findings up, then verify every claim against the ledger).

The lifecycle is enforced in code, not prompts

A finding's state is derived by the new finding_ledger MCP server and never set by a model. No tool accepts a state as an argument: a model records evidence, and the backend decides what that evidence entitles the finding to. Adjudication is the only path to confirmed and requires that both advocates actually filed; reproduction only promotes from confirmed; a reproduced finding is immune to later adjudication. The ledger is also where the stages meet: multi-model fan-outs do not feed the implicit last-tool-result channel, so a store that survives a crash is the only place they can share state.

Reproduction proves reachability, not exploitation

The reproduce stage validates that attacker-controlled input actually reaches the dangerous sink at runtime, using a benign, distinctive marker or lightweight instrumentation rather than a weaponised exploit. That is enough for a maintainer to confirm a bug and write the patch, and it is also what keeps the public models willing to do the work: framed as exploitation, the newest models soft-refuse under content filtering; framed as reachability, they validate cleanly and record literal container output.

Model configuration

Three families do the analysis so no advocate ever grades a sibling's case: gpt-5.6-sol, claude-opus-* and gemini-3.6-flash. Every slot pins its backend and API type explicitly (responses where available, native /v1/messages for Anthropic with adaptive thinking, chat completions for Gemini). grok-4.5 is deliberately absent from every role: CAPI rejects any vulnerability-analysis content sent to it with 403 permission-denied, failed check SAFETY_CHECK_TYPE_CYBER, down to a five-line snippet. The reproduction slot runs claude-opus-4.8 rather than claude-opus-5, which soft-refuses the agentic reproduction loop under content filtering.

Validation

Every stage has been run live end-to-end against a vulnerable test app: three bugs (OS command injection, SQL injection, path traversal) are found by multiple families, deduplicated with their provenance unioned, confirmed through the contest, and each reproduced in a container with a control case and a benign marker observed reaching the sink. 329 tests pass and every audit_v2 prompt renders through a corpus gate.

Dependency note

The live pipeline needs the OpenAI SDK bump in seclab-taskflow-agent#288 to run on Python 3.14; without it the Responses streaming path raises on a typing.Union attribute.

[Copilot]

Pull request overview

This PR introduces audit taskflows v2, a new, multi-stage vulnerability discovery pipeline that adds adversarial validation and runtime reachability reproduction, while keeping the existing v1 audit taskflows unchanged. It adds new MCP-backed durable stores (survey + finding ledger), new taskflows/prompts/personalities for the five-stage lifecycle, container tooling for reproduction, and a corpus gate to ensure shipped taskflows and prompts lint and render correctly.

Changes:

• Add audit_v2 pipeline taskflows (survey/hunt/contest/reproduce/report) plus shared prompts and personas for each role.
• Add new audit_v2 MCP servers and durable SQLite-backed backends (repo survey + finding ledger) with concurrency-tolerant schema initialization.
• Add end-to-end validation support: tests for stores and corpus gating, a reproduction container image + publishing hooks, and a runner script for the v2 pipeline.

Show a summary per file

File	Description
tests/test_taskflow_corpus.py	Adds corpus gate covering YAML validation, taskflow linting, and audit_v2 prompt rendering/over checks.
tests/test_repo_survey.py	Adds behavioral tests for the audit_v2 repo survey backend (dedup/validation/durability + toolbox wiring).
tests/test_finding_ledger.py	Adds extensive tests for ledger lifecycle rules, dedup, attribution, durability, and MCP tool exposure.
tests/conftest.py	Forces MCP servers’ state/log dirs into a temp location to avoid mutating user/global state during tests.
src/seclab_taskflows/toolboxes/container_shell_reproduction.yaml	Introduces a dedicated reproduction toolbox pointing at the reproduction container image.
src/seclab_taskflows/toolboxes/audit_v2_repo_survey.yaml	Wires the repo survey MCP server as a toolbox with operator guidance.
src/seclab_taskflows/toolboxes/audit_v2_finding_ledger.yaml	Wires the finding ledger MCP server as a toolbox with lifecycle rules guidance.
src/seclab_taskflows/taskflows/audit_v2/survey.yaml	Stage 1: fetch source, map components + entry points, and publish typed outputs.
src/seclab_taskflows/taskflows/audit_v2/hunt.yaml	Stage 2: multi-model hunting fanout, post-hunt attribution, candidate fetch + dedup.
src/seclab_taskflows/taskflows/audit_v2/contest.yaml	Stage 3: prosecution/defense/adjudication fanouts using the ledger as join point.
src/seclab_taskflows/taskflows/audit_v2/reproduce.yaml	Stage 4: sequential reproduction of confirmed findings inside a sandboxed container.
src/seclab_taskflows/taskflows/audit_v2/report.yaml	Stage 5: report drafting from the ledger + a verification pass against recorded evidence.
src/seclab_taskflows/taskflows/audit_v2/README.md	Documents audit_v2 goals, lifecycle, model roles, running instructions, and safety constraints.
src/seclab_taskflows/prompts/audit_v2/engagement.yaml	Shared “authorized audit” framing and scope constraints.
src/seclab_taskflows/prompts/audit_v2/evidence_rules.yaml	Shared non-speculation and citation requirements for all audit stages.
src/seclab_taskflows/prompts/audit_v2/finding_contract.yaml	Defines what qualifies as a “finding” and filing rules for hunters.
src/seclab_taskflows/prompts/audit_v2/contest_rules.yaml	Defines adversarial contest mechanics, roles, and allowed positions.
src/seclab_taskflows/prompts/audit_v2/reproduction_rules.yaml	Defines reachability-focused reproduction requirements and outcome recording.
src/seclab_taskflows/prompts/audit_v2/severity_rubric.yaml	Defines severity levels and how to apply preconditions/reachability.
src/seclab_taskflows/personalities/vulnerability_hunter.yaml	Persona for hunt stage emphasizing taint/path tracing and non-padding.
src/seclab_taskflows/personalities/exploit_prosecutor.yaml	Persona for prosecution role in contest stage.
src/seclab_taskflows/personalities/exploit_defender.yaml	Persona for defense role in contest stage.
src/seclab_taskflows/personalities/finding_adjudicator.yaml	Persona for adjudication role with citation checking focus.
src/seclab_taskflows/personalities/reproduction_engineer.yaml	Persona for reproduction role emphasizing literal observed evidence.
src/seclab_taskflows/mcp_servers/audit_v2/schema_init.py	Adds race-tolerant schema initialization for concurrent server startups.
src/seclab_taskflows/mcp_servers/audit_v2/repo_survey_models.py	Defines SQLAlchemy schema + projections for repo survey wire contracts.
src/seclab_taskflows/mcp_servers/audit_v2/repo_survey.py	Implements repo survey backend + MCP tool surface.
src/seclab_taskflows/mcp_servers/audit_v2/finding_ledger_models.py	Defines SQLAlchemy models + constants for ledger lifecycle.
src/seclab_taskflows/mcp_servers/audit_v2/finding_ledger.py	Implements ledger backend + MCP tool surface + lifecycle enforcement.
src/seclab_taskflows/containers/base/Dockerfile	Installs upstream Go toolchain into the base image to avoid old Debian Go.
src/seclab_taskflows/containers/reproduction/Dockerfile	Adds a reproduction container image with debugging/runtime tooling.
src/seclab_taskflows/configs/model_config_audit_v2.yaml	Defines default v2 role→model mapping and backend/api_type per role.
src/seclab_taskflows/configs/model_config_audit_v2_lowercost.yaml	Provides a cheaper drop-in model mapping for exploratory runs.
scripts/build_container_images.sh	Adds a reproduction build target and includes it in all.
scripts/audit_v2/run_audit_v2.sh	Adds a runner for audit_v2 stages with model override and stage selection.
.github/workflows/publish-container-images.yml	Publishes the new reproduction image to GHCR.

Review details

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Suppressed comments (2)

src/seclab_taskflows/mcp_servers/audit_v2/finding_ledger.py:370

• store_reproduction_attempt also does not verify that the repo argument matches the finding’s repo. This can record reproduction attempts against the wrong repository context, which is especially risky because reproduced findings become immune to later adjudication.

        with Session(self.engine) as session:
            finding = session.get(Finding, finding_id)
            if finding is None:
                return f"No finding with id {finding_id}"

src/seclab_taskflows/mcp_servers/audit_v2/finding_ledger.py:286

• adjudicate_finding does not verify that the repo argument matches finding.repo. This allows a caller to confirm/reject a finding while supplying a different repo context, which breaks the per-repository isolation the pipeline relies on (and can lead to state changes being attributed to the wrong audit run).

        with Session(self.engine) as session:
            finding = session.get(Finding, finding_id)
            if finding is None:
                return f"No finding with id {finding_id}"
            if finding.state == STATE_REPRODUCED:

• Files reviewed: 36/36 changed files
• Comments generated: 5
• Review effort level: Lite

[anticomputer]

Addressed the review feedback in 84e409f:

• Per-repo invariant: store_contest_verdict, adjudicate_finding and store_reproduction_attempt now verify the target finding belongs to the named repo, via a shared _repo_mismatch helper that attribute_finding also routes through. Previously only attribution and merge checked this, so a stage could attach a verdict, adjudication or reproduction attempt to another repo's finding.
• store_entry_point now fetches the component and refuses to attach when component.repo != repo, instead of only checking the component exists.
• Fixed the tautological assert isinstance(list(value), list) in test_audit_v2_over_expressions_resolve to assert isinstance(value, list).
• README model table now lists the reproduction slot as claude-opus-4.8 (matches model_config_audit_v2.yaml), and the lifecycle diagram uses the real outcome reproduced rather than reachable.

Added regression tests for each of the four new cross-repo guards. Full suite: 333 passed, 1 xfailed; ruff clean.

[Copilot]

Review details

Suppressed comments (2)

src/seclab_taskflows/mcp_servers/audit_v2/finding_ledger.py:453

• get_finding() returns verdicts and reproduction_attempts from SQLAlchemy queries without an explicit ordering. SQLite often appears to preserve insertion order, but this is not guaranteed, and it can make the API nondeterministic (and can make tests/test_finding_ledger.py::TestAdjudication::test_adjudication_is_recorded_as_a_verdict flaky if row order changes). Ordering these lists by id (or created_at) keeps results stable.

            data["verdicts"] = [
                verdict_to_dict(v)
                for v in session.query(ContestVerdict).filter_by(finding_id=finding_id).all()
            ]
            data["reproduction_attempts"] = [
                attempt_to_dict(a)
                for a in session.query(ReproductionAttempt).filter_by(finding_id=finding_id).all()
            ]

src/seclab_taskflows/mcp_servers/audit_v2/finding_ledger.py:572

• The MCP tool docstring for find_similar_findings says it is "to avoid duplicates", but the audit_v2 design explicitly relies on hunters filing independently and not pre-checking for duplicates (dedup happens later). This wording could steer models into calling this tool during hunting and reduce the corroboration signal the pipeline is designed to preserve.

def find_similar_findings(
    owner: str = Field(description="The owner of the GitHub repository"),
    repo: str = Field(description="The name of the GitHub repository"),
    component: str = Field(description="Component to match", default=""),
    vuln_class: str = Field(description="Vulnerability class to match", default=""),
):
    """Find existing findings in the same component and class, to avoid duplicates."""
    repo = process_repo(owner, repo)
    return json.dumps(backend.find_similar_findings(repo, component, vuln_class))

• Files reviewed: 36/36 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Copilot]

Review details

Suppressed comments (2)

tests/conftest.py:28

• tempfile.mkdtemp() creates a real directory that is never cleaned up. This will leave seclab-taskflows-tests-* directories behind on developer machines/CI runners. Also, _TMP is created even when all relevant env vars are already set, which is unnecessary.
  src/seclab_taskflows/mcp_servers/audit_v2/finding_ledger.py:581
• The find_similar_findings tool docstring says it's "to avoid duplicates", but audit_v2 intentionally preserves duplicates until the dedicated deduplication step to capture cross-model convergence signal. This wording is likely to push models toward the wrong behavior.

    """Find existing findings in the same component and class, to avoid duplicates."""

• Files reviewed: 37/37 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Copilot]

Review details

Suppressed comments (2)

src/seclab_taskflows/mcp_servers/audit_v2/finding_ledger.py:445

• get_findings(..., state=...) accepts any string and will silently return an empty list for typos (for example confirmed with a trailing space). That can mask configuration errors and make later stages think there are no findings.

Validate state against FINDING_STATES and fail fast on unknown values.

    @normalizes_repo
    def get_findings(self, repo, state=None):
        with Session(self.engine) as session:
            query = session.query(Finding).filter_by(repo=repo)
            if state:

src/seclab_taskflows/mcp_servers/audit_v2/finding_ledger.py:582

• The tool docstring says find_similar_findings is “to avoid duplicates”, but the v2 pipeline explicitly instructs hunters not to use it for filing decisions (it’s intended for the later dedup step / related-work lookup). This mismatch can lead to the wrong stage relying on it.

Update the docstring to reflect its intended use in audit_v2.

def find_similar_findings(
    owner: str = Field(description="The owner of the GitHub repository"),
    repo: str = Field(description="The name of the GitHub repository"),
    component: str = Field(description="Component to match", default=""),
    vuln_class: str = Field(description="Vulnerability class to match", default=""),
):
    """Find existing findings in the same component and class, to avoid duplicates."""
    repo = process_repo(owner, repo)

• Files reviewed: 37/37 changed files
• Comments generated: 1
• Review effort level: Lite

[Copilot]

Review details

Suppressed comments (6)

src/seclab_taskflows/mcp_servers/audit_v2/repo_survey.py:141

• store_entry_point() stores file verbatim. Normalizing whitespace avoids creating duplicate entry points for the same site when the caller includes leading/trailing spaces or newlines.

        trust_boundary = _require(trust_boundary, TRUST_BOUNDARIES, "trust_boundary")
        line = int(line or 0)
        with Session(self.engine) as session:

src/seclab_taskflows/mcp_servers/audit_v2/repo_survey.py:213

• get_component() fetches entry points by component_id only. While cross-repo attachment is guarded on write, filtering by repo here makes the read path robust against any accidental cross-repo records (or future schema changes) and prevents leaking entry points from another repo if the DB is ever inconsistent.

            rows = (
                session.query(EntryPoint)
                .filter(EntryPoint.component_id == component_id)
                .all()
            )

src/seclab_taskflows/mcp_servers/audit_v2/repo_survey.py:90

• store_component() uses location as the deduplication key, but it is not normalized. If a model emits trailing/leading whitespace (common in LLM outputs), the same component can be stored twice under slightly different spellings and the hunt fan-out will duplicate work.

This issue also appears on line 139 of the same file.

        kind = _require(kind, COMPONENT_KINDS, "kind", default=KIND_OTHER)
        with Session(self.engine) as session:

src/seclab_taskflows/mcp_servers/audit_v2/schema_init.py:44

• create_all_tolerating_races() only retries when the error string contains "already exists". Under concurrent SQLite DDL, a common transient failure mode is also "database is locked"; treating that as retryable improves reliability when many servers start at once.

        except OperationalError as exc:
            if "already exists" not in str(exc).lower():
                raise
            # Another server won this table. It may still be creating the
            # rest, so pause and re-check rather than assuming all are ready.

src/seclab_taskflows/mcp_servers/audit_v2/finding_ledger.py:590

• The find_similar_findings tool docstring says it is "to avoid duplicates", but the audit_v2 design (and prompts) explicitly say hunters should not use it to decide whether to file. Aligning this docstring with the intended usage prevents confusing future prompt/tool authors.

    """Find existing findings in the same component and class, to avoid duplicates."""

tests/conftest.py:18

• tempfile.mkdtemp() is never cleaned up, so local test runs will leave seclab-taskflows-tests-* directories behind. Registering an atexit cleanup keeps the "set env before imports" behavior but avoids accumulating temp directories.

• Files reviewed: 37/37 changed files
• Comments generated: 1
• Review effort level: Lite

[Copilot]

Review details

Suppressed comments (1)

src/seclab_taskflows/mcp_servers/audit_v2/finding_ledger.py:590

• find_similar_findings tool docstring suggests using it “to avoid duplicates”, but the audit_v2 finding contract explicitly says hunters must not use it to decide whether to file (they should file first and let the later dedup step merge). This mismatch is likely to cause incorrect tool use by models and reduce the convergence signal the pipeline relies on.

    """Find existing findings in the same component and class, to avoid duplicates."""

• Files reviewed: 37/37 changed files
• Comments generated: 0 new
• Review effort level: Lite