test-chat

Extract test-generation trajectories from the SALT-NLP/SWE-chat dataset.

test-chat offers two extraction pipelines, selected with --extract:

• test-first (default) scans SWE-chat's conversations for each session's first user message and keeps the sessions whose opening prompt signals a test-generation request (a test term plus, by default, a generative verb).
• code-test-convo keeps the sessions where the user first asks for code generation and later, in the same conversation, asks for something test-like: a user prompt the dataset labels create new code followed by a strictly later user prompt our test matcher accepts, or a single create new code prompt that asks for both the code and its tests in the same breath.

In both modes, injected client and harness wrappers are stripped and AI-generated/automated prompts are excluded so the result reflects real human requests. Each kept trajectory is written as one record keyed by session_id, from which the full original trajectory can be re-opened against SWE-chat.

This project is derivative of SWE-chat: it does not redistribute the dataset, it only indexes and reshapes a slice of it. Access to SWE-chat is gated and requires Hugging Face authentication.

Install

uv sync

Usage

# Authenticate first (SWE-chat is gated).
huggingface-cli login        # or: export HF_TOKEN=...

# Extract test-generative trajectories into outputs/.
parse-swe-chat --output-dir outputs --dedupe

# Extract code-then-test conversations into outputs/.
parse-swe-chat --extract code-test-convo --output-dir outputs

# Re-open one session's full, ordered turn list by its session_id.
# DuckDB pushes the filter into conversations.parquet, so this fetches only the
# matching row groups instead of scanning the whole table.
recover-swe-chat <session_id>

test-first writes outputs/test_trajectories.jsonl (one record per trajectory) and outputs/summary.json (the aggregate run summary); code-test-convo writes outputs/code_test_convos.jsonl and outputs/code_test_convos_summary.json. The file names are distinct, so both modes can share one output directory.

Filtering: test-first

A session is kept only if its first user message reads as a genuine, human-written test-generation request. Two checks always apply, a prompt-cleanup pass and an intent gate are on by default, and a few options are opt-in.

Always applied

• First user message only. Each session is judged by its earliest user turn (smallest turn number) that the dataset does not label system_injected; sessions with no recoverable first prompt are dropped.
• Test-generation match. The prompt must contain a standalone test term (test, tests, unit tests, integration tests, test cases, pytest, junit, ...). Word boundaries are strict, so paths and identifiers like test.py, TestClass, test-gemini.txt, or latest do not count. In the default generative mode the prompt must also carry a generative verb (write, add, create, generate, implement, cover, extend, draft, ...), which keeps "write tests for X" and drops "the tests are failing".

On by default (disable with --any-test-intent)

• Intent gate. The first prompt must also carry SWE-chat's own prompt_intent label of test, the same intersection code-test-convo applies to its test side. This is the single most effective filter: the matcher's dominant failure mode is long pasted content (CI logs, plan documents, agent-harness text) where a generative verb and a test token co-occur by accident, and the label removes it. The cost is that the label marks a turn's primary intent, so test requests bundled into a larger ask ("fix issue 712 and write a test to repeat it", filed under debug) are dropped too. Pass --any-test-intent for matcher-only behavior. (The code-generation flavor of that bundled case — "build feature X and write tests for it" — is recovered by code-test-convo's mixed-turn handling.)

On by default (disable with --keep-ai-prompts)

• Injected-wrapper stripping. Client/harness scaffolding is removed before matching so the stored prompt is the human's own words: paired blocks like <system-reminder>, <system_instruction> (e.g. Conductor), and <ide_opened_file> are deleted, and slash-command <command-args> are unwrapped (tags dropped, content kept).
• Non-human prompt exclusion. Prompts that look machine-authored are dropped: plan/spec execution preambles ("Implement the following plan: ..."), role/persona system prompts ("You are a ...", "Your task is ..."), pasted agent/skill definitions (YAML front matter), pasted skill and plan-template bodies ("Base directory for this skill: ..."), continuation summaries, and teammate-message payloads. A prompt left empty after stripping is dropped too.

Opt-in

• --match-mode keyword — looser matching: any test term counts, no generative verb required.
• --pattern '<regex>' — replace the built-in matcher with a custom regex (overrides --match-mode).
• --any-test-intent — drop the intent gate: accept matcher-positive first prompts whatever their prompt_intent label says. The label is still recorded on every entry as an annotation.
• --dedupe — collapse prompts that are identical once whitespace and case are normalized, keeping the most recent by first-turn timestamp.

Filtering: code-test-convo

A session is kept when a code-generation request is followed later in the same conversation by a test-like request. Only rows the dataset labels as genuine human prompts (role='user', turn_type='user_prompt') are considered, and the same prompt-cleanup pass as test-first applies per turn (injected-wrapper stripping plus non-human turn exclusion; disable with --keep-ai-prompts).

• Code side: the dataset's own label. A turn counts as code generation when SWE-chat's prompt_intent label says create new code. There is no code-generation regex for this side; the label decides membership, and it is far more reliable than word matching here.
• Test side: our matcher AND the dataset label. A turn counts as test-like when the same matcher as test-first accepts it (--match-mode / --pattern apply) and, by default, the turn's prompt_intent label is test. The intersection kills prompts whose test vocabulary is incidental (pasted templates, harness text); pass --any-test-intent to accept matcher-positive turns under any label.
• Mixed turns: the text vouches when the label cannot. A turn labeled create new code whose text asks for tests in the same breath — a generative verb within a few words of a test term, with no sentence break or for/to in between ("generate a feature and create tests for it") — also joins the test side. Because the create new code label cannot vouch for test intent, this uses a stricter proximity matcher than the ordinary generative mode: "generate feature X and run the tests" does not qualify. A custom --pattern replaces this matcher too. Disable with --no-mixed-intent.
• A session qualifies when a test-side turn appears after a code-generation turn in the session's order, or when it has a mixed turn — that turn asks for both at once and qualifies on its own. The gate reads sequence position, not turn numbers, so a turn the dataset left unnumbered still counts and two turns sharing a turn number are ordered as they appear. Unnumbered turns sort last (the dataset's own NULLS LAST convention), so such a turn can follow a code turn but never precede one.
• One record per session, carrying code_prompts and test_prompts: every qualifying prompt on each side as {turn number: prompt text}, not one anchor pair, so sessions that iterate between code and tests keep all their turns. As in test-first, the stored text is the cleaned prompt the matcher saw, not the raw turn. Keys are strings (JSON object keys always are) in ascending turn order; a key in both mappings is a mixed code+test prompt and the intersection is the only mixed-turn signal. A turn the dataset left unnumbered has no key to file it under and is omitted even when it is what qualified the session, so a mapping can be smaller than the turns behind it — or empty. user_prompt_turns always counts them. The full trajectory stays recoverable with recover-swe-chat <session_id>.
• --any-test-intent — loosen the test side to matcher-only: accept matcher-positive turns whatever their prompt_intent label says.
• --no-mixed-intent — do not treat mixed create new code turns as test-like; restores the strict label-and-matcher test side only.
• --dedupe has no effect here: with no prompt text stored there is nothing to compare, and one record per session is already unique.

The bulk loaders in this mode use DuckDB column pruning over conversations.parquet (only the prompt rows and columns are downloaded), so a full run is much cheaper than the streaming test-first scan.

Source dataset

SWE-chat: Coding Agent Interactions From Real Users in the Wild. Joachim Baumann, Vishakh Padmakumar, Xiang Li, John Yang, Diyi Yang, Sanmi Koyejo.

• Paper: https://arxiv.org/abs/2604.20779
• Project: https://www.swe-chat.com/
• Dataset: https://huggingface.co/datasets/SALT-NLP/SWE-chat

@article{baumann2026swechat,
  title={SWE-chat: Coding Agent Interactions From Real Users in the Wild},
  author={Baumann, Joachim and Padmakumar, Vishakh and Li, Xiang and Yang, John and Yang, Diyi and Koyejo, Sanmi},
  year={2026},
  journal={arXiv preprint arXiv:2604.20779}
}