Script to kill TraceLens Agent eval jobs

[kyle-hoffmeyer]

Summary

This PR adds a script kill_eval_jobs.sh that kills all eval jobs spawned by run_repeatability_parallel.sh and generate_ref.sh.

The analysis eval pipeline (run_repeatability_parallel.sh, and generate_ref.sh) fans out into a large tree of processes per run: the orchestrator scripts, their worker subshells, the Cursor agent/pi analysis and eval calls, the node worker each of those spawns, and the scripted-eval python processes. There was no clean way to stop a run:

The processes from a single run don't share one kill handle. Here is a live snapshot of the tree from one generate_ref.sh run, showing the different kernel IDs:

PID	PPID	PGID	SID	Process
110120	1	110120	110085	bash …/generate_ref.sh (the script; reparented to init)
110127	110120	110120	110085	bash …/generate_ref.sh (worker subshell)
110136	110127	110120	110085	bash …/generate_ref.sh (worker subshell)
110138	110136	110138	110085	timeout 1800 agent … (starts a new group)
110139	110138	110138	110085	agent … (Cursor CLI)
110299	110139	110138	110085	node …/cursor-agent/… (the LLM worker)
110987	110139	110987	110987	bash -c … (agent tool-call; setsid → own group and session)

• Killing the top-level script's PID doesn't cascade — its children reparent to init and keep going, and the pipeline's retry loop respawns new agents.
• A process-group kill (kill -- -PGID) can't cover the run in one call, because a single run is spread across multiple process groups: the timeout wrapper around each agent call starts a new group, so the timeout/agent/node worker chain sits outside the script's group.
• A session kill doesn't help either — the agent's tool-call subprocesses call setsid and leave the session entirely, so there's no single session that contains the whole tree.

This PR tags each process with an inherited environment marker and adds a kill_eval_jobs.sh script that that reliably kills an entire eval run by killing jobs with this environment marker.

Motivation

The core difficulty is identifying which processes belong to an eval run. Matching by command name is unreliable (the real workers are generic node and python processes), and matching by parentage or process group fails because of reparenting and the agent CLI's own process group. What every process in a run does have in common is that it descends from one of the two entrypoint scripts. An inherited environment marker survives reparenting, process-group changes, and renames.

Solution

Both entrypoints export TRACELENS_EVAL_JOB=1, which is inherited by every descendant. kill_eval_jobs.sh then finds the whole run by scanning /proc/<pid>/environ for that marker and signalling those PIDs.

Implementation

agent_evals/Analysis/eval_scripts/kill_eval_jobs.sh (new)

A standalone cleanup tool. It scans /proc/<pid>/environ for TRACELENS_EVAL_JOB=1 (guarding on file ownership so it only ever touches the current user's processes) and signals the matches. Because the marker is unique to the pipeline, unrelated agent, claude, or python sessions are never touched.

Two subtleties drove the design:

• The kill isn't atomic. A process caught mid-run can fork a child — a retry-backoff sleep, or a fresh agent — in the window between the scan and the kill; that child inherits the marker but wasn't in the snapshot. So the tool runs a bounded rescan loop: scan, signal, repeat until a scan comes back empty (or MAX_PASSES is reached). Because the orchestrator scripts are themselves tagged (see below), they die on the first pass and stop respawning, and later passes just mop up orphaned stragglers.
• Graceful then forceful. The first pass sends SIGTERM to let processes shut down cleanly; every pass after that sends SIGKILL. If anything is still alive after the last pass (e.g. stuck in uninterruptible I/O), it prints a warning with an inspection command and exits non-zero.

It also supports --list (show what would be killed, kill nothing) and -9 (SIGKILL from the first pass).

agent_evals/Analysis/eval_scripts/generate_ref.sh and run_repeatability_parallel.sh

Each entrypoint gains a small guard at the top that sets the marker and re-execs itself once:

if [[ "${TRACELENS_EVAL_JOB:-}" != 1 ]]; then
    export TRACELENS_EVAL_JOB=1
    exec bash "$0" "$@"
fi

The re-exec is needed since a process's /proc/<pid>/environ is fixed when it starts. Exporting an environment variable in the script tags its children but not the script itself. Without the re-exec, the scan would find the agents but not the orchestrator scripts, and a kill would leave those scripts alive to respawn the agents.

Tests

Manual verification for both entrypoints:

• Launched generate_ref.sh and, separately, run_repeatability_parallel.sh, let each spin up its agent tree, and confirmed via /proc/<pid>/environ that the orchestrator script process itself, its worker subshells, the timeout/agent launchers, the node worker, and tool-call subprocesses were all tagged.
• Ran kill_eval_jobs.sh --list and confirmed it enumerated the full tree (script + subshells + agent tree).
• Ran kill_eval_jobs.sh and independently verified afterward (a fresh /proc scan plus pgrep) that zero marked processes remained and nothing respawned.
• Confirmed the precision guard: an unmarked, unrelated agent/python process is left untouched, and a marked but generically-named process (e.g. a plain sleep) is still found and killed — demonstrating the match is by environment marker, not command name.
• Confirmed the frozen-environ behavior directly: a runtime-only export does not appear in the exporting process's own /proc/environ, while the re-exec form does — validating why the re-exec is necessary.