feat(server): cancel a running job by id

CLAUDE.md

@@ -85,8 +85,8 @@ Each variant carries a lazy `getValue: Task[T]` so callers can branch on `isUnch
 ### Backend Server
 
 `CcasServer` (`ccas.server`) is a zio-http server that exposes REST endpoints and runs background jobs:
-- **Routes:** `HealthRoutes` (health/readiness), `JobRoutes` (submit and query jobs, plus recruitment-result delivery: `GET /api/jobs/{jobId}/recruitment/invited` for a completed run's paste-ready invited usernames, `GET .../recruitment/found` for still-`Deferred` candidates, the mutating `POST .../recruitment/confirm` which flips that run's `Deferred` candidates to `Invited` in one transaction and returns `ConfirmResult{marked, usernames}`, and `GET /api/recruitment/clubs/{slug}/latest/invited` + `GET /api/recruitment/runs/{runId}/invited` for reporting a past run — these drive the `ccas recruit` `--stdout` / interactive-confirm / `--report` delivery modes, where an interactive scout sends `autoConfirm=false` so candidates stay `Deferred` until the operator confirms), `ScheduleRoutes` (CRUD for scheduled jobs), `BlacklistRoutes` (synchronous CRUD for `RecruitmentBlacklist` entries — delegates to `BlacklistApp` directly without going through `JobRunner`), `RecruitmentCriteriaRoutes` (synchronous `POST /api/recruitment-criteria` to set criteria, `GET .../{slug}/{alias}` to show, `GET .../{slug}` to list aliases — delegates to `RecruitmentCriteriaApp`). Route handlers use inline JSON codecs for request/response types. User-facing errors (`BadRequestException`, `NotFoundException`, `ConflictException`) form a sealed `UserFacingError` trait (`ccas.utils.errors`) that carries the HTTP status and a pre-encoded JSON body; `RouteHelpers.withErrorHandling` renders them uniformly (default body shape is `{"error": "<message>"}`, but `.of[B: JsonEncoder](body, msg)` companion constructors support structured payloads). Escaping `HttpStatusException`s from the Chess.com client render as 502. Anything else — including defects — collapses to a generic 500 with the full cause logged via `ZIO.logErrorCause` so operational visibility isn't lost; pure-interrupt causes are re-propagated so shutdown / client-disconnect noise stays out of the error log.
-- **JobRunner:** Trait-based (`JobRunnerLive`) async executor that forks fibers per job, tracks state in `JobRun` table (ULID IDs via `ulid-creator`). Marks orphaned running jobs as failed on startup. The `submit` method takes an `Option[JobRunId] => RIO[..., Any]` effect function, passing the generated job run ID so analysis apps can link their run records back via `job_run_id`. Job kinds: `Recruitment`, `Membership`, `MatchRef`, `History`, `Stats`, `ClubData` (the latter has no HTTP route — scheduler only).
+- **Routes:** `HealthRoutes` (health/readiness), `JobRoutes` (submit, query, and cancel jobs — `POST /api/jobs/{jobId}/cancel` interrupts a running job's fiber in this process (200 `CancelResult`, or 404 when no live job fiber exists here: unknown id, already-terminal, or owned by another instance), plus recruitment-result delivery: `GET /api/jobs/{jobId}/recruitment/invited` for a completed run's paste-ready invited usernames, `GET .../recruitment/found` for still-`Deferred` candidates, the mutating `POST .../recruitment/confirm` which flips that run's `Deferred` candidates to `Invited` in one transaction and returns `ConfirmResult{marked, usernames}`, and `GET /api/recruitment/clubs/{slug}/latest/invited` + `GET /api/recruitment/runs/{runId}/invited` for reporting a past run — these drive the `ccas recruit` `--stdout` / interactive-confirm / `--report` delivery modes, where an interactive scout sends `autoConfirm=false` so candidates stay `Deferred` until the operator confirms), `ScheduleRoutes` (CRUD for scheduled jobs), `BlacklistRoutes` (synchronous CRUD for `RecruitmentBlacklist` entries — delegates to `BlacklistApp` directly without going through `JobRunner`), `RecruitmentCriteriaRoutes` (synchronous `POST /api/recruitment-criteria` to set criteria, `GET .../{slug}/{alias}` to show, `GET .../{slug}` to list aliases — delegates to `RecruitmentCriteriaApp`). Route handlers use inline JSON codecs for request/response types. User-facing errors (`BadRequestException`, `NotFoundException`, `ConflictException`) form a sealed `UserFacingError` trait (`ccas.utils.errors`) that carries the HTTP status and a pre-encoded JSON body; `RouteHelpers.withErrorHandling` renders them uniformly (default body shape is `{"error": "<message>"}`, but `.of[B: JsonEncoder](body, msg)` companion constructors support structured payloads). Escaping `HttpStatusException`s from the Chess.com client render as 502. Anything else — including defects — collapses to a generic 500 with the full cause logged via `ZIO.logErrorCause` so operational visibility isn't lost; pure-interrupt causes are re-propagated so shutdown / client-disconnect noise stays out of the error log.
+- **JobRunner:** Trait-based (`JobRunnerLive`) async executor that forks fibers per job, tracks state in `JobRun` table (ULID IDs via `ulid-creator`). Marks orphaned running jobs as failed on startup. The `submit` method takes an `Option[JobRunId] => RIO[..., Any]` effect function, passing the generated job run ID so analysis apps can link their run records back via `job_run_id`. Job kinds: `Recruitment`, `Membership`, `MatchRef`, `History`, `Stats`, `ClubData` (the latter has no HTTP route — scheduler only). `JobRunStatus` is `Running | Completed | Failed | Cancelled`. `cancel(id)` interrupts the job's forked fiber (handles retained per-process in a `runningFibers` map registered before the fork so de-register can't race the register): it registers the id in a `cancelRequested` set then `interruptFork`s the fiber, and the job's own `.onInterrupt` records `Cancelled` **only if** its id is in that set. This gates operator cancellation apart from the `layerScope` interrupt fired at every in-flight job on server **shutdown** — those are left `Running` so the next boot's `markOrphansAsFailed` records them as `Failed`/"Service restarted" (which also still covers hard crashes / SIGKILL, where finalizers never run). Cancellation is best-effort/asynchronous — an in-flight uninterruptible blocking JDBC statement runs to completion before the interrupt lands (`markCancelled`'s `WHERE status = Running` guard makes the terminal write a no-op if the job reached Completed/Failed first). All `completed_at` writers (`updateStatus`, `markCancelled`, `markOrphansAsFailed`) take an `Instant` from the caller's `Clock.instant` rather than SQL `NOW()`, so every job timestamp comes from the one testable app clock, coherent with `started_at`. Being per-process, `cancel` inherits the single-server-per-DB assumption (#110/#111). The CLI surface is `ccas cancel <job-id>`; a job cancelled out from under an active `ccas logs` follow renders "was cancelled" and exits non-zero.
 - **JobScheduler:** Polling daemon (configurable interval) that checks enabled `JobSchedule` entries and submits due jobs to the `JobRunner`.
 - **ServerTables:** Ensures both analysis and server database tables exist on startup.
 

README.md

@@ -57,6 +57,7 @@ sbt "runMain ccas.cli.Main server up"     # boots CcasServer on 127.0.0.1:8080
 # Terminal 2 — commands. No environment needed; they just call the server.
 ccas use-club team-alpha          # set the current club once…
 ccas jobs
+ccas cancel <job-id>              # request cancellation of a running job
 ccas membership                   # …then commands target it with no slug
 ccas membership --club team-beta  # override for one command
 ccas membership --all             # every managed club
@@ -66,7 +67,9 @@ ccas --help              # full command tree
 ccas <command> --help    # per-command flags
 ```
 
-Commands: `server {up|down|status}`, `use`, `membership`, `history`, `recruit`, `stats`, `jobs`, `logs`, `blacklist {add|list|remove}`, `schedule {list|add|remove}`, `club {add|remove|list}`.
+Commands: `server {up|down|status}`, `use`, `membership`, `history`, `recruit`, `stats`, `jobs`, `logs`, `cancel`, `blacklist {add|list|remove}`, `schedule {list|add|remove}`, `club {add|remove|list}`.
+
+**Cancelling a job.** `ccas cancel <job-id>` requests cancellation of a running job — it interrupts the job's fiber on the server, which records the run as `Cancelled`. Cancellation is best-effort: an in-flight blocking database statement runs to completion first, so the job stops at its next interruptible point rather than instantly. A cancelled recruitment run leaves the candidates it found so far as **deferred** (nothing invited), so you can review and confirm them afterwards. Cancel is single-server-scoped — it reaches jobs running on the server it is sent to.
 
 **Club targeting.** Slug-requiring commands take the club via `--club <slug>` rather than a positional argument; `membership`/`history` accept a comma-separated `--club a,b` or `--all` (every managed club) — but not both at once (`--all` with `--club` is rejected as a likely mistake). When neither is given, the command falls back to the **current club** set with `ccas use-club <slug>` (stored as `current_club` in the [config file](#cli-config-file)); an explicit `--club`/`--all` always wins. `ccas use-club` is a local config write — no server call — so it works offline and only warns (does not reject) if the slug isn't among the cached clubs.
 
@@ -92,7 +95,7 @@ The detached server writes its pid to `${XDG_STATE_HOME:-~/.local/state}/ccas/cc
 
 ### Shell completion
 
-`ccas completion <bash|zsh|fish>` prints a self-contained, **pure-shell** completion script. It boots the JVM once when you install it; afterwards every `<TAB>` runs entirely in the shell (no JVM, no network), so completion is instant. The scripts complete subcommands and flags, plus **club slugs** (after `--club`, and for the `ccas use-club` argument) and **recent job ids** (for `ccas logs`) read from cache files.
+`ccas completion <bash|zsh|fish>` prints a self-contained, **pure-shell** completion script. It boots the JVM once when you install it; afterwards every `<TAB>` runs entirely in the shell (no JVM, no network), so completion is instant. The scripts complete subcommands and flags, plus **club slugs** (after `--club`, and for the `ccas use-club` argument) and **recent job ids** (for `ccas logs` and `ccas cancel`) read from cache files.
 
 ```bash
 # bash — install once (bash-completion auto-loads it)

completions/ccas.bash

@@ -31,7 +31,7 @@ _ccas() {
     words=("${COMP_WORDS[@]}")
   fi
 
-  local top="server use-club membership history recruit stats jobs logs blacklist schedule club config completion"
+  local top="server use-club membership history recruit stats jobs logs cancel blacklist schedule club config completion"
   local global="--help --version"
   local envkeys="CCAS_CONTACT_EMAIL DATABASE_URL DB_HOST DB_PORT DB_NAME DB_USER DB_PASSWORD DB_SCHEMA SERVER_PORT SERVER_HOST JOB_LOGS_DIR SCHEDULER_POLL_MINUTES SCHEDULER_MATCHREF_INTERVAL_HOURS SCHEDULER_MATCHREF_ENABLED SCHEDULER_CLUBDATA_INTERVAL_HOURS SCHEDULER_CLUBDATA_ENABLED SCHEDULER_HISTORY_INTERVAL_HOURS SCHEDULER_HISTORY_ENABLED SCHEDULER_MEMBERSHIP_INTERVAL_HOURS SCHEDULER_MEMBERSHIP_ENABLED DB_POOL_MAX DB_POOL_MIN_IDLE DB_POOL_CONNECTION_TIMEOUT DB_POOL_IDLE_TIMEOUT DB_POOL_MAX_LIFETIME DB_POOL_KEEPALIVE_TIME DB_POOL_CONNECTION_TEST_QUERY DB_POOL_INIT_FAIL_TIMEOUT DB_SOCKET_TIMEOUT_SECONDS DB_CONNECT_TIMEOUT_SECONDS DB_TCP_KEEP_ALIVE DB_RETRY_BASE_DELAY_MS DB_RETRY_MAX_RETRIES CHESS_COM_API_RECOVERY_TIERS CHESS_COM_API_COOLDOWN_SECONDS CHESS_COM_API_CF_COOLDOWN_SECONDS CHESS_COM_API_FAILURE_WINDOW_SIZE CHESS_COM_API_FAILURE_THRESHOLD CHESS_COM_API_MIN_SAMPLE_SIZE CHESS_COM_API_MIN_REQUEST_DELAY_MS CHESS_COM_API_EMA_TAU_MS CHESS_COM_API_MIN_TIER_OBSERVATION_SECONDS CHESS_COM_API_RETRY_BASE_SECONDS CHESS_COM_API_CF_RETRY_DELAY_SECONDS CHESS_COM_API_CONNECTION_RETRY_BASE_SECONDS CHESS_COM_API_MAX_429_RETRIES CHESS_COM_API_MAX_CF_RETRIES CHESS_COM_API_MAX_CONNECTION_RETRIES CHESS_COM_API_STATS_FLUSH_INTERVAL_SECONDS"
 
@@ -68,6 +68,7 @@ _ccas() {
     stats) opts="--server --since --until --club --no-progress" ;;
     jobs) opts="--server --limit" ;;
     logs) opts="--server --no-progress"; pos="jobid" ;;
+    cancel) opts="--server"; pos="jobid" ;;
     blacklist)
       case "$sub" in
         "") COMPREPLY=( $(compgen -W "add list remove --help" -- "$cur") ); return ;;

src/main/scala/ccas/cli/CliCommand.scala

@@ -89,6 +89,7 @@ object CliCommand {
   ) extends ServerCommand
   final case class Jobs(server: String, limit: Option[Int]) extends ServerCommand
   final case class Logs(server: String, jobId: String, noProgress: Boolean) extends ServerCommand
+  final case class Cancel(server: String, jobId: String) extends ServerCommand
   final case class BlacklistAdd(
     server: String,
     club: Option[String],
@@ -256,6 +257,11 @@ object CliCommand {
       .withHelp("Poll a job's status and logs until it finishes")
       .map { case ((server, noProgress), jobId) => Logs(server, jobId, noProgress) }
 
+  private def cancel(default: String): Command[CliCommand] =
+    Command("cancel", serverOpt(default), Args.text("jobId") ?? "Job run id to cancel")
+      .withHelp("Request cancellation of a running job (the job stops as soon as it reaches an interruptible point)")
+      .map { case (server, jobId) => Cancel(server, jobId) }
+
   private def blacklistAdd(default: String): Command[CliCommand] =
     Command(
       "add",
@@ -415,6 +421,7 @@ object CliCommand {
       stats(defaultServer),
       jobs(defaultServer),
       logs(defaultServer),
+      cancel(defaultServer),
       blacklist(defaultServer),
       schedule(defaultServer),
       clubs(defaultServer),

src/main/scala/ccas/cli/CompletionSpec.scala

@@ -77,6 +77,7 @@ object CompletionSpec {
     ),
     Leaf(List("jobs"), List(server, "--limit"), List(server, "--limit"), NoArgs),
     Leaf(List("logs"), List(server, "--no-progress"), List(server), JobId),
+    Leaf(List("cancel"), List(server), List(server), JobId),
     Leaf(
       List("blacklist", "add"),
       List(server, "--reason", "--months", clubFlag),
@@ -120,7 +121,7 @@ object CompletionSpec {
 
   /** Top-level subcommand names, in tree order (matches `CliCommand.command(...).subcommands`). */
   val topLevel: List[String] =
-    List("server", "use-club", "membership", "history", "recruit", "stats", "jobs", "logs",
+    List("server", "use-club", "membership", "history", "recruit", "stats", "jobs", "logs", "cancel",
       "blacklist", "schedule", "club", "config", "completion")
 
   /** Flags available on every command. */
@@ -154,6 +155,7 @@ object CompletionSpec {
     List("stats")               -> "Submit a club performance-stats job",
     List("jobs")                -> "List recent jobs and their status",
     List("logs")                -> "Poll a job's status and logs until it finishes",
+    List("cancel")              -> "Request cancellation of a running job (the job stops as soon as it reaches an interruptible point)",
     List("completion")          -> "Emit a shell completion script (bash, zsh, or fish)",
     List("club", "add")         -> "Mark a club as one you manage with CCAS",
     List("club", "remove")      -> "Remove a club from the ones you manage",

src/main/scala/ccas/cli/Dispatcher.scala

@@ -11,6 +11,7 @@ import ccas.api.player.ApiPlayer
 import ccas.server.routes.BlacklistRoutes.{BlacklistEntryResponse, CreateBlacklistRequest}
 import ccas.server.routes.ManagedClubRoutes.{ManagedClubResponse, MarkManagedRequest}
 import ccas.server.routes.JobRoutes.{
+  CancelResult,
   ClubJobResult,
   ConfirmResult,
   HistoryRequest,
@@ -155,6 +156,12 @@ object Dispatcher {
     case CliCommand.Logs(_, jobId, _) =>
       follower.followJob(jobId)
 
+    // A 404 (no running job to cancel) surfaces via the client's decode path as a CliError, rendered by `dispatch`'s
+    // catchAll as "error: …" with exit 1 — so success here means the interrupt was dispatched.
+    case CliCommand.Cancel(_, jobId) =>
+      api.postEmpty[CancelResult](s"/api/jobs/$jobId/cancel")
+        .flatMap(_ => Console.printLine(s"cancellation requested for job $jobId").orDie.as(0))
+
     case CliCommand.BlacklistAdd(_, club, usernames, reason, months) =>
       resolveClub(club, currentClub).flatMap(slug =>
         api.postUnit[CreateBlacklistRequest](

src/main/scala/ccas/cli/JobFollower.scala

@@ -175,6 +175,9 @@ final class JobFollower(
       job.status match {
         case "Completed" => ZIO.succeed(0)
         case "Failed"    => Console.printLineError(s"$jobId failed: ${job.error.getOrElse("unknown error")}").orDie.as(1)
+        // A followed job cancelled out from under us (e.g. `ccas cancel` from another terminal): a clean terminal
+        // outcome, not an "unexpected status". Non-zero — the job did not complete.
+        case "Cancelled" => Console.printLineError(s"$jobId was cancelled").orDie.as(1)
         case other       => Console.printLineError(s"$jobId: unexpected status '$other'").orDie.as(1)
       }
     }

src/main/scala/ccas/server/jobs/JobRun.scala

@@ -95,11 +95,25 @@ object JobRun {
              WHERE id = $id""".update.run()
     }
 
-  def markOrphansAsFailed: ZIO[PostgresClient, SQLException, Int] =
+  // `completedAt` is supplied by the caller (from `Clock.instant`) rather than SQL `NOW()` so every job timestamp comes
+  // from the one testable app clock — consistent with `startedAt` / `updateStatus`, and skew-free against the DB host.
+  def markOrphansAsFailed(completedAt: Instant): ZIO[PostgresClient, SQLException, Int] =
     connectZIO {
       val failed  = JobRunStatus.Failed
       val running = JobRunStatus.Running
-      sql"""UPDATE job_run SET status = $failed, completed_at = NOW(), error = 'Service restarted'
+      sql"""UPDATE job_run SET status = $failed, completed_at = $completedAt, error = 'Service restarted'
             WHERE status = $running""".update.run()
     }
+
+  // Flip a job to Cancelled — the terminal write when an operator interrupts a running job (see JobRunner.cancel). The
+  // `WHERE status = Running` guard makes it a no-op if the job already reached Completed/Failed between the interrupt
+  // request and this write, so a cancel can never clobber a genuine terminal outcome. Returns rows affected (0 or 1).
+  // `completedAt` comes from the caller's `Clock.instant` for the same clock-consistency reason as `markOrphansAsFailed`.
+  def markCancelled(id: JobRunId, completedAt: Instant): ZIO[PostgresClient, SQLException, Int] =
+    connectZIO {
+      val cancelled = JobRunStatus.Cancelled
+      val running   = JobRunStatus.Running
+      sql"""UPDATE job_run SET status = $cancelled, completed_at = $completedAt, error = 'Cancelled by operator'
+            WHERE id = $id AND status = $running""".update.run()
+    }
 }

src/main/scala/ccas/server/jobs/JobRunStatus.scala

@@ -3,7 +3,7 @@ package ccas.server.jobs
 import ccas.utils.sql.EnumSql
 
 enum JobRunStatus {
-  case Running, Completed, Failed
+  case Running, Completed, Failed, Cancelled
 }
 
 object JobRunStatus extends EnumSql[JobRunStatus]