Add robust local Harbor runtime and CLI interop

docs/v6/advanced/harbor-convert.mdx

@@ -9,13 +9,27 @@ task dirs - is a *frontend* that loads into the same primitives (`Environment`,
 `Task`, `Taskset`). Integrations are **loaders, not converters**: no codegen
 roundtrip to run foreign tasks. The Harbor integration lives in the SDK repo at
 [`integrations/harbor.py`](https://github.com/hud-evals/hud-python/blob/main/integrations/harbor.py)
-- a recipe built only on the public SDK surface; copy it into your project or
-run it from a checkout.
+- a public-surface loader that maps Harbor folders into SDK primitives. The
+included `HarborRuntime` is maintained with the SDK for local Docker execution;
+copy the loader into your project or run it from a checkout.
 
 ## Prerequisites
 
 - A Harbor task directory - each task has `task.toml` + `instruction.md`, and
   usually an `environment/` (with a `Dockerfile`) and `tests/`.
+- For local execution, a running Docker Engine or Docker Desktop daemon that
+  your user can access, plus the Docker Compose v2 plugin (`docker compose`).
+
+Loading and exporting task metadata does not require Docker. `HarborRuntime`
+does: it builds and starts each task through the local Docker daemon.
+
+<Warning>
+**Run only trusted Harbor tasks.** `HarborRuntime` executes task-supplied
+Dockerfiles, Compose definitions, and `tests/test.sh` against your local Docker
+daemon. Review them first, especially bind mounts, devices, privileged settings,
+build inputs, and scripts; Docker task definitions are not a security boundary
+from the host running the daemon.
+</Warning>
 
 ## Load Harbor tasks
 
@@ -26,24 +40,75 @@ directly - one row per task dir (`id` = the dir name), sharing one declarative
 ```python
 from integrations.harbor import detect, load
 
-assert detect("./terminal-bench")
-taskset = load("./terminal-bench")
+assert detect("./harbor_tasks")
+taskset = load("./harbor_tasks")
 
 for task in taskset:
     print(task.env, task.id)
 ```
 
-Like every task row, the result carries no placement. Run it by supplying one -
-today that means a substrate already serving the control channel
-(`runtime=Runtime(url)`); a docker provider that builds and runs each task's
-`environment/` image is the planned follow-up:
+Like every task row, the result carries no placement. Run it by supplying one.
+For local Docker-backed Harbor execution, use `HarborRuntime`; it builds the
+task's `environment/` image, runs a fresh container, exposes the workspace
+through HUD's normal shell capability, and grades by running `tests/test.sh`:
 
 ```python
-from hud import Runtime
+from integrations.harbor import HarborRuntime
+
+job = await taskset.run(agent, runtime=HarborRuntime("./harbor_tasks"))
+```
+
+The eval CLI can run local Harbor task directories and datasets when you opt
+into the Harbor source format:
 
-job = await taskset.run(agent, runtime=Runtime("tcp://127.0.0.1:8765"))
+```bash
+hud eval ./harbor_tasks claude --format harbor --task-ids cancel-async-tasks --max-steps 30
 ```
 
+Each concurrent Harbor rollout starts its own Compose project and may build an
+image. Set `--max-concurrent` to a value your Docker host can sustain; start
+with `1` to `3` for unfamiliar tasks, then increase it after observing CPU,
+memory, disk, and build-cache pressure.
+
+### Supported Harbor subset
+
+`HarborRuntime` validates `task.toml` before starting Docker and fails closed on
+features it does not implement:
+
+| Area | Supported | Rejected |
+|------|-----------|----------|
+| Task shape | Linux, single-step tasks with `instruction.md` | Windows, `steps`, multi-step reward strategies, artifact collection |
+| Environment | `Dockerfile`, a prebuilt `docker_image`, or Compose with a `main` service and sidecars | GPU/TPU resources, task-config healthchecks, MCP servers, `skills_dir` |
+| Placement | Public/default networking, environment variables, users, absolute workdirs, CPU and memory limits | Restricted network policies, host allowlists, agent/verifier network overrides |
+| Verification | Shared-container `tests/test.sh`, verifier user/env, verifier timeout, `reward.json` or `reward.txt` | Separate verifier environments and verifier collect hooks |
+
+`[environment].storage_mb` is accepted for Harbor schema compatibility but is
+not currently enforced by the Docker/Compose runtime. `[agent].timeout_sec`
+bounds only the agent phase, after provisioning and task startup. An explicit
+`Taskset.run(..., rollout_timeout=...)` still imposes an earlier whole-rollout
+limit when configured.
+
+Verification uses Harbor's shared-container mode. Hidden tests are staged only
+after the agent phase, but the agent and verifier still share one container;
+this is parity with Harbor's shared mode, not a security boundary against a
+root agent that leaves background processes behind.
+
+### Choose the primary reward
+
+`reward.txt` supplies one numeric reward. A `reward.json` object may contain
+multiple numeric rewards; `HarborRuntime` selects `reward`, or the only key when
+the object has exactly one entry. If neither rule is unambiguous, select the
+primary metric explicitly:
+
+```python
+runtime = HarborRuntime("./harbor_tasks", reward_key="security")
+job = await taskset.run(agent, runtime=runtime)
+```
+
+The configured key must exist and contain a numeric reward. An invalid or
+ambiguous reward is reported as a grading error rather than silently choosing a
+metric.
+
 ## Export HUD tasks to Harbor
 
 `export(source, out_dir)` goes the other way: it turns a HUD task source (a

docs/v6/reference/cli.mdx

@@ -105,6 +105,7 @@ For a platform taskset, pass its name or id directly: `hud eval "My Tasks" claud
 | `--config`, `-c` | Agent config `key=value` (repeatable). |
 | `--verbose`, `-v` | Show agent logs (step progress, tool calls) for batch runs too. |
 | `--very-verbose`, `-vv` | Debug-level logs. |
+| `--format` | Task source format: `hud` (default) or `harbor`. |
 | `--runtime` | Placement: `local` (a child process per rollout, serving the env source — `SubprocessRuntime`), `hud` (HUD runtime tunnel), or `tcp://host:port`. Defaults to `local` for a tasks file; platform tasksets default to remote hosted execution. |
 | `--remote` | Run the whole rollout remotely on the HUD platform. |
 | `--yes`, `-y` | Skip confirmation prompt. |
@@ -133,7 +134,9 @@ hud sync env                   # sync environment metadata
 ```
 
 External benchmark formats (currently Harbor) load directly into the runtime
-as `Taskset`s - no conversion step. See [Harbor interop](/v6/advanced/harbor-convert).
+as `Taskset`s - no conversion step. For local Harbor directories, opt in with
+`--format harbor` so the CLI uses the Harbor loader and Docker-backed runtime
+provider. See [Harbor interop](/v6/advanced/harbor-convert).
 
 ## Inspect
 

docs/v6/reference/runtime.mdx

@@ -176,11 +176,15 @@ completion. It requires a gateway agent that can serialize its identity (Claude/
 ### `Runtime`
 
 ```python
-Runtime(url, params=..., config=...)
+Runtime(url, params=..., config=..., agent_timeout_s=None)
 ```
 
 - **`url`** - control-channel address of an already-running substrate (e.g. `tcp://host:8765`).
 - **`params`** - connection-time data a transport may need (auth token, sandbox id).
+- **`agent_timeout_s`** - optional positive agent-phase limit. Its clock starts after provisioning,
+  connection, and task startup. If `Task.run(..., rollout_timeout=...)` or
+  `Taskset.run(..., rollout_timeout=...)` supplies an earlier whole-rollout deadline, that earlier
+  deadline still wins.
 
 ## Run on your own infra
 

hud/cli/eval.py

@@ -63,6 +63,7 @@ def _resolve_model_from_catalog(model_id: str) -> tuple[AgentType, str] | None:
 
 _CONFIG_PATH = ".hud_eval.toml"
 _PLACEMENT_CONFLICT_ERROR = "--runtime and --remote are mutually exclusive placement options"
+_SOURCE_FORMATS = ("hud", "harbor")
 
 
 def _resolve_env_vars(obj: Any) -> Any:
@@ -167,6 +168,7 @@ class AgentPreset:
 # very_verbose = true
 # auto_respond = true
 # gateway = false  # Route LLM API calls through HUD Gateway
+# format = "hud"  # hud or harbor
 # runtime = "local"  # local, hud, or tcp://host:port
 # remote = false  # Run the whole rollout remotely on HUD
 
@@ -264,6 +266,7 @@ class EvalConfig(BaseModel):
         "group_size",
         "auto_respond",
         "gateway",
+        "format",
         "runtime",
         "remote",
     }
@@ -279,6 +282,9 @@ class EvalConfig(BaseModel):
     auto_respond: bool | None = None
     group_size: int = 1
     gateway: bool = False
+    #: Source format. ``None``/``hud`` means normal HUD task source loading;
+    #: ``harbor`` opts into the Harbor integration loader/runtime.
+    format: str | None = None
     #: Placement: "local" (spawn each row's env from the source), "hud"
     #: (HUD runtime tunnel), or a tcp:// url of an already-served env.
     #: ``None`` means "infer from the source": a local file runs locally, a
@@ -306,6 +312,20 @@ def _parse_agent_type(cls, v: Any) -> AgentType | None:
                 ) from None
         return v
 
+    @field_validator("format", mode="before")
+    @classmethod
+    def _parse_format(cls, v: Any) -> str | None:
+        if v is None:
+            return None
+        if not isinstance(v, str):
+            return v
+        normalized = v.strip().lower()
+        if normalized in ("", "hud"):
+            return None
+        if normalized in _SOURCE_FORMATS:
+            return normalized
+        raise ValueError(f"Invalid format: {v}. Must be one of: {', '.join(_SOURCE_FORMATS)}")
+
     def source_is_local_file(self) -> bool:
         """Whether ``source`` points at an on-disk taskset (vs. a platform slug/id)."""
         return self.source is not None and Path(self.source).exists()
@@ -319,6 +339,13 @@ def resolve_runtime(self) -> EvalConfig:
         ``--runtime`` is always honored, except ``local`` against a platform
         taskset, which has no env to spawn.
         """
+        if self.format == "harbor":
+            if not self.source_is_local_file():
+                hud_console.error("--format harbor requires a local Harbor task directory")
+                raise typer.Exit(1)
+            if self.remote or (self.runtime is not None and self.runtime != "local"):
+                hud_console.error("--format harbor currently supports only local runtime placement")
+                raise typer.Exit(1)
         if self.runtime is None:
             if self.source_is_local_file():
                 return self.model_copy(update={"runtime": "local"})
@@ -502,6 +529,7 @@ def merge_cli(
         gateway: bool = False,
         config: list[str] | None = None,
         task_ids: str | None = None,
+        format: str | None = None,
         runtime: str | None = None,
         remote: bool = False,
     ) -> EvalConfig:
@@ -517,6 +545,7 @@ def merge_cli(
                 "max_concurrent": max_concurrent,
                 "max_steps": max_steps,
                 "group_size": group_size,
+                "format": format,
                 "runtime": runtime,
             }.items()
             if value is not None
@@ -604,6 +633,8 @@ def display(self) -> None:
         table.add_column("Value", style="green")
 
         table.add_row("source", str(self.source or "-"))
+        if self.format:
+            table.add_row("format", self.format)
         table.add_row("runtime", str(self.runtime or "-"))
         table.add_row("agent", self.agent_type.value if self.agent_type else "-")
         if self.task_ids:
@@ -728,6 +759,28 @@ def _spawn_target(source: Path) -> Path:
     return resolved.parent
 
 
+def _load_local_taskset(source_path: Path, source_format: str | None) -> Any:
+    from hud.eval import Taskset
+
+    format_name = source_format or "hud"
+    if format_name == "hud":
+        taskset = Taskset.from_file(source_path)
+        if len(taskset) == 0:
+            from integrations.harbor import detect
+
+            if detect(source_path):
+                hud_console.hint(
+                    f"{source_path} looks like a Harbor task directory; "
+                    "rerun with --format harbor to load it."
+                )
+        return taskset
+    if format_name == "harbor":
+        from integrations.harbor import load
+
+        return load(source_path)
+    raise ValueError(f"unsupported task source format: {format_name}")
+
+
 def _resolve_placement(cfg: EvalConfig, source_path: Path | None) -> Any:
     """Map the config's ``runtime`` onto a placement for ``Taskset.run``.
 
@@ -744,6 +797,10 @@ def _resolve_placement(cfg: EvalConfig, source_path: Path | None) -> Any:
     if cfg.runtime == "local":
         if source_path is None:
             raise ValueError("local placement requires a local source path")
+        if cfg.format == "harbor":
+            from integrations.harbor import HarborRuntime
+
+            return HarborRuntime(source_path)
         return SubprocessRuntime(_spawn_target(source_path))
     if cfg.runtime == "hud":
         require_api_key("run HUD runtime tunnel evals")
@@ -774,7 +831,7 @@ async def _run_evaluation(cfg: EvalConfig) -> Any:
     if is_local:
         hud_console.info(f"Loading tasks from: {cfg.source}")
         try:
-            taskset = Taskset.from_file(source_path)
+            taskset = _load_local_taskset(source_path, cfg.format)
         except Exception as e:
             hud_console.error(f"Failed to load tasks from {cfg.source}: {e}")
             raise typer.Exit(1) from e
@@ -888,6 +945,11 @@ def eval_command(
     gateway: bool = typer.Option(
         False, "--gateway", "-g", help="Route LLM API calls through HUD Gateway"
     ),
+    format: str | None = typer.Option(
+        None,
+        "--format",
+        help="Task source format: hud (default) or harbor.",
+    ),
     runtime: str | None = typer.Option(
         None,
         "--runtime",
@@ -908,6 +970,7 @@ def eval_command(
         hud eval "My Tasks" claude-sonnet-4-6 --full   # Platform taskset, run on the platform
         hud eval tasks.json claude --config max_tokens=32768
         hud eval tasks.json claude --gateway           # Route LLM calls through HUD Gateway
+        hud eval ./harbor_tasks claude --format harbor # Run Harbor task dirs locally
         hud eval tasks.json claude-sonnet-4-6 --runtime hud  # Use HUD runtime tunnel
         hud eval tasks.json claude-sonnet-4-6 --remote       # Execute rollout remotely
     """
@@ -938,6 +1001,7 @@ def eval_command(
             group_size=group_size,
             config=config,
             gateway=gateway,
+            format=format,
             runtime=runtime,
             remote=remote,
         )