Add Copilot runtime connection support and update documentation

AGENTS.md

@@ -111,7 +111,7 @@ step-by-step checklist.
 
 - **providers/**: SDK provider abstraction
   - `base.py` - `AgentProvider` ABC defining `execute()`, `validate_connection()`, `close()`
-  - `copilot.py` - GitHub Copilot SDK implementation
+  - `copilot.py` - GitHub Copilot SDK implementation. By default spawns a nested `copilot` runtime via `CopilotClient()` (in `_build_client`, called from `_ensure_client_started`). When a runtime connection is resolved (`runtime.provider.runtime_url` or `COPILOT_PROVIDER_RUNTIME_URL`, optional `runtime_token` / `COPILOT_PROVIDER_RUNTIME_TOKEN`), it instead builds `CopilotClient(connection=RuntimeConnection.for_uri(url, connection_token=token))` to connect to an already-running `copilot --server`; the SDK skips spawning for URI connections and its `stop()` leaves the externally-owned server running. `_resolve_runtime_connection()` reads YAML first, then the namespaced env var (env activates on its own — the zero-YAML path for external orchestrators). Mutually exclusive with custom endpoint routing.
   - `claude.py` - Anthropic Claude API implementation
   - `claude_agent_sdk.py` - Claude Agent SDK implementation (uses `claude-agent-sdk` package)
   - `factory.py` - Provider instantiation
@@ -154,6 +154,7 @@ step-by-step checklist.
 - **Periodic checkpoints** (`runtime.checkpoint`, issue #244): opt-in `CheckpointConfig` (`every_agent: bool`, `every_seconds: int|None`, `keep_last: int=5`; `is_enabled = every_agent or every_seconds is not None`). Off by default → failure-only behavior preserved. `WorkflowEngine._maybe_save_periodic_checkpoint()` is called once at the **top of `_execute_loop`** (single choke point), where prior outputs are committed and `_current_agent_name` is the step *about to run* — so a periodic checkpoint reuses failure-checkpoint `current_agent` semantics and resume continues forward with no special-casing. Gated via the `_periodic_checkpoints_active` property (**root engine only**, `_subworkflow_depth == 0`, + `is_enabled`) and skips the first iteration (`limits.current_iteration == 0`). The save decision is `_periodic_checkpoint_due(now)` (`every_agent` OR `every_seconds` throttle; first save always fires). `_save_checkpoint_on_failure` and the periodic path share `_write_checkpoint(error, trigger)` (which best-effort-guards provider `get_session_ids()` so it never raises). The periodic save wraps write+emit+rotate; on any failure it calls `_record_periodic_checkpoint_failure()` which emits a **`checkpoint_save_failed`** event (consecutive-failure count; surfaced by `ConsoleEventSubscriber` + JSONL + dashboard) so a recovery-reliant user isn't silently left without checkpoints. After a save the engine calls `rotate_periodic_checkpoints`; at a terminal **non-resumable** outcome (clean completion via `run()`/`resume()`, or an explicit `status: failed` terminate) `_cleanup_run_periodic_checkpoints()` deletes the run's periodic checkpoints (an unexpected failure leaves them in place alongside the failure checkpoint). `conductor checkpoints` shows a `Trigger` column and `—` for periodic rows' error type. See `examples/periodic-checkpoints.yaml` and `docs/workflow-syntax.md` (Periodic Checkpoints section).
 - **Terminate steps** (`type: terminate`): explicit terminal step with `status` (`success` | `failed`), Jinja2 `reason`, and optional `output_template` (a `dict[str, str]` that replaces `workflow.output:` when set; each value is rendered then passed through `_maybe_parse_json` so `"true"` becomes `True`, `"42"` becomes `42`, JSON literals are parsed). Reaching a terminate step ends the workflow immediately (no routes evaluated after). `success` → CLI exit 0, dashboard ✅, `workflow_completed { termination_reason, terminated_by, is_explicit: true, status }`; runs `on_complete` hook. `failed` → CLI exit 1 (with rendered output JSON still printed to stdout for downstream tooling), dashboard ❌, raises `WorkflowTerminated` (subclass of `ExecutionError`), emits `workflow_failed { error_type: "WorkflowTerminated", is_explicit: true, status, output }`, runs `on_error` hook, and **does not** save an on-failure checkpoint (explicit terminations are intentionally non-resumable). Terminate steps cannot have `routes`, `tools`, `output`, `prompt`, `model`, etc.; cannot be used as parallel-group members or as a for_each inline agent (route to one from those groups' `routes:` instead). Inside a sub-workflow, a `status: failed` terminate is downgraded at the parent boundary to `SubworkflowTerminatedError` (also a subclass of `ExecutionError`) preserving the child's rendered `terminated_output` / `terminated_reason` / `terminated_by` as structured attributes — the parent treats it as a normal sub-workflow failure (its own `workflow_failed` does NOT inherit `is_explicit: true`). For more detail see `examples/terminate.yaml`, `docs/workflow-syntax.md` (Terminate Steps section), and `plugins/conductor/skills/conductor/references/authoring.md`.
 - **Structured `runtime.provider` (Copilot custom routing)**: `runtime.provider` accepts either the bare string shorthand (`provider: copilot`) or a structured `ProviderSettings` object that routes the Copilot SDK at OpenAI-compatible / Azure / Anthropic endpoints (Ollama, vLLM, LM Studio, Azure OpenAI, etc.). Object fields: `name` (defaults to `copilot`), `type` (`openai`|`azure`|`anthropic`), `wire_api` (`completions`|`responses`), `base_url`, `api_key`, `bearer_token`, `headers`, `azure.api_version`. `api_key` and `bearer_token` are `SecretStr` (redacted in `model_dump` / dashboard / event logs). The model is frozen after construction. Custom routing activates only when at least one non-`name` field is set in YAML — ambient `OPENAI_*` env vars never divert default routing on their own. Once activated, missing fields fall back from env vars in this order: `base_url` ← `COPILOT_PROVIDER_BASE_URL` → `OPENAI_BASE_URL`; `api_key` ← `COPILOT_PROVIDER_API_KEY` (only — ambient `OPENAI_API_KEY` is intentionally NOT a fallback to avoid credential leaks); `bearer_token` ← `COPILOT_PROVIDER_BEARER_TOKEN`. The schema rejects every non-`name` field when `name != "copilot"` (structured config for other providers is a follow-up). It also rejects anchorless / broken combinations that would silently no-op at the SDK boundary: `wire_api` / `type` / `headers` / `azure` cannot stand alone without `base_url` / `api_key` / `bearer_token`; empty `headers`, empty `SecretStr`, and `azure: {api_version: null}` are rejected. The resolver raises `ProviderError` when custom routing is activated but every resolved field is falsy (e.g. expected env vars all unset). Custom routing applies to both agent execution and dialog turns so all sessions hit the same endpoint. `--provider <name>` CLI override replaces the whole `ProviderSettings` (logs a notice when YAML had structured fields). See `examples/copilot-local-llm.yaml`.
+- **Connect to an existing Copilot runtime (Copilot)**: `runtime.provider.runtime_url` (Copilot-only) points the provider at an already-running `copilot --server` runtime instead of spawning a nested one — every agent/model reuses the server's single authenticated session. Optional `runtime_token` (`SecretStr`, redacted, requires `runtime_url`) is the connection secret. Both fields fall back to env vars (`COPILOT_PROVIDER_RUNTIME_URL` / `COPILOT_PROVIDER_RUNTIME_TOKEN`) which activate the connection on their own (zero-YAML path for external orchestrators). `has_external_runtime()` is a separate axis from `has_custom_routing()` (so it does NOT trigger the endpoint-provider resolver), but both keep `ProviderSettings` from collapsing to the bare-string serialization. Schema rejects: `runtime_url` combined with any custom-routing field (`base_url`/`api_key`/`bearer_token`/`type`/`wire_api`/`headers`/`azure`) — mutually exclusive since the runtime is the endpoint; `runtime_token` without `runtime_url`; empty `runtime_token`; either field when `name != "copilot"`. Provider layer: `_resolve_runtime_connection()` (YAML then env) and `_build_client()` (in `copilot.py`). See `examples/copilot-existing-runtime.yaml` and `docs/configuration.md` (Connecting to an Existing Copilot Runtime).
 - **Validator block** (`validator:` on a provider-backed agent, issue #220): semantic output validation with retry-once. After the primary agent completes, `WorkflowEngine._apply_validator` runs `OutputValidator` (`engine/validator.py`) — a second LLM call (synthetic agent via `provider.execute`, `tools=[]`, `{passed, issues}` output schema) grading the output against `validator.criteria`. Fields: `criteria` (required, non-empty), `model` (defaults to the agent's model), `max_retries` (`Field(1, ge=0, le=1)` — hard-capped at 1; `0` = report-only). On `passed: false` and `max_retries > 0`, the primary re-runs **once** via `executor.execute` with a `## Validation feedback` section (the issues) appended to `guidance_section`; the second output is final (no second validation loop). **Fail-open**: validator errors / unparseable responses → treated as pass with a logged warning. Wired into the main loop, parallel groups, and for-each loops (guarded by `agent.validator and not output.partial`; for-each passes a `usage_label` so the row matches `<group>[<key>]`). Emits `agent_validator_start` / `agent_validator_complete { passed, issues, errored, tokens, cost_usd }` / `agent_validation_failed { issues, will_retry }` through the per-agent `event_callback` (so for-each events carry `item_key`). Cost: the validation call and any discarded first attempt are recorded as a separate `"<agent> (validator)"` usage row (primary row = effective output). Rejected on `script` / `human_gate` / `workflow` / `wait` / `set` / `terminate` types. Frontend: event types in `web/frontend/src/types/events.ts`, store handlers + `NodeData.validator_*` fields in `web/frontend/src/stores/workflow-store.ts`, detail UI in `web/frontend/src/components/detail/ValidatorDetail.tsx` (rebuild with `make build-frontend`). See `examples/validator.yaml` and `docs/workflow-syntax.md` (Validator section).
 
 ### Debugging `--web-bg` failures

CHANGELOG.md

@@ -9,6 +9,17 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Added
 
+- **Connect the Copilot provider to an existing runtime** — `runtime.provider`
+  gains a Copilot-only `runtime_url` (plus optional `runtime_token`) that points
+  Conductor at an already-running `copilot --server` runtime instead of spawning
+  its own nested one; every agent/model then reuses the server's single
+  authenticated session. Both fields also resolve from the namespaced
+  `COPILOT_PROVIDER_RUNTIME_URL` / `COPILOT_PROVIDER_RUNTIME_TOKEN` environment
+  variables, which activate the connection with no YAML — the zero-config path
+  for external orchestrators that already own an authenticated
+  Copilot process. Mutually exclusive with custom endpoint routing. See
+  `examples/copilot-existing-runtime.yaml` and the "Connecting to an Existing
+  Copilot Runtime" section of `docs/configuration.md`.
 - **Provider-supplied model pricing** — cost reporting now resolves pricing via a
   new `AgentProvider.get_model_pricing` hook before falling back to the static
   table. Resolution order is workflow `cost.pricing` → provider hook → built-in

docs/configuration.md

@@ -233,6 +233,83 @@ worrying about per-call drift.
 demonstrates the full pattern with both Ollama (active) and Azure
 OpenAI (commented variant).
 
+### Connecting to an Existing Copilot Runtime
+
+By default the Copilot provider spawns its own nested `copilot` runtime
+process (one per provider instance) to run agents. Instead, you can point
+Conductor at an **already-running** Copilot runtime that was started in
+server mode by some other process. Conductor then connects to that runtime
+and reuses its single authenticated session for every agent — no nested
+`copilot` process is spawned.
+
+This is the recommended way to run Conductor inside an external
+orchestrator that already owns an authenticated Copilot process. For
+example, an orchestrator can launch one authenticated
+`copilot --server` and hand Conductor a connection handle, so all of
+Conductor's agents/models are just new sessions on that shared server.
+Authentication is handled once, at the server — Conductor never needs the
+runtime's credentials.
+
+#### YAML
+
+```yaml
+workflow:
+  runtime:
+    provider:
+      name: copilot
+      runtime_url: localhost:3000          # "port", "host:port", or a full URL
+      runtime_token: ${COPILOT_RUNTIME_TOKEN}   # optional shared secret
+    default_model: gpt-4o
+```
+
+- `runtime_url` — where the running runtime is listening. Accepts a bare
+  `"port"`, `"host:port"`, or a full URL.
+- `runtime_token` — the shared secret the runtime was started with, if
+  any. Stored as a `SecretStr` (redacted in events/checkpoints/dashboard);
+  prefer `${VAR}` interpolation. Requires `runtime_url`.
+
+#### Environment variables (zero-YAML)
+
+The connection also activates from environment variables alone, so an
+orchestrator can enable it without editing the workflow YAML:
+
+| Field | Env var |
+|---|---|
+| `runtime_url` | `COPILOT_PROVIDER_RUNTIME_URL` |
+| `runtime_token` | `COPILOT_PROVIDER_RUNTIME_TOKEN` |
+
+The YAML value takes precedence; the environment variable is used as a
+fallback when the YAML field is unset. These variables are namespaced under `COPILOT_PROVIDER_*` on
+purpose: unlike ambient `OPENAI_*` variables, they can safely activate the
+connection because they are specific to this feature.
+
+```bash
+# Orchestrator side, conceptually:
+copilot --server --port 3000 &            # one authenticated runtime
+export COPILOT_PROVIDER_RUNTIME_URL=localhost:3000
+export COPILOT_PROVIDER_RUNTIME_TOKEN=<connection-token>
+conductor run review.yaml                 # connects; spawns no nested runtime
+```
+
+#### Rules and notes
+
+- `runtime_url` is **mutually exclusive** with custom endpoint routing
+  (`base_url` / `api_key` / `bearer_token` / `type` / `wire_api` /
+  `headers` / `azure`) — the external runtime *is* the endpoint. The schema
+  rejects the combination at config load.
+- `runtime_token` requires `runtime_url` (a token with nowhere to connect
+  is a misconfiguration) and, like the other secrets, may not be empty.
+- Closing the provider does **not** terminate the external runtime — the
+  SDK only shuts down runtimes it spawned itself, so the orchestrator-owned
+  server keeps running.
+- Runtime-spawn-only options (custom CLI path, injected env, etc.) do not
+  apply when connecting to an existing runtime.
+
+#### Example workflow
+
+[`examples/copilot-existing-runtime.yaml`](../examples/copilot-existing-runtime.yaml)
+demonstrates connecting to an already-running Copilot runtime.
+
 ## Common Configuration Options
 
 These options work with both providers:

examples/copilot-existing-runtime.yaml

@@ -0,0 +1,83 @@
+# Connect the Copilot provider to an already-running Copilot runtime
+#
+# By default conductor's Copilot provider spawns its own nested `copilot`
+# runtime process to execute agents. This example instead points conductor
+# at an EXISTING runtime that some other process started in server mode
+# (e.g. `copilot --server --port 3000`). Conductor connects to that runtime
+# via the SDK's `RuntimeConnection.for_uri(...)` and reuses its single
+# authenticated session for every agent — no nested `copilot` process is
+# spawned, and authentication is handled once, at the server.
+#
+# This is the recommended pattern for running conductor inside an external
+# orchestrator that already owns an authenticated Copilot process (for
+# example, an orchestrator launches one authenticated `copilot --server`
+# and hands conductor a connection handle).
+#
+# Two ways to configure the connection:
+#
+# 1. YAML (below): set `runtime_url` (and optional `runtime_token`) under
+#    `runtime.provider`.
+#
+# 2. Environment variables (no YAML needed — ideal for orchestrators):
+#
+#      export COPILOT_PROVIDER_RUNTIME_URL=localhost:3000
+#      export COPILOT_PROVIDER_RUNTIME_TOKEN=<connection-token>
+#      conductor run examples/copilot-existing-runtime.yaml \
+#        --input question="What is Python?"
+#
+#    The YAML value takes precedence; the env var is a fallback.
+#
+# Notes:
+# - `runtime_url` accepts a bare "port", "host:port", or a full URL.
+# - `runtime_token` is the shared secret the runtime was started with (if
+#   any). It is stored as a SecretStr and redacted from events/checkpoints;
+#   prefer ${VAR} interpolation so the literal never lands in YAML.
+# - `runtime_url` is mutually exclusive with custom endpoint routing
+#   (base_url / api_key / type / ...): the external runtime IS the endpoint.
+# - Closing conductor does NOT terminate the external runtime — the SDK only
+#   shuts down runtimes it spawned itself.
+
+workflow:
+  name: copilot-existing-runtime
+  description: Connect the Copilot provider to an already-running runtime server
+  version: "1.0.0"
+  entry_point: answerer
+
+  runtime:
+    provider:
+      name: copilot
+      # Where the already-running `copilot --server` is listening.
+      runtime_url: ${COPILOT_PROVIDER_RUNTIME_URL:-localhost:3000}
+    # Optional shared secret the server was started with. Uncomment and set
+    # $COPILOT_RUNTIME_TOKEN if your server requires a connection token
+    # (the SDK rejects an empty token, so keep this commented when unused):
+    #   runtime_token: ${COPILOT_RUNTIME_TOKEN}
+    # Alternatively, set COPILOT_PROVIDER_RUNTIME_TOKEN in the environment.
+
+    default_model: gpt-4o
+
+  input:
+    question:
+      type: string
+      required: true
+      description: The question to answer
+
+agents:
+  - name: answerer
+    description: Answers the user's question via the shared runtime
+    prompt: |
+      You are a helpful assistant. Please answer the following question
+      clearly and concisely:
+
+      Question: {{ workflow.input.question }}
+
+      Provide a direct answer without unnecessary preamble.
+    output:
+      answer:
+        type: string
+        description: The answer to the question
+    routes:
+      - to: $end
+
+output:
+  answer: "{{ answerer.output.answer }}"