10x: clk kickoff subcommand, quality gates, and orchestration decomposition#103
Merged
Conversation
Port the body of kickoff.sh into a native `clk kickoff` subcommand
(clk_harness/kickoff.py):
- .env loading/exporting via the existing env_file helpers, with the
same source-order precedence as `set -a; . .env`.
- apply_config_env_overrides(): exact port of the script's first Python
heredoc (CLK_* env vars -> clk.config.json), preserving the key map
and the _bool/_csv/int/float cast semantics.
- activate_provider(): exact port of the second heredoc (provider
activation -> providers.json), including the cli/apikey auth_mode
logic, per-provider key wiring, and the `${VAR:-default}` wrapper
defaults the shell put on the invocation.
- Non-interactive pipeline calls the existing cmd_idea -> cmd_plan ->
cmd_run -> cmd_loop in-process; the TUI path calls cmd_tui.
- --setup wizard reusing config/env_file with per-step resume via
.clk/.setup-progress, delegating tool install/config to
scripts/install_tool.sh (single source of truth).
- Early-exit modes --restore / --list / --clean ported too, plus the
first-run nudge and the missing-config -> offer-setup flow.
cli.py grows a `kickoff` subparser mirroring the script's flags:
--setup --restore --list --clean --provider --max-iterations
--project-name --no-tui --tui --run-install.
tests/test_kickoff_cmd.py asserts golden-file parity: the goldens under
tests/golden/kickoff/ were generated by running the ORIGINAL kickoff.sh
heredocs against snapshot base configs for a matrix of CLK_* inputs
(bool/csv edge cases, invalid casts, every auth_mode/provider branch);
the port must produce byte-identical JSON. Flag parsing, validation,
and the --list/--clean/--restore contracts are covered as well.
kickoff.sh itself is unchanged in this commit; it becomes a thin
wrapper over `clk kickoff` in the follow-up.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012jUkqbwroQmYRAdFDsp5r9
Replace kickoff.sh's 1100-line body with a thin wrapper: resolve a Python interpreter (project-local .clk/venv first, then python3 / python), put the adjacent clk_harness/ on PYTHONPATH so a bare clone works without pip install, and `exec python -m clk_harness.cli kickoff "$@"`. The old entry point keeps working with identical flags; the embedded heredocs and the shell setup wizard are gone (they live in clk_harness/kickoff.py since the previous commit). - user_tests/test_mission_e2e.py (the offline shell-provider e2e that ci.yml runs) gains test_kickoff_cli_drives_mission_end_to_end: runs `clk kickoff --no-tui --provider shell` in a temp dir, asserts the bootstrap artifacts, the CLK_* env->config override wiring, and that the mission behaves exactly like the direct e2e (telemetry fires, done-gate blocks premature completion). - tests/test_docs_consistency.py and the telegram kickoff-prompt structural tests now read clk_harness/kickoff.py, where the env-var mapping and wizard live. - user_tests/test_kickoff_e2e.py accepts argparse's "unrecognized arguments" wording alongside the old "unknown option". - README Quick start: `clk kickoff` is the primary form; ./kickoff.sh is documented as the equivalent wrapper. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012jUkqbwroQmYRAdFDsp5r9
Add ruff (E,F,W,I,E722; line-length 120 matching the prevailing style) and a lenient mypy baseline (ignore_missing_imports, untyped defs allowed) to pyproject.toml, and wire both into CI as a new lint job. Mechanical cleanups to reach a clean run, no behavior changes: - ruff --fix: unused imports, import order, f-string/redef fixes - rename ambiguous `l` loop variables (E741) - wrap 27 over-long lines (E501); prompt templates get a per-file ignore because their long lines live inside sent-verbatim strings - split two semicolon statements (E702), drop three unused locals (F841) - mypy: annotate heterogeneous request bodies, Optional log handle, MutableMapping for os.environ, distinct loop vars per engine branch, curses.window annotation, and two targeted type: ignore comments Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012jUkqbwroQmYRAdFDsp5r9
New clk_harness/log.py builds the harness's diagnostic output on stdlib
logging while preserving the exact historic line format
(`2026-07-05T18:32:29 [INFO] ...`, WARN tag, [where] Class: msg +
traceback for exceptions) on stderr and in the per-run .clk/logs/ file.
utils/logging_utils.py becomes a re-export shim so every existing
import keeps working; get_logger() hands out per-module loggers under
the `clk` root, and CLK_LOG_LEVEL=DEBUG surfaces debug breadcrumbs.
Migrate orchestration/{workflow,agent,mission,casting}.py from the
one-shot log() helper to per-module loggers (41 call sites; identical
user-visible output, verified against the file/stderr sinks).
Triage the silently-swallowed `except Exception` blocks across
clk_harness/: 56 sites now emit logger.debug (or logger.warning for
dropped user intent like stop_when/cancel markers) with context; no
control-flow changes. Six TUI sites are deliberately left silent:
four are the log-write plumbing itself (recursion risk through the
redirected stderr stream) and two are per-frame curses render guards.
Web bind check: `clk web` / REST API already default to 127.0.0.1
(api.py DEFAULT_HOST) with explicit CLK_API_HOST/--host opt-in and a
0.0.0.0 warning, and Docker docs pass the opt-in explicitly — no
change needed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012jUkqbwroQmYRAdFDsp5r9
The 109 KB README moves into docs/, following its own section
structure (content moved verbatim; only cross-file anchor links were
rewritten):
- docs/QUICKSTART.md — Pick your path, Quick start, lower-level CLI
- docs/ARCHITECTURE.md — Why CLK, package layout
- docs/CONFIGURATION.md — Cost guardrails (+ multipliers), customization, dry-run
- docs/PROVIDERS.md — Providers, auth modes, Ollama, Pi provider, Pi extension
- docs/TUI.md — Recoverability, GitHub integration, diagnostics, tutorial, workspaces
- docs/WEBUI.md — Web dashboard
- docs/KICKOFF.md — Docker / kickoff.sh path + kickoff workspace layout
- docs/MISSIONS.md — Missions, chief loop, casting, action protocol,
workflows, loops, robustness layers, completion criteria
- docs/TELEGRAM.md — Telegram bot
- docs/TESTING.md — Test suites
- docs/CHANGELOG.md — the "What's new" release notes
- docs/REST_API.md — gains the README's server setup/security section
README shrinks to overview + experimental-software warning, a short
quick start, a documentation index, Safety, and License.
tests/test_docs_consistency.py now reads the robustness documentation
from docs/MISSIONS.md and docs/CONFIGURATION.md (same assertions, new
locations); scripts/run_loop.sh's doc pointer follows. Verified every
non-link line of the old README appears verbatim in the new files.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012jUkqbwroQmYRAdFDsp5r9
…eserves imports) Mechanical split of the 2,088-line workflow.py into a package: - workflow/stages.py — data model + YAML parsing (WorkflowStage, Workflow, StageResult, load_workflow, mini-YAML fallback) - workflow/engine.py — WorkflowRunner: supervise-cycle driver, _run_once stage scheduling, _run_stage execution - workflow/review.py — ReviewMixin: chief review objectives, checkpoints, critic-judge refine loop, adversarial debate panel - workflow/recovery.py — RecoveryMixin: provider-failure classification, recovery dispatches, stall rescue, rollback policy - workflow/validation.py — ValidationMixin: stage validation, outputs contract, stage commit, done-gate integration WorkflowRunner now composes the three mixins; behavior, signatures, and log messages are unchanged. workflow/__init__.py re-exports the full previous module surface so every existing import keeps working. Tests: pytest tests/ 463 passed; user_tests/test_mission_e2e.py 6 passed; ruff + mypy clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012jUkqbwroQmYRAdFDsp5r9
…s imports) Mechanical split of the 1,724-line agent.py into a package: - agent/runner.py — AgentRunner: public run() dispatch entry point, quality-retry loop, proactive auto-consensus, consensus sampling, and the single provider dispatch (_dispatch_once) - agent/prompts.py — PromptsMixin: prompt template loading, context collection, safe substitution, and the meta-prompting layer with its on-disk cache (+ _read_recent_casting_rejections) - agent/transcript.py — AgentSpec / AgentRun / AgentObserver and TranscriptMixin: POST-block application + blocking-question routing, PROPOSE-block application, ACTION-block application with the per-batch auto-commit, and run-history persistence AgentRunner composes the two mixins; behavior, signatures, and log messages are unchanged. agent/__init__.py re-exports the previous module surface so every existing import keeps working. Tests: pytest tests/ 463 passed; user_tests/test_mission_e2e.py 6 passed; ruff + mypy clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012jUkqbwroQmYRAdFDsp5r9
…erves imports) Mechanical split of the 1,270-line casting.py into a package: - casting/director.py — dynamic team casting logic: the PROPOSE_ROLE / PROPOSE_WORKFLOW / PROPOSE_CONSENSUS / PROPOSE_CHARTER / PROPOSE_PLAN parsers, apply_response_proposals, and the casting protocol / objective handed to the chief - casting/roster.py — role/agent definitions: BASELINE_AGENTS, name/prompt duplicate detection (synonym table, suffix stemming, Jaccard prompt similarity), prompt-file scaffolding with the harness protocol suffix, and the registry mutators (register_role / remove_role / write_workflow) + roster summary Behavior, signatures, and log messages are unchanged. casting/__init__.py re-exports the full previous module surface (including the private helpers tests exercise) so every existing import keeps working. Tests: pytest tests/ 463 passed; user_tests/test_mission_e2e.py 6 passed; ruff + mypy clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012jUkqbwroQmYRAdFDsp5r9
Mechanical split of the 2,834-line curses TUI into a package, cut at its existing class boundaries: - tui/app.py — TuiApp: curses layout, rendering, key handling, the heartbeat/telemetry card views, and the run() entry point that cli.cmd_tui invokes - tui/dashboard.py — DashboardState (thread-safe UI/worker shared state), AgentStatus/AgentCard/LogLine, and DashboardObserver - tui/commands.py — the Worker thread and its Job slash-command handlers (/idea /run /mission /loop /doctor /workspaces ...) - tui/stream.py — _StreamToLog stderr/stdout capture that keeps stray writes from corrupting the curses display - tui/theme.py — presentation helpers (_format_tokens, _word_wrap) Behavior, signatures, and log messages are unchanged; the P2 deliberately-silent TUI except blocks are preserved verbatim (except handler parity checked line-by-line against the old module). tui/__init__.py re-exports the previous module surface (run, TuiApp, Worker, DashboardState, classify_error, ...) so every existing import keeps working. Tests: pytest tests/ 463 passed; user_tests/test_mission_e2e.py 6 passed; ruff + mypy clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012jUkqbwroQmYRAdFDsp5r9
…mports) Mechanical split of the 1,068-line web-UI REST surface: - webui_api/router.py — the shared APIRouter, workspace/path guards and secret-masking helpers, pydantic request models, per-workspace config endpoints, doctor, global .env endpoints, idea seeding, and the provider listing - webui_api/events.py — activity history, harness log tail, snapshot, and the SSE activity stream - webui_api/files.py — workspace file list/read/write/download and the git-history endpoints - webui_router.py — kept as the compatibility shim: it re-exports the full previous surface (router, helpers, models) so api.py's `from .webui_router import router` and all test imports keep working. The provider probe/discovery cluster (_probe_blocking, _discover_blocking, /api/providers/discover, /api/providers/probe) deliberately stays here rather than moving: the test suite patches webui_router._probe_blocking as a module global, and the discovery path must resolve that patched name at call time. All 28 routes register exactly as before (parity checked); behavior and log messages are unchanged. Tests: pytest tests/ 463 passed; user_tests/test_mission_e2e.py 6 passed; ruff + mypy clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012jUkqbwroQmYRAdFDsp5r9
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three-phase overhaul of CLK. Every commit kept CI green (
pytest tests/+ offline shell-provider mission e2e +ruff+mypy).Phase 1 — Features
clk kickoffsubcommand replaces the 51 KBkickoff.sh(now a 55-line wrapper thatexecs the CLI). The two embedded-Python heredocs (config overrides + provider activation) were ported toclk_harness/kickoff.pyand verified byte-identical against goldens generated from the original shell before deletion (21-case matrix: bool/csv casts, all provider/auth-mode branches). Offlineclk kickoff --no-tui --provider shelle2e added.Suite 421 → 463.
Phase 2 — Quality gates + logging + docs
ruff+mypywired into CI (166 lint findings + 24 type errors fixed mechanically).clk_harness/log.py) unifies the harness output format; 41 orchestration call sites migrated to per-module loggers; ~56 silentexceptblocks now log with context.docs/set (content moved verbatim, anchors verified).clk webalready binds loopback by default — no change needed.Phase 3 — Full decomposition
Each god-file became a package behind an import-preserving shim — all 50 test files untouched:
orchestration/workflow.py(2,088) →workflow/(engine/stages/review/recovery/validation)orchestration/agent.py(1,724) →agent/(runner/prompts/transcript)orchestration/casting.py(1,270) →casting/(director/roster)tui.py(2,834) →tui/(app/commands/dashboard/stream/theme)webui_router.py(1,068) →webui_api/with a compatibility shimSingle-class files were split via behavior-identical mixins with
TYPE_CHECKINGstubs somypystays clean at zero runtime cost.🤖 Generated with Claude Code
https://claude.ai/code/session_012jUkqbwroQmYRAdFDsp5r9
Generated by Claude Code