Skip to content

feat(runs): record each dbt run's results and show them in /runs - #27

Open
hkaltai wants to merge 15 commits into
mainfrom
feat/runs-pane
Open

hkaltai wants to merge 15 commits into
mainfrom
feat/runs-pane

Conversation

@hkaltai

@hkaltai hkaltai commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

What

Every dbt command Claude runs is recorded with its results, shown in /runs, and can be re-run through the prompt.

  • After a Bash call that ran build, run, test, seed, snapshot, compile, retry or clone, the plugin calls altimate-code's dbt_run_results and records the run once per invocation id (an unchanged run_results.json is ignored). Up to 50 runs per project are kept in the plugin store; messages are cut to 300 characters, rows to 100 per run, and the whole store is held under 2.5 MB.
  • Claude gets one context line after each recorded run, e.g. build: 41 ok, 2 errors: stg_orders, fct_sales.
  • /runs opens a pane: one button per run (command, seconds, passed of total); inside a run the failures come first. Re-run fills the prompt with the recorded command; Re-run failed (shown on the newest run when it has failures) fills it with dbt retry.

The feature fills in hooks/features/runs.tsx and RunsState in types/index.d.ts only.

Depends on

The companion altimate-code PR (AltimateAI/altimate-code#1729) that implements dbt_run_results. Merge that one first. This branch also carries the shared foundation commit; retarget or rebase once the foundation lands on main.

Verification

claude plugin test plugins/altimate-code: 212 pass, including hooks/features/runs.test.tsx (a run recorded once, the same file ignored, the pane on terminal and desktop, Re-run and Re-run failed filling the prompt, the 50-run, row, message and store caps). claude plugin validate passes.

Demo video (hosted)

P-11.mp4 — watch on Notion (Altimate workspace; recorded from the approved head of this PR).

🤖 Generated with Claude Code

Demo video: P-11.mp4 (re-recorded; 58s: dbt_run_results tool on a dbt fixture, then dbt builds in Claude Code and the /runs pane).

hkaltai and others added 7 commits November 13, 2026 16:31
Each feature gets its own file under hooks/features/, called by name from
features/index.ts in a fixed order: its own hooks, the session and turn
events, its band line and its tool-call phases. One tool.call hook
(tool-chain.ts) runs Bash and file handling in fixed phases: bashRewrite,
bashGate, the call, bashAfter, editAfter, readOrEditAfter. The status line
is built from parts (status-line.ts). runTool moves to altimate-cli.ts with
a `write` option that goes through the CLI with --allow-write.
dbt-command.ts finds the dbt invocations in a shell command line. New
userConfig fields: maxScanBytes, healthBlocksPush, healthConfigPath,
commentsPollSeconds, commentsInContext.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Subshells after keywords, case patterns, process substitution, arrays,
arithmetic and function definitions parse again; quotes in a heredoc inside
a substitution are text; a function or case pattern named dbt is not an
invocation.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…t wrote it, keep Re-run complete

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…is per project

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@hkaltai

hkaltai commented Nov 13, 2026

Copy link
Copy Markdown
Contributor Author

Review loop on this PR: two independent reviewers (claude -p, codex exec) in two rounds, plus attack-the-fix gates. Round 1 raised a CRITICAL, so a second round ran. Findings and what happened:

Finding Resolution
CRITICAL: no run recorded with the plugin's MCP server connected (it does not offer dbt_run_results) Fixed: the tool is called through the CLI. Test covers a connected server.
MAJOR: wrong project read; cd, --project-dir and the session's own folder ignored; the globally configured project was read for a command run in the session folder Fixed: the plugin always passes the command's folder (the session's current folder, moved by any cd, or --project-dir); the tool looks for the project above it.
MAJOR: an older results file recorded as the new run Fixed: each call's start time is kept (by tool-use id) and the file must be newer; the invocation is chosen by the artifact's own command (dbt run && dbt test that stopped after the run records the run).
MAJOR: Re-run and Re-run failed dropped cd, environment and flags Fixed: both lines keep the folder (always), the environment, the launcher and --project-dir/--profiles-dir/--target-path/--target/--profile/-t. A secret-named assignment is replayed as NAME="$NAME" and its value is never stored.
MAJOR: store budget in characters, too large Fixed: counted in UTF-8 bytes, 1,000,000 byte budget, 50 rows per run; the test reaches the trim branch.
MAJOR: concurrent read-modify-write of the store key Fixed within one session by serializing updates; two Claude Code sessions finishing together can still race.
MAJOR: artifact text in Claude's context Fixed: names are reduced to [\w.-], 80 characters.
MINOR: atom shape, doc comments, "and N more" count, backtick in the Re-run text, pane update clobbering open, Re-run failed per project Fixed.

Left as known limits (round 2, not fixed): history is kept per session folder (50 runs shared by projects under it); a cd inside a subshell or after a failed && is treated as taken; ~, $VAR, DBT_PROJECT_DIR and --target-path overrides are not expanded or read (nothing is recorded, never a wrong run); two dbt commands of one call keep only the last one's run; secret redaction covers leading NAME=value assignments with common secret names only; a reload mid-call loses that call's record.

hkaltai and others added 2 commits November 13, 2026 21:21
…ead per project folder

Where a command ran is read only from plain cd statements; anything the shell could read
differently (quotes, variables, subshells, pipes, comments, if, pushd, eval) records nothing
instead of another project's run. A run written with --target-path or DBT_TARGET_PATH is read
from that folder. A call that runs dbt in several projects records each; reads run together and
one failing does not lose the others. Calls that overlap in one artifact folder are not
recorded, since the file does not say which command wrote it. History keeps 50 runs per project,
newest first by completion. Bare secret names (TOKEN, PASSWORD, KEY) are left out of Re-run.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@hkaltai

hkaltai commented Nov 13, 2026

Copy link
Copy Markdown
Contributor Author

Review-loop result (Claude + Codex, independent reviewers)

Two independent external reviews, then three attack-the-fix gates. Replies to every finding raised against runs.tsx:

Finding Reply
--target-path / DBT_TARGET_PATH ignored when reading Fixed: sent to the tool as target_path; a value the shell expands ($, ~, globs) is not recorded.
Working-directory inference wrong (leading space, subshell, quotes, comments, redirections, if, pipes, c\d, cd -, pushd, eval, source) Fixed by replacing the hand-written scanner (two gates kept finding holes): only plain cd <word> statements joined by &&, ; or a new line are followed; anything else around a cd records nothing instead of another project's run. A cd after another command with && is trusted only when the dbt call is in the same && chain. A fixed list of 40 commands is in the test.
A call spanning projects records only the last Fixed: one read per project and target folder, together; one failing does not lose the others.
Concurrent runs saved with the wrong Re-run line Fixed for what can be told: calls that overlap in one artifact folder (the project's own target, named or not) are not recorded; an artifact written after the call ended is rejected; of two spellings of one folder the later call's line is kept. A call that never ended (refused/cancelled) no longer blocks later calls. Mutation checks confirm the tests catch each.
50 runs per project, not per session folder Fixed.
History order across batches Fixed: newest first by completion time.
Bare TOKEN / PASSWORD / KEY / SECRET not redacted Fixed.
--project-dir $VAR guessed Fixed: not recorded.
dbt build && dbt docs generate, dbt run && dbt test lose the first run Not changed: one run_results.json is overwritten by the next command before bashAfter can read it.
Background runs not recorded Not changed: the hook reads before dbt ends; needs a separate lifecycle.
A failed cd; dbt build replays with the failed cd Not changed: the command was already wrong; its read is dropped by the freshness check.

The last gate's findings were all fixed after it ran and covered by tests, but that last round was not re-gated. 240 plugin tests pass. claude plugin validate --strict reports warnings in files this PR does not own (dbt-command.ts, lineage.test.tsx); none are in runs.tsx or its test.

Demo video (real Claude Code session with this plugin and the altimate-code branch CLI, a dbt-duckdb project): /Users/harish/.quikim/artifacts/pu-mods/videos/P-11.mp4.

@hkaltai

hkaltai commented Nov 14, 2026

Copy link
Copy Markdown
Contributor Author

Final-gate review (independent Claude + Codex CLI reviewers on HEAD).

Open, NOT fixed (so this PR stays a draft):

  • MAJOR runs.tsx:348 / :344 — an older in-flight dbt call in the same folder is ignored by the overlap guard, so a later failed call can record the older call's artifact under its own command line.
  • MAJOR runs.tsx:299 (folderOf) — guard keys are compared as typed, so /repo and /repo/models (same project target) don't count as overlapping.
  • MAJOR runs.tsx:474 (Claude) — calls whose folder can't be resolved (cd "$P" && dbt build, --project-dir $X) are never registered as in-flight writers.
  • MINOR: unparseable completedAt passes the time-window check; env -C dir not treated as a folder change. NIT: stale bashAfter doc comment; rows.indexOf(row) in render.

I tried a redesign (any overlapping dbt call in the session blocks recording, no folder comparison, 30-minute staleness for never-ended calls) but could not run the plugin tests here (claude-code/testing module is unavailable in this checkout) and it conflicts with the existing "cancelled call must not block later calls" test, so I reverted it rather than push an unverified change. This guard has now failed four review rounds; it needs an owner decision on the intended trade-off (block on any overlap vs. per-folder) before another patch.

@hkaltai

hkaltai commented Nov 14, 2026

Copy link
Copy Markdown
Contributor Author

Verify-only gate: no commits since the final-gate review, so nothing new to approve. The three MAJOR findings in the runs.tsx overlap guard stay open and this PR stays a draft until the owner picks the intended trade-off (block on any overlapping dbt call vs. compare per folder).

hkaltai and others added 3 commits November 14, 2026 13:20
…t, in any folder; otherwise Claude is told

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ll over an hour is not recorded

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@hkaltai

hkaltai commented Nov 14, 2026

Copy link
Copy Markdown
Contributor Author

Replies to the three open MAJOR findings in the runs.tsx overlap guard, per the owner's decision: if any other dbt call overlapped this one in time (any folder), the run is not recorded and Claude gets one line, run not recorded: overlapping dbt commands. invocationId de-dup is unchanged.

  • Older in-flight call ignored: fixed. An older call still running now counts, so the later call is left out. Test: 'an older call still running in the same folder holds back the later call that ends first'.
  • /repo and /repo/models as different folders: fixed by the rule, since folder no longer matters. Test: 'two spellings of one project folder overlap'.
  • Unreadable folders never registered: fixed. Every dbt call is tracked in flight, whatever folder it names. A lone call whose folder cannot be told records nothing and says nothing. Tests: 'a call whose folder cannot be told still counts…' and '…alone, records nothing and says nothing'.

Review of the new commits (Claude + Codex, separate processes) found more, all fixed with tests that fail on the earlier code: bookkeeping ran after an await, so two calls ending together could both look alone; dbt ls/parse did not count; a call over an hour could record; a call whose hour ran out mid-run was forgotten. A cancelled call blocks overlapping calls for one hour, then stops counting.

Gate status: the first verify pass on db4f1c1 returned REQUEST CHANGES from both reviewers. The second, on 040d424, returned APPROVED from Claude and REQUEST CHANGES from Codex (the hour-expiry case). That is fixed in 3fd6303 with a test (248 plugin tests pass), but no reviewer has seen it, and the gate's two-pass limit is used up. So this PR stays a draft and has no APPROVED comment.

…ping it is not recorded

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@hkaltai
hkaltai marked this pull request as ready for review November 15, 2026 14:21
@hkaltai

hkaltai commented Nov 15, 2026

Copy link
Copy Markdown
Contributor Author

review-loop: APPROVED at 2283219

Gate on the overlap-guard fixes: Claude and Codex both APPROVED. A real MAJOR found on the way (a backgrounded dbt call dropped out of the overlap bookkeeping) was fixed in 2283219 with a regression test that fails without the fix; 249 plugin tests pass. Accepted MINOR limits: backgrounding forms not ending in '&' (e.g. '& disown'), 1-hour age-out, 50/500 caps.

Comment thread plugins/altimate-code/types/index.d.ts Resolved
@kilo-code-bot

kilo-code-bot Bot commented Nov 15, 2026 •

Copy link
Copy Markdown

Code Review Summary

Status: 3 Issues Found | Recommendation: Address before merge

Incremental review of e8f97f5..9824ab0 (one commit, two comment-only lines). The two documentation findings from the previous pass are resolved: hooks/features/index.ts:7 now states that a register failure is silent and the other hooks log to debug (matching failed()/catch {}), and types/index.d.ts:193 now describes the fallback id as the project folder plus completion time (matching toRun's ${projectDir}\u0000${completedAt}). The remaining findings below were re-verified against current HEAD and are unchanged, already covered by active inline comments, so no new inline comments were posted.

Overview

Severity Count
CRITICAL 0
WARNING 2
SUGGESTION 1
Issue Details (click to expand)

WARNING

File Line Issue
plugins/altimate-code/hooks/features/runs.tsx 353 A backgrounded dbt call returns before started.delete(...), so its entry survives for up to IN_FLIGHT_MS (1 hour); every later dbt call then sees live.length > 0, returns OVERLAP_LINE, records nothing, and the backgrounded run itself is never recorded. Author accepted this as a documented limitation, but a single dbt build & still disables /runs recording for an hour.
plugins/altimate-code/.claude-plugin/plugin.json 24 None of maxScanBytes, healthBlocksPush, healthConfigPath, commentsPollSeconds, commentsInContext is read anywhere in plugins/; the descriptions promise unimplemented behaviour (/allow-scan, refusing git push). Author notes sibling PRs will consume them, but shipping this manifest alone shows dead settings.

SUGGESTION

File Line Issue
plugins/altimate-code/hooks/features/index.test.ts 5 Eight of nine register stubs are empty, so the throwing proxy is only reached by runs and calls is always 1. A single whole-block try/catch around all nine would also satisfy both assertions, so the test does not prove per-feature isolation. Assert later registers still reached on once a second feature has a body.
Files Reviewed (5 files)
  • plugins/altimate-code/hooks/features/index.ts - 0 issues (fixed)
  • plugins/altimate-code/types/index.d.ts - 0 issues (fixed)
  • plugins/altimate-code/hooks/features/runs.tsx - 1 issue
  • plugins/altimate-code/.claude-plugin/plugin.json - 1 issue
  • plugins/altimate-code/hooks/features/index.test.ts - 1 issue

Fix these issues in Kilo Cloud

Previous Review Summaries (2 snapshots, latest commit e8f97f5)

Current summary above is authoritative. Previous snapshots are kept for context only.

Previous review (commit e8f97f5)

Status: 5 Issues Found | Recommendation: Address before merge

Overview

Severity Count
CRITICAL 0
WARNING 2
SUGGESTION 3
Issue Details (click to expand)

WARNING

File Line Issue
plugins/altimate-code/hooks/features/runs.tsx 353 A backgrounded dbt call's started entry is never cleared, so one dbt build & blocks recording (and reports run not recorded: overlapping dbt commands) for every dbt run for up to an hour; the backgrounded run itself is never recorded.
plugins/altimate-code/.claude-plugin/plugin.json 24 The new userConfig keys are read by nothing, yet their descriptions promise unimplemented behaviour (/allow-scan, refusing git push, comment polling/injection).

SUGGESTION

File Line Issue
plugins/altimate-code/hooks/features/index.test.ts 5 The new test passes without the per-feature guard it is named for: eight of the nine register stubs are empty, so calls is always 1 and a single whole-block try/catch would pass too.
plugins/altimate-code/hooks/features/index.ts 161 The new catch {} guards drop the error, so a feature that fails to register disappears silently, contrary to the module header's "logged and skipped".
plugins/altimate-code/types/index.d.ts 193 RunRecord.id is documented as the completion time when the artifact has no invocation id, but toRecord now prefixes the project folder.
Files Reviewed (6 files)
  • plugins/altimate-code/.claude-plugin/plugin.json - 1 issue
  • plugins/altimate-code/hooks/features/index.ts - 1 issue
  • plugins/altimate-code/hooks/features/index.test.ts - 1 issue
  • plugins/altimate-code/hooks/features/runs.tsx - 1 issue
  • plugins/altimate-code/hooks/features/runs.test.tsx
  • plugins/altimate-code/types/index.d.ts - 1 issue

Previous review (commit 2283219)

Status: 6 Issues Found | Recommendation: Address before merge

Overview

Severity Count
CRITICAL 0
WARNING 2
SUGGESTION 4
Issue Details (click to expand)

WARNING

File Line Issue
plugins/altimate-code/hooks/features/runs.tsx 352 A backgrounded dbt call's started entry is never cleared, so one dbt build & blocks recording (and reports run not recorded: overlapping dbt commands) for every dbt run for up to an hour; the backgrounded run itself is never recorded.
plugins/altimate-code/.claude-plugin/plugin.json 24 The new userConfig keys are read by nothing, yet their descriptions promise unimplemented behaviour (/allow-scan, refusing git push, comment polling/injection).

SUGGESTION

File Line Issue
plugins/altimate-code/hooks/features/runs.tsx 228 Runs are deduplicated by id across projects, but id falls back to completedAt when the artifact has no invocation id; two projects' runs finishing in the same second evict each other (and collide in the pane key). Key by project + id.
plugins/altimate-code/hooks/features/runs.tsx 373 const free = mine is a leftover alias (the folder filter was removed); readRun's run && written is likewise redundant.
plugins/altimate-code/types/index.d.ts 220 RunsState.runs is documented as "at most 50", but withRun now keeps 50 per project.
plugins/altimate-code/hooks/integrations.tsx 822 registerFeatures is unguarded; a feature register that throws prevents installToolChain(on) from installing, leaving the plugin with no tool.call hook.
Files Reviewed (21 files)
  • plugins/altimate-code/.claude-plugin/plugin.json - 1 issue
  • plugins/altimate-code/hooks/altimate-cli.ts
  • plugins/altimate-code/hooks/dbt-command.test.ts
  • plugins/altimate-code/hooks/dbt-command.ts
  • plugins/altimate-code/hooks/features/comments.tsx
  • plugins/altimate-code/hooks/features/cost.tsx
  • plugins/altimate-code/hooks/features/cte-profile.tsx
  • plugins/altimate-code/hooks/features/defer.tsx
  • plugins/altimate-code/hooks/features/docs.tsx
  • plugins/altimate-code/hooks/features/health.tsx
  • plugins/altimate-code/hooks/features/index.ts
  • plugins/altimate-code/hooks/features/runs.test.tsx
  • plugins/altimate-code/hooks/features/runs.tsx - 3 issues
  • plugins/altimate-code/hooks/features/sql-graph.tsx
  • plugins/altimate-code/hooks/features/to-model.tsx
  • plugins/altimate-code/hooks/integrations.tsx - 1 issue
  • plugins/altimate-code/hooks/lineage.tsx
  • plugins/altimate-code/hooks/status-line.ts
  • plugins/altimate-code/hooks/tool-chain.test.ts
  • plugins/altimate-code/hooks/tool-chain.ts
  • plugins/altimate-code/types/index.d.ts - 1 issue

Fix these issues in Kilo Cloud


Reviewed by deepseek-v4.1-flash-1017 · Input: 76.4K · Output: 8.6K · Cached: 578.8K

…ng feature register no longer drops the tool hook, doc and alias tidy

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

import { registerFeatures } from './index'

test('a feature whose register throws never reaches the caller, so the tool hook after the features still installs', () => {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SUGGESTION: The test would pass without the per-feature guard it is named for

Eight of the nine features still have an empty register(on, context) {} (defer, to-model, comments, cost, health, sql-graph, docs, cte-profile), so the proxy is only ever reached by runs and calls is always 1. A single try { ...all nine... } catch {} around the whole block — the case this change exists to prevent, where the first failure stops the rest — also satisfies both assertions, so nothing here shows that one feature's throw leaves the others or installToolChain alive. Once a second feature has a body, make the proxy throw on its first call only and assert the later registers still reached on (calls should equal the number of features, not 2).

// One feature's register throwing (a duplicate command, a refused matcher) must not stop the others or the tool hook after them.
try {
runsRegister(on, context)
} catch {}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SUGGESTION: A feature that fails to register now fails silently

The module header (line 7) says a function that throws "is logged and skipped", but these empty catch {} blocks drop the error: a duplicate command name or a refused matcher makes a feature disappear with nothing in the debug log, where previously the plugin failed loudly at load. failed() needs $, which registerFeatures does not receive, so the failure could be collected here and logged from featuresSessionStart, the first place $ is available. Otherwise the header comment should say register failures are silent.

Comment thread plugins/altimate-code/types/index.d.ts Outdated
}

export type RunRecord = {
/** The artifact's invocation id; its completion time when the artifact has none. */

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SUGGESTION: The fallback id is no longer just the completion time

toRecord now builds ${view.projectDir ?? ''}\u0000${completedAt} when the artifact has no invocation id (hooks/features/runs.tsx:204), so the id is the project plus the completion time when the artifact has none. Worth saying so here, since this comment is the contract consumers of RunRecord.id read.

@hkaltai

hkaltai commented Nov 15, 2026

Copy link
Copy Markdown
Contributor Author

review-loop: APPROVED at e8f97f5

…ompletion time

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant