Skip to content

✨ Harness: descriptive plan ladder and push titles for viewers - #230

Merged
ibolton336 merged 7 commits into
konveyor:mainfrom
ibolton336:fix/harness-plan-stage-rung
Sep 15, 2026
Merged

ibolton336 merged 7 commits into
konveyor:mainfrom
ibolton336:fix/harness-plan-stage-rung

Conversation

@ibolton336

@ibolton336 ibolton336 commented Sep 14, 2026

Copy link
Copy Markdown
Member

Problem

The plan ladder the harness publishes to viewers (rendered in the console's session panel) has three rungs. Two of them were fixed text for every run: "Prepare workspace: clone, branch, grounding data" and "Agent works the stage task". Only the third named anything concrete (the push branch). The push tool calls viewers see as "Harness: git push (final)" did not say where they pushed. And the middle rung was one spinner for the entire agent turn, which can be minutes and hundreds of tool calls.

Change

Build the rungs from what the harness already has in hand, and keep the middle one moving.

First rung (commit 2): repository as host and path with credentials and .git dropped, the branch, and the number of analysis insights written. "no analysis insights" when the fetch returned none; nothing about insights when the run has no skills and never fetched them.

Prepare workspace: github.com/konveyor/coolstore on branch migration-1789401169, 49 analysis insights

Middle rung (commits 1 and 3): workflow stage position when there is one, the first non-empty line of the stage instructions (falling back to the agent prompt) stripped of Markdown markers and cut to 80 runes, the model, and the turn budget. Before the prompt is sent it reads up to 40 turns; once running, the ladder is re-emitted on every turn with progress:

Stage 2 of 3 — agent works the task: “Fix the findings from the assess stage.” (gemini-2.5-pro, turn 12 of 200)

Stage text is rendered with parameter values substituted, so the excerpt goes through the same redactor the closing-message path uses (#170). "Turn" is the measure the harness already reports as TurnsUsed and compares against the native limit in classifyOutcome: one per tool_call notification. The handoff prompt continues the count from where the primary stopped.

Push titles (commit 2): git push to branch <branch> (final) and (auto-commit watcher).

Plumbing (commit 3):

  • SessionClient.SetTurnHandler fires from SendPrompt's loop each time the count advances; runStage re-emits the ladder from it.
  • Re-emitting per turn would fill the tee's 32-frame replay ring and evict the push and outcome frames a late viewer needs, so the ring now keys plan frames: a new ladder replaces the previous one instead of stacking. Unkeyed frames accumulate as before.
  • fetchAndWriteAnalysis returns the count it already logs. workflowStagePosition is split out of isIntermediateWorkflowStage so both read the stage metadata the same way; an index past the count now counts as invalid there too, which changes nothing for the intermediate check.

Verification

  • go build, go vet, gofmt -l clean in harness/; full go test ./... in harness/ passes.
  • New tests: the rung text across standalone/staged runs, prompt fallback, truncation, empty case, invalid stage metadata, secret masking, and progress wording with and without a budget; the prep rung's credential stripping, .git trimming, zero vs. unfetched insights and unparseable URL; the turn handler firing once per tool_call with the running count and not for other updates (demux-server test); and the replay ring keeping only the latest plan while unkeyed frames accumulate.

🤖 Generated with Claude Code

The plan ladder the harness publishes to viewers had a middle rung that
read "Agent works the stage task" for every run. The rungs around it say
what prep did and which branch the push targets; this one said nothing a
viewer could act on, for the whole time the agent was running.

Build the rung from what the harness already knows: the workflow stage
position ("Stage 2 of 3"), the first line of the stage instructions (the
agent prompt when a run has none) cut to 80 runes, and the model and turn
budget. Stage text is rendered with parameter values substituted, so the
excerpt goes through the redactor like every other text the harness
publishes.

workflowStagePosition is split out of isIntermediateWorkflowStage so both
read the stage metadata the same way; an index past the count now counts
as invalid there too, which changes nothing for the intermediate check.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: ibolton336 <ibolton@redhat.com>
@coderabbitai

coderabbitai Bot commented Sep 14, 2026

Copy link
Copy Markdown
Contributor

Important

  • 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: b082ea2e-5d8a-4921-96d7-e671acd552fb


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

ibolton336 and others added 2 commits September 14, 2026 14:01
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: ibolton336 <ibolton@redhat.com>
… push titles

The first plan rung read "Prepare workspace: clone, branch, grounding
data" for every run, as static as the middle one was. Build it from what
prep produced: the repository (host and path, credentials and .git
dropped), the branch, and how many analysis insights were written —
"no analysis insights" when the fetch returned none, nothing at all when
the run has no skills and so never fetched them.

fetchAndWriteAnalysis returns the count it already logs. The two push
tool calls viewers see as "Harness: git push (final)" and "(auto-commit
watcher)" now name the branch they push to.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: ibolton336 <ibolton@redhat.com>
@ibolton336 ibolton336 changed the title ✨ Harness: name the stage, task and budget on the viewer plan rung ✨ Harness: descriptive plan ladder and push titles for viewers Sep 14, 2026
… budget

The middle rung was one spinner for the whole agent turn, which can be
minutes and hundreds of tool calls: it flipped from in_progress to
completed only when goose returned. The harness already counts turns
during the prompt (TurnsUsed, one per tool_call notification, the same
measure classifyOutcome uses for the native limit) but only reported the
total afterwards.

SessionClient gains SetTurnHandler, invoked from SendPrompt's loop each
time the count advances. runStage re-emits the ladder from it, so the
rung reads "(model, turn 12 of 40)" while running and "up to 40 turns"
before the prompt is sent; the handoff prompt continues the count from
where the primary stopped.

Re-emitting the ladder per turn would fill the tee's 32-frame replay
ring and evict the push and outcome frames a late viewer needs, so the
ring now keys plan frames: a new ladder replaces the previous one rather
than stacking. Unkeyed frames (pushes, notices) accumulate as before.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: ibolton336 <ibolton@redhat.com>
Comment thread harness/cmd/migration-harness/main.go Outdated
ibolton336 and others added 3 commits September 14, 2026 16:45
planTaskRung cut the excerpt to 80 runes and then redacted it. Exact-match
redaction cannot recognise a token the cutoff has split, so a known secret
straddling the boundary left its head in the rung — and in the tee's
replay ring for every late viewer. Redact the full stage instructions and
agent prompt first, then excerpt.

Regression test places a 30-rune token so the cut lands inside it, for
both the instructions and the agent-prompt fallback.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: ibolton336 <ibolton@redhat.com>
Seen live: a run with no instructions quoted the agent prompt's first
line — the persona, "You are a senior Java engineer performing one stage
of a staged Java EE 7 to" — cut at the prompt's hard wrap with no
ellipsis, because the excerpt took the first LINE and YAML prose wraps
at ~76 columns.

The rung now quotes only the stage instructions (what a person typed for
this run) and says "agent works its standing prompt" when there are none.
The excerpt is the first paragraph joined into one line, so a hard wrap
is not mistaken for the end and a real cut always shows its ellipsis.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: ibolton336 <ibolton@redhat.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: ibolton336 <ibolton@redhat.com>

@djzager djzager left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed the updates. The redaction issue is fixed: full instructions are redacted before excerpting, and the regression test plus the original reproduction pass. No further actionable findings.

@ibolton336
ibolton336 merged commit 6133e4d into konveyor:main Sep 15, 2026
25 checks passed
ibolton336 added a commit to ibolton336/agentic-controller that referenced this pull request Sep 18, 2026
Two things a viewer reads on the middle rung, from watching a coolstore run
on the ROKS demo cluster:

  Agent works its standing prompt (us.anthropic.claude-sonnet-4-5-20250929-v1:0, turn 17 of 200)

"standing prompt" is our word, not theirs, and it collides with the run's
own prompt. There is nothing to quote in that branch — the run was given no
task text, and the agent prompt opens with a persona — so the rung now just
says what is true: "Agent is working". The stage, model and turn budget
beside it carry the rest.

The model is a Bedrock cross-region inference profile id: the geo group,
the vendor namespace and the "-v1:0" API version say nothing about which
model is answering. modelDisplayName drops those three and keeps identity —
name and snapshot date — so the rung reads "claude-sonnet-4-5-20250929".
It only touches that dotted shape: a vendor segment comes off only when it
is all letters, so "gemini-2.5-pro" and "gpt-4.1" keep their versions, and
a path-shaped Vertex ref is left alone. The full id is still on the Gateway
the run named and in the pod's KONVEYOR_LLM_MODEL.

No changelog fragment: the plan ladder (konveyor#230) is not in alpha.4, so this
wording has never been in a release.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: ibolton336 <ibolton@redhat.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants