Skip to content

🐛 ask_user is opt-in: spec.execution.askUser - #242

Merged
ibolton336 merged 2 commits into
konveyor:mainfrom
ibolton336:claude/ask-user-opt-in
Sep 18, 2026
Merged

ibolton336 merged 2 commits into
konveyor:mainfrom
ibolton336:claude/ask-user-opt-in

Conversation

@ibolton336

@ibolton336 ibolton336 commented Sep 17, 2026

Copy link
Copy Markdown
Member

Why

ask_user (#161, ADR 0017) shipped default-on, with HARNESS_HITL_ASK=off
as the opt-out. The gate behind it is fail-closed by design — an unanswered
question fails the run, the harness never answers for the human — and I think
that part is right. The default is what's wrong:

  • Whether the model asks is its own call. On the ROKS demo cluster today,
    the same Agent, prompt and gateway Succeeded once (ui-gckqt), then on the
    next run decided at turn 38 to ask a clarifying question and was Failed
    180 s later: stage failed: ask_user question unanswered (HITL gate).
    Nothing on the run or in the console said that could happen.
  • Default-on doesn't just expose the tool, it tells every run a human is
    there.
    Mounting the tool also appends askUserGuideline to the prompt
    (AskUser: len(mcpServers) > 0, same commit as the tool): "A human may be
    watching this run. When you need a decision only a human can make — a missing
    prerequisite, an ambiguous requirement, a choice between approaches, anything
    destructive — call the ask_user tool and wait…"
    . A migration is made of
    choices between approaches, so a question is the behavior the run was told
    to perform, not a model whim — and for an unattended run the premise is
    false. The Agent's own prompt in the failing run (236 chars) says nothing
    about asking; all of it comes from the harness. The guideline is right for
    its purpose (it stops the ask-in-prose-then-end-the-turn bug from before
    ✨ harness: ask_user tool — the agent can stop and ask the human (ACP elicitation) #161) and is untouched here: it stays gated on the tool being mounted, so
    with opt-in it only reaches runs where "a human may be watching" is true.
  • Unattended is the common case — workflow stages, anything started from
    the inventory and left, anyone who closes the tab. ADR 0017 says "a run that
    must complete unattended sets HARNESS_HITL_ASK=off", which puts the burden
    on the wrong runs.
  • Opting out was possible but nobody would find it. A run can carry
    spec.env: [{name: HARNESS_HITL_ASK, value: "off"}], but nothing surfaces
    that: the console's create-run modal has no env field, the default Agents
    don't set it, and the run itself gives no hint it can be failed this way.
  • mode: auto wasn't headless-safe, which is what CONTEXT.md says it is.

The code comment for the default-on block says it exists so "the one E2E path
must exercise the tee (and steering)". That holds for the tee and steering,
which are passive. ask_user is the only one of the three that can fail a run.

What

  • API: ExecutionSpec.askUser (bool, default false), next to mode — so
    it is settable on AgentRun.spec.execution and per AgentWorkflow stage,
    and, like mode, not on the Agent (execution-time concern, ADR 0011/0018).
    resolveExecution carries it from the override; executionSection writes
    execution.askUser: true into params.json only when set.
  • Harness: HITLAsk = (params execution.askUser || HARNESS_HITL_ASK=on) && !HARNESS_HITL_ASK=off.
    on keeps a path for harness runs outside the controller (hack/harness-*,
    the live ask test mounts the MCP server itself and is unaffected); off
    still beats everything. The startup log now says which way it went.
  • Docs: ADR 0017 amended in place with a dated update (the gate, and the
    rejection of "ask but proceed on timeout", are unchanged); harness README
    and agent-base-images env tables; a CONTEXT.md Ask User entry.
    I deliberately did not call the field interactive — the glossary reserves
    that as a term to avoid — and did not fold it into mode: approve:
    approval gates the agent's tool calls, asking is the agent's initiative, and
    a 200-turn migration wants questions without approving every shell call.

Filed as :bug:: mode: auto is documented as headless-safe and wasn't, and
this restores that. Upgrade note all the same — a run that relied on the
old default loses the tool until it sets askUser: true, and the console
cannot set it until the Hub bump and UI checkbox below land (planned as fast
follow-ups, before the next alpha). Opting out never made the agent guess — it
removes the tool, which is the pre-#161 behavior.

Follow-ups (not here)

  • Hub binds the create body straight into this repo's AgentRun Go type
    (type AgentRun = agent.AgentRun), vendored at agentic-controller/api
    22fee49 (2026-08-26). Until Hub bumps that dependency past this PR, an
    askUser in the request is dropped on unmarshal — silently. Then the
    console's create-run modal wants a checkbox ("Let the agent ask me questions
    — the run fails if one goes unanswered"), unchecked by default.
  • A pending question in the console's session panel is as easy to scroll past
    as the plan was (🐛 Session panel: pin the plan, and stop the usage chip reporting what it cannot know tackle2-ui#3608); it wants pinning and a countdown.

Verification

  • make generate manifests — three CRDs gain askUser, nothing else moves
  • make test — all packages pass (controller envtest suite included)
  • cd harness && go test ./... — pass; new TestHITLAskIsOptIn covers
    default-off, run opt-in, env opt-in, env-off-beats-run, junk env value
  • TestResolveExecution extended: askUser comes only from the override, an
    askUser-only override stays non-nil, and the key is absent from params.json
    unless opted into
  • make lint — 0 issues

Not run: the kind e2e suite, and no live cluster run on an image built from
this branch.

🤖 Generated with Claude Code

ask_user shipped default-on (konveyor#161). The gate behind it is fail-closed by
design — an unanswered question fails the run — and whether the model
asks is its own call, so any run could be failed by a question nobody was
attached to see. On the ROKS demo cluster the same Agent and prompt
Succeeded once, then Failed at turn 38 on an unanswered ask. It also made
mode: auto not the headless-safe policy CONTEXT.md says it is.

The tool is now mounted only when the run opts in:
spec.execution.askUser on an AgentRun or AgentWorkflow stage, carried
next to mode and delivered as execution.askUser in params.json; or
HARNESS_HITL_ASK=on for a harness run outside the controller.
HARNESS_HITL_ASK=off still wins. The gate itself is unchanged.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: ibolton336 <ibolton@redhat.com>
@coderabbitai

coderabbitai Bot commented Sep 17, 2026

Copy link
Copy Markdown
Contributor

Important

  • 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: 606119b3-cf42-43e9-b74d-827733a6b437


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: ibolton336 <ibolton@redhat.com>
@ibolton336 ibolton336 changed the title ⚠️ ask_user is opt-in: spec.execution.askUser 🐛 ask_user is opt-in: spec.execution.askUser Sep 17, 2026

@djzager djzager left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the opt-in path through workflow stages, AgentRuns, params.json, and harness configuration. No issues found.

@ibolton336
ibolton336 merged commit 3804647 into konveyor:main Sep 18, 2026
27 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants