Summary
When an agent run yields no usable answer, PraisonAI collapses several distinct terminal causes — a provider content-filter block, a safety refusal, a finish_reason: "length" truncation, or a genuinely empty completion — into one indistinguishable falsy result (or a generic failed). For CLI-first and CI workflows this produces the classic "the session just went quiet" failure: the caller cannot tell why nothing came back, cannot branch on it, and cannot surface an actionable message to the user.
Core already models a rich set of terminal reasons and a per-run stop reason, but neither inspects the provider's finish_reason, so provider-side blocks/refusals/truncations are invisible.
Current behaviour
- The public outcome taxonomy has no reason for a provider block/refusal/truncation:
src/praisonai-agents/praisonaiagents/agent/run_outcome.py — TerminalReason = Literal["completed", "hard_timeout", "cancelled", "aborted", "failed"].
- The per-run stop reason only ever records completion or the step cap:
src/praisonai-agents/praisonaiagents/llm/llm.py sets self._last_stop_reason to "completed" or "max_steps" only (e.g. llm.py:486, :2708, :2941).
- The provider
finish_reason is never inspected for a filter/refusal/length signal: a search for content_filter / refusal / finish_reason-based branching in llm/llm.py returns no handling.
- The public
run/start surface then collapses distinct failures to a falsy value:
src/praisonai-code/praisonai_code/cli/commands/run.py _run_succeeded() documents that "The public Agent.start/run surface collapses failures (swallowed LLM/auth error, guardrail block, tool failure, max_iter without completion) to a falsy result … while a real answer is a non-empty string."
So a blocked/refused/truncated turn reaches the CLI either as completed with empty/partial text, or as generic failed, with no distinguishing reason.
Desired behaviour
- Core inspects the provider
finish_reason/refusal signal and records a distinct, additive terminal reason, e.g. content_filtered, refused, length_truncated (names TBD), on RunOutcome.reason and _last_stop_reason — alongside the existing max_steps precedent.
- These reasons are actionable end-to-end:
praisonai run maps them to a clear, non-zero exit outcome and a human-readable message (not a silent empty success), and includes the reason in --output json.return_outcome=True callers (Python) receive the specific reason.
- Backward-compatible and additive: existing
completed|failed|... semantics are unchanged; unknown/absent finish reasons behave exactly as today; zero overhead on the success path.
Layer placement
- Primary layer: core (
praisonaiagents)
- Why not core → n/a (this is core). Detecting the provider's
finish_reason/refusal and mapping it to a terminal reason must happen where the response is parsed (llm/llm.py) and where the outcome is modelled (agent/run_outcome.py).
- Why not wrapper: the wrapper cannot recover a reason the core never captured; it can only render what core reports. Wrapper is a secondary touch (exit code +
--output json field + message).
- Why not tools: this is response interpretation and run lifecycle, not an agent-callable integration.
- Why not plugins: although a plugin could observe an
ON_ERROR-style hook, the canonical terminal-reason taxonomy is core runtime state that the public run/start contract and CLI exit codes depend on; it is not an optional lifecycle add-on.
- Secondary touch (optional): wrapper (
praisonai-code) — surface the reason in run exit handling and --output json.
- 3-way surface (CLI + YAML + Python): yes (Python via
RunOutcome.reason; CLI via exit/--output json; YAML runs inherit the same outcome semantics)
Proposed approach
- In
llm/llm.py, read the provider finish_reason (and any explicit refusal field) on both the custom and OpenAI-native response paths; classify content-filter / refusal / length cutoffs.
- Record the classification on
self._last_stop_reason (extending the existing completed/max_steps set) and thread it into RunOutcome via the existing _run_with_outcome/_astart_with_outcome path.
- Add the new reasons to
run_outcome.py:TerminalReason with a defined precedence relative to failed (a specific block/refusal reason should win over generic failed, mirroring the existing sticky-precedence design in run_outcome.py).
- In the wrapper, map the new reasons to a clear exit outcome and message, and include
reason in --output json.
Resolution sketch
agent/run_outcome.py: extend TerminalReason and the precedence map additively.llm/llm.py: add a _classify_finish_reason(...) helper; set _last_stop_reason accordingly at the points that currently set "completed".run.py: in _run_succeeded/outcome handling, distinguish the new reasons and emit a specific message + non-zero exit; add the reason to the JSON output shape.- Tests: a real agentic test that a prompt provoking a provider refusal/length cutoff yields the specific reason rather than an empty
completed.
Severity
Medium — a correctness/observability gap that turns provider-side blocks into silent, un-actionable "empty" runs; particularly impactful for unattended/CI usage where exit codes and machine-readable reasons drive downstream behaviour.
Validation
run_outcome.py:TerminalReason lists only completed|hard_timeout|cancelled|aborted|failed — no block/refusal/truncation reason.llm.py sets _last_stop_reason only to completed/max_steps, and performs no finish_reason-based content-filter/refusal handling.run.py:_run_succeeded documents the collapse of distinct failures to a falsy result.- The existing
max_steps stop reason and the RunOutcome precedence map are the precedents to extend.
Summary
When an agent run yields no usable answer, PraisonAI collapses several distinct terminal causes — a provider content-filter block, a safety refusal, a
finish_reason: "length"truncation, or a genuinely empty completion — into one indistinguishable falsy result (or a genericfailed). For CLI-first and CI workflows this produces the classic "the session just went quiet" failure: the caller cannot tell why nothing came back, cannot branch on it, and cannot surface an actionable message to the user.Core already models a rich set of terminal reasons and a per-run stop reason, but neither inspects the provider's
finish_reason, so provider-side blocks/refusals/truncations are invisible.Current behaviour
src/praisonai-agents/praisonaiagents/agent/run_outcome.py—TerminalReason = Literal["completed", "hard_timeout", "cancelled", "aborted", "failed"].src/praisonai-agents/praisonaiagents/llm/llm.pysetsself._last_stop_reasonto"completed"or"max_steps"only (e.g.llm.py:486,:2708,:2941).finish_reasonis never inspected for a filter/refusal/length signal: a search forcontent_filter/refusal/finish_reason-based branching inllm/llm.pyreturns no handling.run/startsurface then collapses distinct failures to a falsy value:src/praisonai-code/praisonai_code/cli/commands/run.py_run_succeeded()documents that "The publicAgent.start/runsurface collapses failures (swallowed LLM/auth error, guardrail block, tool failure,max_iterwithout completion) to a falsy result … while a real answer is a non-empty string."So a blocked/refused/truncated turn reaches the CLI either as
completedwith empty/partial text, or as genericfailed, with no distinguishing reason.Desired behaviour
finish_reason/refusal signal and records a distinct, additive terminal reason, e.g.content_filtered,refused,length_truncated(names TBD), onRunOutcome.reasonand_last_stop_reason— alongside the existingmax_stepsprecedent.praisonai runmaps them to a clear, non-zero exit outcome and a human-readable message (not a silent empty success), and includes the reason in--output json.return_outcome=Truecallers (Python) receive the specific reason.completed|failed|...semantics are unchanged; unknown/absent finish reasons behave exactly as today; zero overhead on the success path.Layer placement
praisonaiagents)finish_reason/refusal and mapping it to a terminal reason must happen where the response is parsed (llm/llm.py) and where the outcome is modelled (agent/run_outcome.py).--output jsonfield + message).ON_ERROR-style hook, the canonical terminal-reason taxonomy is core runtime state that the publicrun/startcontract and CLI exit codes depend on; it is not an optional lifecycle add-on.praisonai-code) — surface the reason inrunexit handling and--output json.RunOutcome.reason; CLI via exit/--output json; YAML runs inherit the same outcome semantics)Proposed approach
llm/llm.py, read the providerfinish_reason(and any explicit refusal field) on both the custom and OpenAI-native response paths; classify content-filter / refusal / length cutoffs.self._last_stop_reason(extending the existingcompleted/max_stepsset) and thread it intoRunOutcomevia the existing_run_with_outcome/_astart_with_outcomepath.run_outcome.py:TerminalReasonwith a defined precedence relative tofailed(a specific block/refusal reason should win over genericfailed, mirroring the existing sticky-precedence design inrun_outcome.py).reasonin--output json.Resolution sketch
agent/run_outcome.py: extendTerminalReasonand the precedence map additively.llm/llm.py: add a_classify_finish_reason(...)helper; set_last_stop_reasonaccordingly at the points that currently set"completed".run.py: in_run_succeeded/outcome handling, distinguish the new reasons and emit a specific message + non-zero exit; add the reason to the JSON output shape.completed.Severity
Medium — a correctness/observability gap that turns provider-side blocks into silent, un-actionable "empty" runs; particularly impactful for unattended/CI usage where exit codes and machine-readable reasons drive downstream behaviour.
Validation
run_outcome.py:TerminalReasonlists onlycompleted|hard_timeout|cancelled|aborted|failed— no block/refusal/truncation reason.llm.pysets_last_stop_reasononly tocompleted/max_steps, and performs nofinish_reason-based content-filter/refusal handling.run.py:_run_succeededdocuments the collapse of distinct failures to a falsy result.max_stepsstop reason and theRunOutcomeprecedence map are the precedents to extend.