Skip to content

fix(ai): only capture terminal Responses statuses as stop reasons - #4736

Open
bernatixer wants to merge 3 commits into
mainfrom
posthog/ai-responses-terminal-stop-reason
Open

fix(ai): only capture terminal Responses statuses as stop reasons#4736
bernatixer wants to merge 3 commits into
mainfrom
posthog/ai-responses-terminal-stop-reason

Conversation

@bernatixer

@bernatixer bernatixer commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Problem

  • A LangChain Responses run started with background: true returns while still queued, and the callback records $ai_stop_reason: 'queued' - a lifecycle state, not a stop reason.
  • The same truncated run gets two spellings: the LangChain callback reports max_output_tokens, the native OpenAI wrapper reports incomplete. The PostHog trace view only recognizes the first, so native-wrapper truncations render as normal endings.
  • A comment in the callback claims the two surfaces already match.

Follow-up to #4700, from its post-merge review.

Changes

  • Runs that OpenAI truncated or ended abnormally now carry the same $ai_stop_reason from both the LangChain callback and the native wrapper, and queued or in-progress background runs carry none.
  • One shared responsesStopReason helper does the mapping: non-terminal statuses yield nothing, an incomplete run is named by incomplete_details.reason (e.g. max_output_tokens), the other terminal statuses stand for themselves.
  • The LangChain callback, the native telemetry builder, and the stream accumulator route through the helper.
  • Mechanical: the callback comment now states what the code does; changeset added.

Note

Behavior change: the native wrapper reports a truncated run as max_output_tokens instead of incomplete. Dashboards grouping on $ai_stop_reason see the new value after this ships. Cancelled runs stay cancelled.

Release info Sub-libraries affected

Libraries affected

  • @posthog/ai

Checklist

  • Tests for new code
  • Accounted for the impact of any changes across different platforms
  • Accounted for backwards compatibility of any changes (no breaking changes!)
  • Took care not to unnecessarily increase the bundle size

If releasing new changes

  • Ran pnpm changeset to generate a changeset file

Testing

  • New LangChain row: status: 'queued' produces no stop reason - fails without the terminal gate.
  • The completed-run row now carries incomplete_details: null, the shape every real completed run has.
  • The native openai, azure-openai, openai-stream-accumulators, and openai-background-responses suites expect max_output_tokens for the incomplete status.
  • The streaming fallback response object drops its status field: stopReason is set only alongside terminalResponse now, so the old cast could never hold a real status.
  • Not run: the integration suites, because they need provider API keys.

🤖 Agent context

Autonomy: Human-driven (agent-assisted)

Written by Claude Code from the confirmed findings of a max-level review of #4700: the missing terminal-status gate, the stop-reason vocabulary split between the two wrappers, and the stale comment. The remaining review findings (streamed truncations on @langchain/openai <=1.4.6, untested generation-level metadata legs) were left out as lower priority. Skills invoked: code-review, writing-pr-descriptions.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@bernatixer bernatixer self-assigned this Sep 1, 2026
bernatixer and others added 2 commits September 2, 2026 00:46
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@bernatixer
bernatixer marked this pull request as ready for review September 2, 2026 07:38
@bernatixer
bernatixer requested a review from a team as a code owner September 2, 2026 07:38
@marandaneto
marandaneto requested a review from a team September 2, 2026 08:03
@marandaneto

Copy link
Copy Markdown
Member

@bernatixer make sure to tag @PostHog/team-ai-observability for reviewer as well
the client libs team can check and approve changes basically looking at raw code/quality/etc, but since we dont know deeply about all products built in the sdk, I can't always tell if this change makes sense or not product-wise

// by what cut it short, and only terminal statuses count as stop reasons.
// Shared with the native OpenAI Responses wrapper.
responsesStopReason(messageResponseMetadata) ||
responsesStopReason(generationResponseMetadata)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This can return incomplete before checking the reason in generationResponseMetadata. Is that fine? Could the reason be max_output_tokens?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants