Skip to content

fix(s3): reject responses with incomplete tags - #212

Open
Iams4kura wants to merge 1 commit into
simular-ai:mainfrom
Iams4kura:bugfix/reject-incomplete-thought-tags-20260821t160739z
Open

fix(s3): reject responses with incomplete tags#212
Iams4kura wants to merge 1 commit into
simular-ai:mainfrom
Iams4kura:bugfix/reject-incomplete-thought-tags-20260821t160739z

Conversation

@Iams4kura

Copy link
Copy Markdown

Summary

  • Fixes: BehaviorNarrator accepts plain text or a response missing either the thoughts or answer tag pair, so call_llm_formatted skips its corrective retry and malformed caption data reaches bBoN trajectory comparison.
  • Root cause: split_thinking_response uses the last split segment when a tag is absent, which makes the full untagged response look like parsed thoughts and answer text; thoughts_answer_tag_check only checked that the parsed thoughts value was nonempty instead of verifying that all four tags exist in order.

Regression evidence

  • Before: UV_CACHE_DIR=/tmp/agent-s-python311.20260821T160739Z/cache uv run --python 3.11 --no-project --with pillow -- python -m unittest discover -s tests -p 'test_formatters.py' -v exited 1

  • After: UV_CACHE_DIR=/tmp/agent-s-python311.20260821T160739Z/cache uv run --python 3.11 --no-project --with pillow -- python -m unittest discover -s tests -p 'test_formatters.py' -v exited 0

Verification

  • UV_CACHE_DIR=/tmp/agent-s-python311.20260821T160739Z/cache uv run --python 3.11 --no-project --with pillow --with numpy --with backoff --with openai --with anthropic -- python -m unittest discover -s tests -v
  • UV_CACHE_DIR=/tmp/agent-s-python311.20260821T160739Z/cache uv run --python 3.11 --no-project --with black -- black --check gui_agents tests/test_formatters.py

Scope

  • 2 files changed, +39 / -1 lines

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant