Skip to content

Add new 05/11 IFM tool-call formats for K2 V3 parser#5

Open
hanseungwook wants to merge 5 commits into
tgat/multi-format-tool-parser-v0.12.0from
v0.12.0-ifm-0511
Open

Add new 05/11 IFM tool-call formats for K2 V3 parser#5
hanseungwook wants to merge 5 commits into
tgat/multi-format-tool-parser-v0.12.0from
v0.12.0-ifm-0511

Conversation

@hanseungwook
Copy link
Copy Markdown
Collaborator

What changed

This updates the K2 V3 multi-format tool parser to understand the new IFM tool-call formats from the BBQ
0511 chat template.

The parser now supports these exact tool-call formats:

  • json
  • xml
  • xml_typed

It still supports the existing older formats:

  • qwen3
  • minimax
  • dsv32
  • glm
  • gptoss
  • python

The old default format is no longer accepted because it was confusing. If no format is provided, the
parser now uses xml.

Type handling

For all three new IFM formats, parsed tool arguments are converted using the provided tools schema.

Examples:

  • If the schema says an argument is a string, 12345 becomes "12345".
  • If the schema says an argument is a boolean, "true" becomes true.
  • If the schema says an argument is an integer, "2" becomes 2.
  • Objects and arrays are parsed from JSON text when possible.

For xml_typed, the schema wins. The emitted <ifm|arg_type> is only used when the schema does not
provide a type.

How to run tests

python -m pytest -q tests/entrypoints/openai/tool_parsers/test_multi_format_tool_parser.py

Current result:

29 passed, 2 warnings

## How to run a server

Use the K2 V3 reasoning parser and the K2 V3 tool parser:

PYTHONPATH=/mnt/weka/home/seungwook.han/vllm \
vllm serve "$MODEL_PATH" \
  --served-model-name "$SERVED_MODEL_NAME" \
  --tensor-parallel-size 8 \
  --reasoning-parser k2_v3 \
  --enable-auto-tool-choice \
  --tool-call-parser k2_v3 \
  --chat-template /mnt/weka/home/seungwook.han/bbq-chat-template/bbq-0511/chat_template.jinja \
  --trust-remote-code

Requests can select the tool-call format with:

"chat_template_kwargs": {
  "tool_call_format": "xml"
}

Supported exact values are json, xml, and xml_typed (including the old multi format ones, but `default` has been removed b/c it causes confusion)

The new default when no `kwarg` is provided is `xml` (following our chat template)

Copy link
Copy Markdown
Collaborator

@mylibrar mylibrar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The current PR does not contain the xllm patching in #4 yet. Will we merge tgat/multi-format-tool-parser-v0.12.0 into v0.12.0-ifm untimately?

@hanseungwook
Copy link
Copy Markdown
Collaborator Author

@mylibrar I can bring in that PR into this branch -- I wanted to keep the branches separate actually

@hanseungwook
Copy link
Copy Markdown
Collaborator Author

@mylibrar brought in PR #4 into this branch

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants