Skip to content

unreliable/broken tool calling & unescaped string args in chat_template.jinja #130

Description

@mpetruc

This might be related to #73.

The stock LFM2.5-8B-A1B chat_template.jinja (https://huggingface.co/LiquidAI/LFM2.5-8B-A1B/blob/main/chat_template.jinja) never tells the model what format to emit a tool call in. The system block it builds is only:

Today's date: ... \n\n List of tools: <JSON>

It never mentions <|tool_call_start|> / <|tool_call_end|> nor shows the pythonic list. On a fresh session (no prior assistant tool-call turn to
imitate), the model sometimes drifts to an OpenAI-style JSON object instead of the pythonic call, so the tool call is silently lost or the turn ends
with no structured output.

Repro: chat request with tools + "explore the repo, then call a tool", temperature ~0.8, empty history. The model sometimes emits something like

...reasoning... {"TaskCreate": {...}, "activeForm": ...}

instead of:

<|tool_call_start|>[TaskCreate(subject='...', description='...')]<|tool_call_end|>

Observed from vLLM and tabbyAPI (both load this same template). Severity is lowest on vLLM, the reference path where the sentinels are registered as special tokens; highest on providers that match the sentinels as plain text.

Second, independent bug: format_arg_value wraps string args in single quotes without escaping, so a value like "It's" renders the malformed

[send_message(text='It's fine')]

into the assistant tool-call history.

Suggested fix (applies cleanly; both changes validated with jinja2):

--- a/chat_template.jinja
+++ b/chat_template.jinja
@@ format_arg_value (escape quotes/backslashes before wrapping):
-        {{- "'" + arg_value + "'" -}}
+        {{- "'" + (arg_value | replace("\\", "\\\\") | replace("'", "\\'")) + "'" -}}
@@ tools system block (append a format directive after the tool list):
-    " + (tools | tojson) -%}
+    " + (tools | tojson)
+    + "\n\nTo call a tool, output EXACTLY one line wrapped in "
+      "<|tool_call_start|> and <|tool_call_end|>: "
+      "[function_name(arg='value', arg2=123)]. Never emit JSON. "
+      "If no tool is needed, answer in prose." -%}

I validated this change with LFM2.5-8B in hermes and pi. With both harnesses the model generated long chains of tool calls, thinking blocks and responses. It's working great!

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions