fix(tokenizers): parse Jinja tuple literals in grouping parens (#409)

[jamesburton]

Summary

JinjaParser.ParsePrimary treated any (...) as plain grouping and required exactly one
expression, so a comma-separated tuple literal such as (a, b, c) failed to parse with
Expected RightParen, got Comma. This sits in the unconditional reasoning-instructions prelude
of the real Qwen/Qwen3.8-27B chat_template.jinja (line 48:
resolved_reasoning_effort not in ('xhigh', 'medium', 'low')), so constructing a
JinjaChatTemplate from the unmodified template always threw.

• ( now branches on whether a comma follows the first parsed expression:
  • (a) — plain grouping, returns a itself (NOT a tuple)
  • (a,) — one-element tuple (trailing comma is significant)
  • (a, b, c) / (a, b, c,) — tuple, trailing comma optional
  • () — empty tuple
• Tuples reuse the existing ListExpr node (same representation as [...] list literals)
  rather than a parallel type, since the evaluator already treats it as an ordered sequence and
  in/not in membership testing works against it unchanged.

Files changed

• src/DotLLM.Tokenizers/ChatTemplates/JinjaParser.cs — the fix (ParsePrimary's LeftParen case)
• tests/DotLLM.Tests.Unit/Tokenizers/ChatTemplates/JinjaParserTests.cs — discriminating parser
  tests: (a) vs (a,), multi-element tuples, trailing comma, empty tuple, in/not in
  true/false evaluation against a tuple, and the exact not in (...) construct from line 48 of
  the real template.
• tests/DotLLM.Tests.Unit/Tokenizers/ChatTemplates/Fixtures/qwen3.8-27b-chat-template.jinja —
  byte-for-byte copy of the real template fetched from
  https://huggingface.co/Qwen/Qwen3.8-27B/raw/main/chat_template.jinja.
• tests/DotLLM.Tests.Unit/Tokenizers/ChatTemplates/JinjaQwen3_8_27BAcceptanceTests.cs — the
  end-to-end acceptance test for fix(tokenizers): Jinja parser rejects parenthesized tuple literals (blocks Qwen3.8-27B template parse) #409.

Known limitation — PR #411 (#399) status

PR #411 (loop.previtem/loop.nextitem + is undefined support) is currently OPEN, not merged,
and this branch was cut from origin/dev before it landed, so it is not included here.

The acceptance test (JinjaQwen3_8_27BAcceptanceTests) therefore has three cases:

• FullTemplate_Parses_WithoutThrowing and FullTemplate_ConstructsAsJinjaChatTemplate_WithoutThrowing
  — assert the tuple fix alone is sufficient for the full template to parse. These pass today.
• FullTemplate_Render_KnownRemainingGapIsTracked — attempts a full render. Rendering still
  needs feat(tokenizers): Jinja loop.previtem / loop.nextitem — Qwen3.8 tool-response branch cannot render #399's is undefined test support (used unconditionally at line 46:
  enable_thinking is undefined or enable_thinking is true), so it currently throws. The test
  asserts the failure is specifically the known, tracked Unknown test: undefined error — not a
  tuple/comma parsing error — so a regression in this fix would still be caught. Once feat(tokenizers): Jinja loop.previtem / loop.nextitem #411 merges
  and this branch (or dev) picks it up, the try branch renders successfully and the catch
  branch is simply never reached (verified locally by temporarily patching the evaluator with the
  same shims feat(tokenizers): Jinja loop.previtem / loop.nextitem #411 is expected to add — is undefined and loop.previtem/loop.nextitem — after
  which the full template, including a tool-call/tool-response conversation, renders end-to-end
  with no further errors. No other unsupported constructs were found in the template beyond the
  known feat(tokenizers): Jinja loop.previtem / loop.nextitem — Qwen3.8 tool-response branch cannot render #399 gap, so no follow-up issue was needed.)

Test plan

• dotnet build -c Release — clean, 0 errors.
• dotnet test --filter "FullyQualifiedName~ChatTemplates" — 171 passed, 0 failed.
• dotnet test --filter "FullyQualifiedName~Tokenizers" — 373 passed, 12 skipped (pre-existing,
  unrelated fixture-file-dependent tests), 0 failed.
• Manually verified (via a temporary, unmerged local patch, reverted before this PR) that once
  feat(tokenizers): Jinja loop.previtem / loop.nextitem — Qwen3.8 tool-response branch cannot render #399's gap is closed, the full unmodified template renders successfully with tools and a
  tool-call/tool-response conversation, with no further parsing/evaluation errors.

Closes #409

[Copilot]

Pull request overview

This PR fixes the Jinja subset parser in DotLLM.Tokenizers to correctly accept parenthesized tuple literals (e.g. ('xhigh', 'medium', 'low')) while preserving normal grouping semantics for (expr). This unblocks parsing the unmodified Qwen3.8-27B chat_template.jinja, which uses not in (...) in an unconditional prelude.

Changes:

• Update JinjaParser.ParsePrimary to parse () / (a,) / (a, b, c[,]) as tuple literals (represented as ListExpr), while keeping (a) as plain grouping.
• Add parser + evaluator unit tests that discriminate (a) vs (a,), cover empty/multi-element tuples, trailing commas, and in/not in evaluation.
• Add an end-to-end unit acceptance test and a verbatim Qwen3.8-27B chat template fixture to ensure the real template at least parses successfully.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

File	Description
src/DotLLM.Tokenizers/ChatTemplates/JinjaParser.cs	Extends LeftParen primary parsing to support tuple-literal semantics while preserving grouping.
tests/DotLLM.Tests.Unit/Tokenizers/ChatTemplates/JinjaParserTests.cs	Adds discriminating tuple parsing/evaluation tests, including the real not in (...) construct.
tests/DotLLM.Tests.Unit/Tokenizers/ChatTemplates/Fixtures/qwen3.8-27b-chat-template.jinja	Adds the real Qwen3.8-27B template fixture used for regression/acceptance coverage.
tests/DotLLM.Tests.Unit/Tokenizers/ChatTemplates/JinjaQwen3_8_27BAcceptanceTests.cs	Adds end-to-end acceptance coverage ensuring the full template parses and tracks the known render-gap.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.