[Bug] --max-output-tokens is ignored by /v1/chat/completions and /v1/messages (only /v1/responses honours it)

[salekseev]

Summary

--max-output-tokens is documented as "Default max output tokens for requests that omit one
(default 32k)", but that default is applied only by the Responses API. On
/v1/chat/completions and /v1/messages a request that omits max_tokens gets the
hardcoded DEFAULT_MAX_OUTPUT_TOKENS = 32768 instead of the configured value, so the flag
silently does nothing on the two OpenAI/Anthropic-compatible paths.

On a server with --max-running-requests 1 this is not just a cosmetic gap: one runaway
generation occupies the single slot for its full 32k budget and every queued request waits
behind it.

Where

resolve_sampling hardcodes the constant and never sees the server config
(freetoken/server/generation.py:398,426):

DEFAULT_MAX_OUTPUT_TOKENS = 32768
...
max_tokens=DEFAULT_MAX_OUTPUT_TOKENS if max_tokens is None else max_tokens,

Callers:

path	passes the configured default?
freetoken/server/responses_api.py:153	yes — default_max = getattr(state.config, "max_output_tokens", None) or DEFAULT_MAX_OUTPUT_TOKENS
freetoken/server/openai_api.py:71,521	no — max_tokens=req.max_tokens straight through
freetoken/server/anthropic_api.py:312	no (the file has no reference to max_output_tokens at all)

/v1/messages matters less in practice because max_tokens is required by that API, so
clients always send one. /v1/chat/completions treats it as optional, so this is where it
bites.

Repro

Server started with --max-output-tokens 4096. ignore_eos is used only to guarantee the
generation would otherwise exceed the limit, so the test is deterministic rather than
dependent on the model choosing to ramble:

ft serve --model <any> --max-output-tokens 4096 --max-running-requests 1 ...

# max_tokens OMITTED -> should fall back to the configured 4096
curl -s localhost:1919/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "model": "<served-name>",
  "messages": [{"role": "user", "content": "Say hello."}],
  "ignore_eos": true
}' | jq '{completion_tokens: .usage.completion_tokens, finish_reason: .choices[0].finish_reason}'

Expected 4096. Actual:

{ "completion_tokens": 32767, "finish_reason": "length" }

Controls on the same server, same endpoint:

request	completion_tokens	finish_reason
ignore_eos: true, no max_tokens	32767	length
explicit "max_tokens": 4096	4095	length
no max_tokens, model stops on its own	1327	stop

So an explicit max_tokens is honoured and the engine enforces limits correctly — only the
configured default is not wired into this path. (32767 rather than 32768 is what both this
repro and the benchmark runs below report.)

Observed impact

Found while benchmarking, not synthetically. BFCL v4 parallel (200 entries, 4 client
threads, --max-running-requests 1, --max-output-tokens 4096); BFCL's OpenAI handler does
not send max_tokens. Across two 200-entry runs of the same model family, 3 of 400 entries
emitted 32767 completion tokens:

run	entry	completion tokens	latency
A	parallel_71	32767	550.7 s
A	parallel_122	32767	586.6 s
B	parallel_183	32767	530.6 s

Consequences with a single running slot:

• Head-of-line blocking. In run B the three entries queued behind parallel_183
  completed in 525.5 s / 519.8 s / 512.5 s while emitting only 450 / 598 / 860 tokens each —
  ~8.5 minutes of wall clock spent waiting on one runaway.
• Accuracy loss. Run A's parallel_122 runaway is one of that run's 13 scored failures.
• Median completion length in these runs is 858 tokens, so 32767 is ~38x the typical
  request; the configured 4096 would have been ample.

Suggested fix

Thread the configured default into resolve_sampling the way responses_api.py already
does — e.g. give it a default_max_tokens: int = DEFAULT_MAX_OUTPUT_TOKENS parameter and
pass state.config.max_output_tokens from the OpenAI and Anthropic paths. That keeps the
constant as the fallback and makes the flag mean what its help text says on every API.

Happy to send a PR if the shape above is what you would want.

Environment

FreeToken 0.1.2+gaf71ba432 (nightly wheel from the beta channel, commit af71ba432);
torch 2.11.0+cu130, CUDA 13.3; RTX 4080 SUPER 16 GB (sm_89); Qwen3.6-35B-A3B 4-bit,
--moe-backend offload --moe-cache-auto --max-running-requests 1 --max-output-tokens 4096 --sampling-defaults model --reasoning-parser qwen3 --tool-call-parser qwen3_coder.
Local patches present but unrelated (routed-expert int4 repack paths); the three files above
are unmodified from upstream.

[gdevenyi]

Confirmed on a current tree, and implemented — thank you for the precise pointer, resolve_sampling is exactly where it goes.

The asymmetry is as you describe: responses_api resolves getattr(state.config, "max_output_tokens", None) or DEFAULT_MAX_OUTPUT_TOKENS and passes the result in, so it never reaches the constant, while the OpenAI and Anthropic converters pass the request's own max_tokens straight through and resolve_sampling substitutes the hardcoded 32768 when that is None.

What I did, following the idiom already in responses_api rather than inventing another:

• resolve_sampling grows default_max_tokens: int | None = None and uses default_max_tokens or DEFAULT_MAX_OUTPUT_TOKENS for the None case, so behaviour is unchanged when nothing is configured.
• chat_request_to_genspec, the completions _resolve_sampling, and convert_anthropic_to_genspec thread it through; the handlers read it from state.config, which they already hold.
• An explicit per-request limit still wins, and the existing max_tokens < 1 client error is untouched.

Tests in tests/server/test_default_max_output_tokens.py cover the configured default filling an omitted limit, an explicit limit still winning, the unconfigured fallback staying at the documented constant, non-positive values still being a 400, and the two converters actually threading the value (that last one is what would have caught this bug originally — the unit under test was fine, the wiring was not).

Running on a deploy branch of the fork; happy to open it as a PR against main if that is useful, though it is small enough that a maintainer may prefer to just take the shape.

One note for anyone hitting this: the practical bite scales with --max-running-requests. At 1 it is exactly what you describe. At 8, a runaway still burns a slot for 32k tokens, it just takes seven more to notice.

🤖 Generated with Claude Code

https://claude.ai/code/session_0173pf9k9fSVtwbm3f898HDt