Tag path params as UTF-8 instead of leaving them binary

[ericproulx]

Summary

Mustermann decodes path captures out of PATH_INFO, which Rack hands over tagged ASCII-8BIT, and nothing re-tagged the result. Query and body params arrive UTF-8 because Rack tags those itself — so the same value reached the endpoint with a different encoding depending on where it came from.

That made an API's own declarations disagree with themselves, since a binary string never equals the UTF-8 literal it was written as:

params { requires :id, type: String, values: ['café'] }

request	before	after
GET /?id=café (query)	200	200
GET /café (path)	400 "id does not have a valid value"	200

The same held for same_as, except_values, and any comparison an endpoint made against a non-ASCII literal. It also leaked into serialization: a non-ASCII path param rendered into a JSON response drew warning: JSON.generate: UTF-8 string passed as BINARY, which the json gem says will become an error in json 3.0.

Approach

Re-tag in Route#params_for, the single funnel for path-extracted values.

Why UTF-8, given the path is octets. Nothing obliges a client to send UTF-8: HTTP treats the request target as octets, and Rack's SPEC has CGI keys carry non-ASCII as ASCII-8BIT — which is exactly why PATH_INFO arrives binary in the first place. UTF-8 is the convention rather than a guarantee: it is what browsers percent-encode with, what an IRI maps to, and what Rails settles on — ActionDispatch::Journey::Router force-encodes every path capture to UTF-8 after unescaping (action_dispatch/journey/router.rb:79, actionpack 8.1.3.1):

val = val.include?("%") ? CGI.unescapeURIComponent(val) : val
val.force_encoding(::Encoding::UTF_8)
path_parameters[name.to_sym] = val

Only the encoding changes — the bytes are untouched, so octets that are not UTF-8 stay detectably invalid and are still caught downstream rather than being silently scrubbed into something the client never sent. Unnamed splats capture into an Array ({"splat" => ["a", "b"]}), so those are walked too.

Backward compatibility

UPGRADING entry added. Comparisons against pure-ASCII strings are unaffected, which is the overwhelming majority of code. Code that relied on a path param being binary — concatenating one with genuinely binary data — can now raise Encoding::CompatibilityError and should call .b on the param. An app that already worked around this with its own force_encoding needs no change; that call becomes a no-op.

Perf

No extra allocations: the captures are freshly built and unfrozen, so +value returns self and the re-tag happens in place. Measured on params_for in isolation (two captures, 200 calls):

	allocations/call	i/s
before	12.0	735k
after	12.0	626k

~0.24 µs per matched request, in a method that is itself a small slice of the request. I tried folding the tagging into a single each_with_object pass to avoid the extra traversal; it was slower than keeping the C-optimized compact / symbolize_keys / transform_values! chain (1.45M vs 1.57M i/s on the hash transform alone), so the straightforward form stayed.

Test plan

• Six new examples in api_spec.rb covering single/multiple/splat/unnamed-splat captures, the values: equality case, and a byte-preservation invariant; verified the behavioural ones fail without the lib/ change.
• Full RSpec suite passes locally (2549 examples, 0 failures).
• RuboCop clean.
• CI green.

🤖 Generated with Claude Code

[github-actions]

Danger Report

No issues found.

View run