From 80e9e9994191d87d68c0462fea84020297593a2a Mon Sep 17 00:00:00 2001 From: Eric Proulx Date: Wed, 29 Jul 2026 22:59:04 +0200 Subject: [PATCH] Tag path params as UTF-8 instead of leaving them binary MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Mustermann decodes path captures out of PATH_INFO, which Rack hands over tagged ASCII-8BIT, and nothing re-tagged the result. Query and body params arrive UTF-8 because Rack tags those itself, so the same value reached the endpoint with a different encoding depending on where it came from. That made an API's declarations disagree with themselves -- a binary string never equals the UTF-8 literal it was written as: params { requires :id, type: String, values: ['café'] } GET /?id=café -> 200 GET /café -> 400 "id does not have a valid value" The same held for same_as, except_values and any comparison an endpoint made against a non-ASCII literal. It also leaked into serialization: a non-ASCII path param rendered into a JSON response drew an encoding warning from the json gem, which that gem says will become an error in json 3.0. Re-tag in Route#params_for, the single funnel for path-extracted values. Nothing obliges a client to send UTF-8 -- HTTP treats the request target as octets, and Rack's SPEC has CGI keys carry non-ASCII as ASCII-8BIT -- so UTF-8 is the convention rather than a guarantee: it is what browsers percent-encode with, what an IRI maps to, and what Rails settles on (ActionDispatch::Journey::Router force_encodes every path capture to UTF-8 after unescaping it). Only the encoding changes here: the bytes are untouched, so octets that are not UTF-8 stay invalid and are still caught downstream instead of being silently scrubbed into something the client never sent. Unnamed splats capture into an Array, so those are walked too. No extra allocations -- the captures are freshly built and unfrozen, so the re-tag happens in place. Co-Authored-By: Claude Opus 5 --- CHANGELOG.md | 2 ++ UPGRADING.md | 39 ++++++++++++++++++++++++ lib/grape/router/route.rb | 29 +++++++++++++++++- spec/grape/api_spec.rb | 62 +++++++++++++++++++++++++++++++++++++++ 4 files changed, 131 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 0049465b9..b541b2a86 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -55,6 +55,8 @@ * [#2827](https://github.com/ruby-grape/grape/pull/2827): Make the `cascade` DSL getter return the configured value (`cascade false` read back as `true`) - [@ericproulx](https://github.com/ericproulx). * [#2829](https://github.com/ruby-grape/grape/pull/2829): Fix a cascading route handing over only to the last route registered for the path, making a middle version (3+ mounted versions with a catch-all) answer 406 - [@ericproulx](https://github.com/ericproulx). * [#2826](https://github.com/ruby-grape/grape/pull/2826): Fix `api.version` not being set for the root route of a path-versioned API (`GET /v1`) - [@ericproulx](https://github.com/ericproulx). +* [#2834](https://github.com/ruby-grape/grape/pull/2834): Restore the #2824 fix for cascaded routes leaking `route_info` and path captures, silently reverted by #2829 - [@ericproulx](https://github.com/ericproulx). +* [#2839](https://github.com/ruby-grape/grape/pull/2839): Tag path params as UTF-8 instead of leaving them ASCII-8BIT, so they compare equal to the non-ASCII literals an API declares (see UPGRADING) - [@ericproulx](https://github.com/ericproulx). * Your contribution here. ### 3.3.5 (2026-07-30) diff --git a/UPGRADING.md b/UPGRADING.md index ab4838229..248a34c83 100644 --- a/UPGRADING.md +++ b/UPGRADING.md @@ -3,6 +3,45 @@ Upgrading Grape ### Upgrading to >= 4.0.0 +#### Path params are tagged UTF-8 instead of ASCII-8BIT + +Params captured from the request path — `route_param`, `:id`-style segments, splats — now come back tagged `UTF-8`. They used to carry the `ASCII-8BIT` encoding of Rack's `PATH_INFO`, because Mustermann decodes the path against that raw string and nothing re-tagged the result. Query and body params were already `UTF-8`, since Rack tags those itself. + +**Why UTF-8.** Nothing obliges a client to send it — HTTP treats the request target as octets, and Rack's SPEC has CGI keys carry non-ASCII as `ASCII-8BIT`. UTF-8 is a convention rather than a guarantee. But it is the convention Rack itself already applies to everything *except* the path: `Rack::QueryParser#unescape` decodes the query string and form bodies with `URI.decode_www_form_component(string, Encoding::UTF_8)`, which tags the result without validating it. + +One request carrying the same invalid octets in three places, before this change: + +| source | encoding | `valid_encoding?` | +| --- | --- | --- | +| query — `?q=%C3%28` | `UTF-8` | `false` | +| form body — `form=%C3%28` | `UTF-8` | `false` | +| path — `/%C3%28` | **`ASCII-8BIT`** | `true` | + +The bytes are equally malformed in all three; only the label differed. The path reads `true` merely because `ASCII-8BIT` considers every byte sequence valid. So this change is not Grape adopting an outside convention — it is Grape agreeing with the library handing it the request. The path param was the odd one out only because Mustermann decodes against `PATH_INFO` directly and never had Rack's `unescape` applied to it. + +Grape does exactly what Rack does: re-tag, do not validate. The bytes are untouched, so octets that are *not* UTF-8 stay detectably invalid rather than being scrubbed into something the client never sent. (Rails takes the same approach for path captures — `ActionDispatch::Journey::Router` force-encodes each one to UTF-8 after unescaping.) + +**What this fixes.** An API's own declarations used to disagree with themselves depending on where a value arrived from — a binary string never equals the UTF-8 literal it was written as: + +```ruby +params { requires :id, type: String, values: ['café'] } +``` + +| request | before | 4.0 | +| --- | --- | --- | +| `GET /?id=café` (query) | `200` | unchanged | +| `GET /café` (path) | `400 "id does not have a valid value"` | `200` | + +The same held for `same_as`, `except_values`, and any comparison an endpoint made against a non-ASCII literal. Serialization was affected too: a non-ASCII path param rendered into a JSON response emitted an encoding warning from the `json` gem, and is slated to raise there in json 3.0. + +**What can break.** Only the encoding tag changes; the bytes are untouched, and an invalid byte sequence stays invalid rather than being scrubbed. Comparisons against pure-ASCII strings are unaffected. Code that relied on a path param being binary — concatenating one with genuinely binary data, for instance — can now raise `Encoding::CompatibilityError`, and should call `.b` on the param to opt back into binary: + +```ruby +params[:id].b + binary_blob +``` + +An application that worked around the old behavior with its own `force_encoding(Encoding::UTF_8)` needs no change; that call is now a no-op. + #### `Array`/`Set` of an unsupported type is rejected when the API is defined Declaring a collection whose element type Grape cannot coerce — `type: Array[Foo]` or `type: Set[Foo]` where `Foo` is neither a primitive, a structure, nor a valid custom type — now raises as soon as the `params` block is evaluated, i.e. while the API class is being loaded: diff --git a/lib/grape/router/route.rb b/lib/grape/router/route.rb index 7afa40bc1..4f3e5e55b 100644 --- a/lib/grape/router/route.rb +++ b/lib/grape/router/route.rb @@ -49,7 +49,7 @@ def params_for(input) parsed = pattern.params(input) return unless parsed - parsed.compact.symbolize_keys + parsed.compact.symbolize_keys.transform_values! { |value| tag_utf8(value) } end protected @@ -60,6 +60,33 @@ def convert_to_head_request! private + # Mustermann decodes path captures out of +PATH_INFO+, which Rack hands us + # tagged ASCII-8BIT, so path params came back binary while Rack tags query + # and body params UTF-8. That split makes an API's own declarations + # disagree with themselves: `values: ['café']` matched `?id=café` but not + # `/café`, since a binary string never equals the UTF-8 literal it was + # written as. + # + # Re-tag as UTF-8. Nothing obliges a client to send UTF-8: the request + # target is octets to HTTP, and Rack's SPEC has CGI keys carry non-ASCII + # as ASCII-8BIT. But UTF-8 is what browsers percent-encode with, what an + # IRI maps to, and what Rails settles on — ActionDispatch::Journey::Router + # force_encodes every path capture to UTF-8 after unescaping it. + # + # Only the encoding changes; the bytes are untouched. Octets that are not + # UTF-8 therefore stay invalid and are caught downstream rather than being + # silently scrubbed into something the client never sent. + def tag_utf8(value) + # String first: every capture but a multi-splat is one. + if value.is_a?(String) + value.encoding == Encoding::UTF_8 ? value : (+value).force_encoding(Encoding::UTF_8) + elsif value.is_a?(Array) + value.map { |element| tag_utf8(element) } + else + value + end + end + def upcase_method(method) method_s = method.to_s Grape::HTTP_SUPPORTED_METHODS.detect { |m| m.casecmp(method_s).zero? } || method_s.upcase diff --git a/spec/grape/api_spec.rb b/spec/grape/api_spec.rb index ff3454f15..ba32313b3 100644 --- a/spec/grape/api_spec.rb +++ b/spec/grape/api_spec.rb @@ -309,6 +309,68 @@ expect(last_response.body).to eq('{"foo":1234}') end end + + context 'with a non-ascii segment' do + it 'tags the param as UTF-8 rather than leaving it binary' do + subject.route_param :id do + get { params[:id].encoding.name } + end + + get '/caf%C3%A9' + expect(last_response.body).to eq('UTF-8') + end + + it 'tags every captured segment' do + subject.namespace :a do + route_param :one do + route_param :two do + get { [params[:one].encoding.name, params[:two].encoding.name].join(',') } + end + end + end + + get '/a/caf%C3%A9/th%C3%A9' + expect(last_response.body).to eq('UTF-8,UTF-8') + end + + it 'tags a splat capture' do + subject.get('/files/*path') { params[:path].encoding.name } + + get '/files/a/b/caf%C3%A9' + expect(last_response.body).to eq('UTF-8') + end + + # An unnamed splat captures into an Array rather than a String. + it 'tags every element of an unnamed splat capture' do + subject.get('/files/*') { params[:splat].map { |s| s.encoding.name }.join(',') } + + get '/files/caf%C3%A9' + expect(last_response.body).to eq('UTF-8') + end + + # A path param used to arrive binary while the same value arrived UTF-8 + # through the query string, so an API's own declaration disagreed with + # itself depending on where the value came from. + it 'compares equal to the utf-8 literal the API declares' do + subject.params { requires :id, type: String, values: ['café'] } + subject.route_param :id do + get { 'matched' } + end + + get '/caf%C3%A9' + expect(last_response.status).to eq(200) + expect(last_response.body).to eq('matched') + end + + it 'leaves the bytes untouched' do + subject.route_param :id do + get { params[:id] } + end + + get '/caf%C3%A9' + expect(last_response.body.b).to eq('caf%C3%A9'.b.gsub('%C3%A9', "\xC3\xA9".b)) + end + end end describe '.route' do