Gopher selectors with // or /../ or /./ are mangled

[abbrev]

I have selectors on my Gopher site that contain // or /../ or /./. Lagrange collapses these according to URL path syntax rules, but a selector is not a URL and should not be modified—it should be sent to the server exactly as it's presented to the client in a Gopher menu.

I made a client test for this issue at gopher://asciz.com/1/client-tests/menu ("Query string test 3"). The selector is /client-tests/query3/?http://example.com/a/../b/./c/, but Lagrange modifies it:

Expected request:

    "/client-tests/query3/?http://example.com/a/../b/./c/"

Actual request:

    "/client-tests/query3/?http:/example.com/b/c/"

(Lynx mangles it the same way.)

Here's a proposed fix (in the_Foundation) when converting a selector to a URL path, though I haven't tested it, and it might break some Gemini or other URLs outside of Gopher menus (especially ones that contain .. or . path segments):

diff --git a/src/string.c b/src/string.c
index 9644dea6b..a9f1eafa2 100644
--- a/src/string.c
+++ b/src/string.c
@@ -591,11 +591,22 @@ iString *maybeUrlEncodeExclude_String(const iString *d, const char *excluded) {
     /* TODO: Return NULL if nothing to encode. */
     iString *encoded = new_String();
     /* Note: Any UTF-8 code points are encoded as multiple %NN sequences. */
+    char prevch = '\0';
     for (const char *i = constBegin_String(d), *end = constEnd_String(d); i != end; ++i) {
         char ch = *i;
         if ((ch >= 'A' && ch <= 'Z') || (ch >= 'a' && ch <= 'z') || (ch >= '0' && ch <= '9') ||
             ch == '-' || ch == '_' || ch == '.' || ch == '~' || strchr(excluded, ch)) {
-            appendData_Block(&encoded->chars, i, 1);
+            if (prevch == '/' && ch == '/') {
+                /* Escape doubled slash to prevent collapsing into one slash. */
+                appendCStrN_String(encoded, "%2F", 3);
+            }
+            else if (prevch == '/' && ch == '.') {
+                /* Escape dot after slash to prevent collapsing /../ and /./ segments. */
+                appendCStrN_String(encoded, "%2E", 3);
+            }
+            else {
+                appendData_Block(&encoded->chars, i, 1);
+            }
         }
         else {
             static const char hex[16] = {
@@ -603,6 +614,7 @@ iString *maybeUrlEncodeExclude_String(const iString *d, const char *excluded) {
             char escaped[3] = {'%', hex[(ch >> 4) & 0xf], hex[ch & 0xf]};
             appendCStrN_String(encoded, escaped, 3);
         }
+        prevch = ch;
     }
     return encoded;
 }

Here's an alternative proposed fix which should work in all cases, though it's a bit overkill since it URL-encodes all slashes in a Gopher selector, which is unnecessary most of the time:

diff --git a/src/gopher.c b/src/gopher.c
index 4abad2f4c..300383022 100644
--- a/src/gopher.c
+++ b/src/gopher.c
@@ -171,7 +171,7 @@ static iBool convertSource_Gopher_(iGopher *d) {
                                   isEmpty_Range(&port) ? "70" : cstr_Rangecc(port),
                                   lineType,
                                   cstrCollect_String(
-                                      urlEncodeExclude_String(collectNewRange_String(selector), "/")),
+                                      urlEncode_String(collectNewRange_String(selector))),
                                   cstr_Rangecc(text));
                     append_Block(d->output, utf8_String(buf));
                     iEndCollect();

[abbrev]

Now that I've looked at URI syntax rules more closely (RFC 3986), a path may contain two (or more) slash characters in any position of the path except at the beginning. So http://example.com/foo//bar is valid but http://example.com//foo/bar is not. And a server may interpret a path with two slashes different than a path with one. Why are double (or more) slashes combined into one by Lagrange?

Edit: I'm correcting myself again. A URI is scheme ":" hier-part [ "?" query ] [ "#" fragment ]. hier-part can be "//" authority path-abempty. path-abempty is *( "/" segment ). And segment is *pchar. Putting that all together and you can have a valid URI of http://example.com//foo/bar after all, and it's considered a different URI from http://example.com/foo/bar. The restriction on double slash at the beginning is with relative references; this allows a double slash at the beginning of a relative reference to indicate the same scheme (e.g., http) but a different authority (e.g., domain name) as the referring resource.

Either way, the double slashes should not be condensed into one slash in any case. The dot and dot-dot segments are a different matter that should be handled only when converting a Gopher selector to a URL (percent-encoding is effective).

[skyjake]

IIRC, when Lagrange normalizes a URI path component, it cleans up the relative . and .. references and probably collapses consecutive slashes as well. I can certainly see why this is a problem for Gopher, where the path represents the selector and should not undergo such normalization.

I'll have to review the code and see if this warrants changes in general to the internal URI handling, or should the Gopher URIs specifically have more relaxed handling.