Reddit search overly senstive to query parameters

[yewsiang]

When clicking on the link suggested within this Reddit post, Crowdwise does not suggest the original Reddit link while on the PubMed page. This seems to be due to "?dopt=Abstract" being removed.

To investigate further.

[frizensami]

Reproducible on my end.

Separately, we'd expect the reddit post to be at least the first most relevant result. The reddit post is recommended under the title search but is not the first result under the default sort (highest comment count).

We may want to consider the default sort to be by "most relevant"

[frizensami]

This is a tough problem to solve. Right now, we remove tracking parameters (also the fragment part of URLs) to normalize the URL (e.g., no gclid, etc), because those cases are clear.

However, each website will have its own unique usage of URL parameters (like ?dOpt=abstract being unnecessary).

Some ways out:

• Include exact title matches (or almost close to) as exact match as well
• and/or: have a "special cases" code section for all of these situations. This might get out of hand though, and each check will slow us down ever so slightly.