[dsmiley]

https://issues.apache.org/jira/browse/SOLR-3479

Description

Adds a [hl] document transformer that puts highlighted snippets inside each document, as an alternative to the highlight component's separate highlighting section. One instance highlights one field, so several can be listed in fl:

• fl=id,body_hl:[hl f=body] adds body_hl with snippets of body. The full body isn't returned unless fl also lists it.
• fl=id,body:[hl]: without f, the field is the transformer's name (its alias).
• If the name is the field and nothing matches, the field's original value is returned, so body:[hl] means "body, highlighted where possible". Otherwise a miss gives an empty list, or no value at all.
• A single-valued field with one snippet (the default) gets a plain value; otherwise a list.
• Local params are hl.* params without the prefix (e.g. snippets=3 tag.pre='<b>'), and they override the request's hl.* params, including per-field ones. The names are discovered from HighlightParams by reflection, so new params work automatically.

A multi-field form that returns a map (like today's highlighting section) could be added later; see the JIRA discussion.

Solution

• Built on the Unified Highlighter. A transformer sees one doc at a time, so query term extraction and per-field setup (FieldHighlighter) happen once per request in setContext, and each doc is then highlighted with FieldHighlighter.highlightFieldForDoc. Stored text and postings/term-vector offsets are used as usual.
• UnifiedSolrHighlighter.SolrExtendedUnifiedHighlighter is now public, with a constructor that doesn't need a SolrQueryRequest, so the transformer shares its hl.* handling. No behavior change for the highlight component.
• Works in /select (single-core and distributed, since transformers run on the shards) and in real-time get, where there's no query, so the original value passes through.
• Ref guide: a new section on the document transformers page, and an intro on the highlighting page describing both approaches and when to prefer each.

Found along the way, not fixed here: with hl.method=unified, multi-valued fields can return a leading empty snippet (e.g. [, forest <em>fox</em>]), because Lucene's value separator and Solr's snippet separator are both \u0000. The transformer filters empty snippets; the component deserves its own issue.

Developed with Claude Code (AI coding assistant), reviewed and directed by me.

Tests

New TestHighlightAugmenterFactory (SolrTestCase + EmbeddedSolrServerTestRule) covers an explicit field, the field taken from the alias, pass-through (single- and multi-valued), single value vs list, local params and request fallback/override, several instances in one fl, real-time get, and errors. The existing highlighter tests pass, as does gradlew check -x test. There's no distributed test, since the embedded test rule can't run a sharded request.

Checklist

• I have reviewed the guidelines for How to Contribute and my code conforms to the standards described there to the best of my ability.
• I have created a Jira issue and added the issue ID to my pull request title.

🤖 Generated with Claude Code

[dsmiley]

I've tested this in conjunction with #5113 and it works... albeit found some issues and I filed #5188 in response. Until then, best to use this like snippet:[hl f=myTextField] (i.e. name the field explicitly not implicitly).

I plan to merge Friday morning.

[nick-boss-tech]

One P2 finding:
Implicit highlighting includes ranking-only query terms. Without hl.q, the transformer uses the full search query, including eDisMax bq clauses. Explicit hl.q and the existing highlight component use the parser’s highlight query, which excludes those clauses. Affected code
For example, q=fox with bq=body_t:cats can highlight “cats”; adding hl.q=fox changes that result. Use consistent query selection and add an eDisMax regression test.
Also, the body’s RTG support claim is stale: the latest ticket explicitly acknowledges RTG as unsupported. The /export alias issue is already tracked by #5188.

AI-generated, static review.

Happy to open a ticket/PR to follow up on this, or provide you with a branch to include in this PR.

[dsmiley]

Implicit highlighting includes ranking-only query terms. Without hl.q, the transformer uses the full search query, including eDisMax bq clauses. Explicit hl.q and the existing highlight component use the parser’s highlight query, which excludes those clauses. Affected code

I don't think that's the case. The 'q' should be parsed by the parseHighlightQuery here which calls getHighlightQuery on the qparser and thus wouldn't have bq clauses.

Also, the body’s RTG support claim is stale: the latest ticket explicitly acknowledges RTG as unsupported.

I'm not sure what "the body's RTG support claim" refers to. Body of the PR? I don't see an RTG reference there.

[nick-boss-tech]

🤖 AI text below 🤖 (reviewed by Nick Shanin)

@dsmiley, I now have a small production fix and regression tests on top of your PR. On Linux, the new assertion fails on your current code because a boost-only term appears in the snippet, and passes with the fix. My earlier explanation was too compressed, and I missed an important qualification: this example requires hl.weightMatches=false. It did not reproduce with the default true.

You are right that getHighlightQuery() excludes eDisMax boost clauses. The distinction is that parseHighlightQuery reads hl.q, rather than ordinary q. Without hl.q or transformer-local q, that helper returns null before reaching getHighlightQuery(). The transformer then uses context.getQuery(), which contains the scoring boosts.

Using your existing document 2, the title is "Fox cameo" and the body is "A story about cats. Cats sleep a lot.". The important request parameters are:

q=fox
defType=edismax
qf=body_t title_t
bq=body_t:cats
fl=id,body_hl:[hl f=body_t]
hl.weightMatches=false

The document matches fox through its title. The cats clause is a ranking boost. Before the fix, its inline body_hl is A story about <em>cats</em>. <em>Cats</em> sleep a lot.. Adding hl.q=fox removes that body snippet. The existing highlight component also leaves the body without a snippet. With the fix, the implicit path agrees with those two paths.

The patch uses the search request's existing parser and its highlighting query for the implicit path. It preserves explicit hl.q and local q overrides, and keeps the context-query fallback when no matching search parser is available. It changes only HighlightAugmenterFactory.java and TestHighlightAugmenterFactory.java.

At final commit 8a685825b6, the transformer test class passes 11 of 11 tests and TestUnifiedSolrHighlighter passes 33 of 33. Tidy, compilation with Error Prone, and :solr:core:check -x test all pass. The tests cover both weightMatches settings, the component on and off, explicit/local highlight overrides, q.alt, and a locally selected main-query parser.

By "body" I meant the PR description's Solution bullet saying it works "in real-time get". I was comparing that wording with your JIRA note that RTG is unsupported. That was a description clarification. The proposed code change is limited to query selection; RTG support and the alias issue in #5188 stay separate.

Would this evidence and the two-file fix be sufficient to address the query-selection concern in this PR?

[dsmiley]

Thanks Nick! I took your changes.