Skip to content

SOLR-9957: /export supports document transformers in fl - #5113

Merged
dsmiley merged 3 commits into
apache:mainfrom
dsmiley:SOLR-9957-exportTransformers
Oct 20, 2026
Merged

SOLR-9957: /export supports document transformers in fl#5113
dsmiley merged 3 commits into
apache:mainfrom
dsmiley:SOLR-9957-exportTransformers

Conversation

@dsmiley

@dsmiley dsmiley commented Oct 13, 2026

Copy link
Copy Markdown
Contributor

https://issues.apache.org/jira/browse/SOLR-9957

Description

/export silently ignored document transformers in fl (e.g. [docid], [value], [explain]), and field aliases like fl=x:myfield came back under the original name. fl was also split naively on commas, so transformer local params containing commas broke.

Solution

  • Parse fl with SolrReturnFields (same as /select), so transformers, their local params, and aliases parse correctly. score is still rejected.
  • When fl has a transformer, each doc's fields are collected into a SolrDocument, the transformer is applied with the global docid, and the wanted fields are written out. Without a transformer, the existing streaming path is unchanged (no per-doc allocation).
  • Transformers get a ResultContext with the export's searcher, query and request. There is no DocList, so transformers needing one (or scores) aren't supported; this is documented.
  • Fields a transformer needs (e.g. an alias's source) are fetched even if not listed in fl.
  • Transformer output flows into tuples seen by the local streaming expression (expr).

Behavior change: fl=x:myfield now returns x rather than myfield. Noted in the changelog.

Tests

  • New TestExportWriter.testDocTransformers: [docid] matches /select's, multi-valued fields survive, aliases and comma-containing local params work, transformer output reaches expr, and score is still rejected.
  • Passing: org.apache.solr.handler.export.*, plus StreamExpressionTest, StreamingTest, StreamDecoratorTest; ./gradlew check -x test.

🤖 This PR was written with Claude Code (AI-generated), directed and reviewed by the author.

@dsmiley @claude
SOLR-9957: /export supports document transformers in fl
fb7ca87 
* Parse fl with SolrReturnFields instead of splitting on commas, so transformers, their local params, and field aliases work
* When fl has a transformer, collect each doc's fields into a SolrDocument, apply the transformer, and write the wanted fields; the plain path is unchanged
* Transformer output is visible to the local streaming expression (expr)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@github-actions github-actions Bot added documentation Improvements or additions to documentation tests labels Oct 13, 2026
@dsmiley @claude
SOLR-9957: clarify which fields getFieldWriters fetches vs errors on
a7ee6ac 
Split the combined field-name set into two named sets, one for fetching transformer-read fields and one for deciding whether a non-exportable field errors or is skipped. No behavior change.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
dsmiley
dsmiley commented Oct 13, 2026
View reviewed changes

@dsmiley dsmiley left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was basically a one-shot with Claude and I'm quite happy with the results. 2nd commit improves clarity around use of the confusing ReturnFields API.

Comment thread solr/core/src/java/org/apache/solr/handler/export/ExportWriter.java
Comment on lines +500 to +505
if (transformer == null) {
for (FieldWriter fieldWriter : writers) {
fieldWriter.write(sortDoc, context, ew);
}
return;
}

@dsmiley dsmiley Oct 13, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fast-path for previous logic, unchanged

@dsmiley
dsmiley requested a review from kotman12 October 13, 2026 01:05
kotman12
kotman12 approved these changes Oct 13, 2026
View reviewed changes
Comment thread solr/core/src/java/org/apache/solr/handler/export/ExportWriter.java
String[] fields = null;

if (fl == null) {
if (params.get(CommonParams.FL) == null) {

@kotman12 kotman12 Oct 13, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you check this on the SolrReturnFields object? Just for brevity...

Comment thread solr/solr-ref-guide/modules/query-guide/pages/exporting-result-sets.adoc
Wildcard patterns can be used for the field list (e.g. `fl=*_i`) and will be expanded to the list of fields that match the pattern and are able to be exported, see <<Field Requirements>>.

Field aliases (e.g. `fl=id,price:price_d`) and xref:document-transformers.adoc[document transformers] (e.g. `fl=id,[docid]`) are supported, and their output is also visible to a local streaming expression.
Transformers are applied one document at a time, and those that depend on scores or on a list of matching documents won't work.

@kotman12 kotman12 Oct 13, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generally transformers that don't return maps/lists/values won't work, right? I.e. fl=id,myquery:[subquery] won't work. The DocTransformer integration uncovers some gaps in the existing ExportWriter/javabin writer vs the text writer. It may not be a common use case here but maybe worth calling out.

Comment thread solr/core/src/java/org/apache/solr/handler/export/ExportWriter.java
fieldWriter.write(sortDoc, context, ew);
fieldWriter.write(sortDoc, context, docWriter);
}
transformer.transform(doc, sortDoc.docBase + sortDoc.docId, DocIterationInfo.NONE);

@kotman12 kotman12 Oct 13, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will this will return the transformed fields out of fl order? Not a huge deal but just pointing it out.

@kotman12

kotman12 commented Oct 13, 2026

Copy link
Copy Markdown
Contributor

Looks cool, left some comments. I assume this is motivated by the highlighted stream work?

@dsmiley

dsmiley commented Oct 13, 2026

Copy link
Copy Markdown
Contributor Author

thanks for the review; yes motivated by #5126

@dsmiley

dsmiley commented Oct 13, 2026

Copy link
Copy Markdown
Contributor Author

On line 283: Unfortunately not: an absent fl and fl=* both become "wants all fields" in SolrReturnFields, and fl=* is valid here, so only the raw param distinguishes "not specified".

On the ref guide: Good catch. The javabin path (which streaming uses) has no resolver for core types, so e.g. [subquery]'s result would come out as a toString string; [child]'s anonymous children and raw [json]/[xml] values also don't carry over. I'll call out that only transformers producing simple values, lists, or maps are supported.

done

On field order: Yes. Though /export has never followed fl order (fields come out in the index's field order); transformer-added or renamed fields go last, same as in /select.

🤖 These replies were drafted with AI assistance (Claude).

@dsmiley dsmiley added this to the 9.x milestone Oct 20, 2026
@dsmiley
dsmiley merged commit bd2b2d5 into apache:main Oct 20, 2026
3 of 4 checks passed
@dsmiley
dsmiley deleted the SOLR-9957-exportTransformers branch October 20, 2026 20:49
This was referenced Oct 21, 2026
Merged
Merged
dsmiley added a commit that referenced this pull request Oct 22, 2026
@dsmiley @claude
SOLR-9957: /export supports document transformers in fl (#5113)
cf49460 
Albeit some might not work.

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
(cherry picked from commit bd2b2d5)
dsmiley added a commit that referenced this pull request Nov 2, 2026
@dsmiley
SOLR-9957: /export supports document transformers in fl (#5113)
1899409 
Albeit some might not work.

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
(cherry picked from commit bd2b2d5)
(cherry picked from commit cf49460)

Adapted to Java 11 and branch_9x's DocTransformer API; tests use qt=/export.
agazzarini pushed a commit to SeaseLtd/solr that referenced this pull request Nov 3, 2026
@dsmiley @claude @agazzarini
SOLR-9957: /export supports document transformers in fl (apache#5113)
b1ec955 
Albeit some might not work.

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@kotman12 kotman12 kotman12 approved these changes

Assignees

No one assigned

Labels

documentation Improvements or additions to documentation tests

Projects

None yet

Milestone

9.x

Development

Successfully merging this pull request may close these issues.

2 participants

@dsmiley @kotman12