Add support for table value expressions in ORM by p-r-a-v-i-n · Pull Request #113 · django/deps

p-r-a-v-i-n · 2026-06-01T17:06:22Z

This DEP proposes the implementation of a CompositeField within the Django ORM to support Table-Valued Expressions (TVEs) and functions that return sets of rows.

Refereneces:

Ticket: https://code.djangoproject.com/ticket/37115
Django New Features Thread: Add support for generate_series in postgres new-features#25
forum posts:
- https://forum.djangoproject.com/t/proposal-add-generate-series-support-to-contrib-postgres/21947/4

jacobtylerwalls

Nice 👍

I added minor wording suggestions.

@charettes, @LilyFirefly would you like to take a look?

charettes

I left a few comments but overall the design document seems a like good foundation for further demonstration work of the viability of the approach.

I remain unconvinced of the necessity of the TableExpression abstraction as I believe isinstance(expr, CompositeField) (or a potential future CompositeField type narrowing Field.is_composite -> bool guard) is likely a good heuristic to fit our needs of augmenting sql.Query.alias_map. I believe this the case due to the complexity that isinstance(expr, (Subquery, QuerySet, sql.Query)) caused when Subquery was introduced (e.g. __in lookups).

charettes · 2026-06-14T16:33:28Z

+We will take inspiration from the existing `FilteredRelation`. We would follow the
+mechanism it used to register itself as a table source in the ``FROM`` clause.
+
+This DEP uses ``TableExpression`` as a placeholder name. The final public API
+may use a different name.


I suspect we won't need TableExpression at all as we'll be able to perform the FROM aliasing based on other heuristics.

For example, in the case of QuerySet.resolve_expression if there are more than one field than it's output_field would be a CompositeField instance and we could make GenerateSeries.output_field = CompositeField(IntegerField()) even if it's a singleton.

The heuristic to add a member to sql.Query.alias_map could then be isinstance(output_field, CompositeField).

We were also was in the favor of not adding TableExpression or something. But what we have discovered that .alias_map requires a Table-like object. and it was failing on the line
j for j in self.alias_map if self.alias_map[j].join_type == INNER
Query or Subquery don't have table_name, table_alias, join_type, parent_alias which would require to live inside alias_map as per the contract.

we might don't need objects.alias(TableExpression(Subquery(...)) instead

if isinstance(expr.output_field, CompositeField): alias_map[alias] =TableExpression(expr)

also query object as_sql will generate something (SELECT ... FROM ...) i'm assuming we might need something in prefix or suffix in the FROM clause with another table. for that also we need some handling.

I'm not sure I understand why the need for objects in alias_map to have certain attributes requires that objects passed to QuerySet.alias be TableExpression.

Why can't alias do something like

if isinstance(expr.output_field, CompositeField): join = Join( expr, alias, ... ) self.join(join)

In other words, shouldn't we adapt Join and BaseTable to be more expression like and allow composition of other Expression over introducing TableExpression to bridge between them?

I overlooked that this version of the DEP still suggested that TableExpression was going to be exposed to users. As you mentioned in #113 (comment) with the analogy to Subquery, it should be unnecessary in most instances, perhaps only if you need to specify a special kind of join?

We were just following the implementation hint in https://code.djangoproject.com/ticket/37115#comment:8 to create a base class for BaseTable and Join.

requires that objects passed to QuerySet.alias be TableExpression.

I think @p-r-a-v-i-n is aware that we don't want users to have to supply these wrappers, see #113 (comment), but we definitely need to update the DEP, because it suggests Simon's reading.

In other words, shouldn't we adapt Join and BaseTable to be more expression like and allow composition of other Expression over introducing TableExpression to bridge between them?

I think this is where we're all heading, let's discuss once there's an implementation PR to look at, which I think is imminent.

it should be unnecessary in most instances, perhaps only if you need to specify a special kind of join?

right but I wonder if this should be a concern of this work or not, I think that laying down the foundation for table-like expressions is already a ton of work in itself.

I think this is where we're all heading, let's discuss once there's an implementation PR to look at, which I think is imminent.

Sounds good! Peeking at code will definitely help here.

The question about join type came up right away in the first cut of the implementation, any guidance here @charettes and @LilyFirefly would be very helpful!

django/django#21496 (comment)

charettes · 2026-06-14T16:35:26Z

+For single-column table expressions, the final API may allow a shorter form such
+as ``series__gt=1``. That detail is still open.


That should be relatively straightforward to support by having CompositeField.get_lookup use it's only field when it's the case.

charettes · 2026-06-14T16:41:51Z

+An early implementation could accept column metadata directly:
+
+.. code-block:: python
+
+   TableExpression(
+       JsonEach("payload"),
+       columns={
+           "key": models.TextField(),
+           "value": models.JSONField(),
+       },
+   )
+
+This may be easy to prototype, but it creates a separate metadata path from
+``Expression.output_field``.


Not convinced this is warranted or not something that could not be provided by JSONEach directly.

e.g.

JsonEach("payload", key_column="attr")

could be turned into

(SELECT key AS "attr", value FROM json_each("payload"))

p-r-a-v-i-n · 2026-06-15T09:34:59Z

Just to make sure I understand. You mean that we would only support usage for annotation purpose in when building queries and not as a field that can be used when defining a model class?

yes

obstacles I see without Table-Expression
for Subqueries:

Query set don't have spacesuite to live inside the .alias_map. that spacesuite could be a Tableexpression.
We might need to add some prefix or suffix to sql generated by subquery like prefix , for cross join or something else

Genarito · 2026-06-15T14:14:05Z

Hi! Hope you're doing well!
Sorry if this is a silly question, but wouldn't this DEP conflict with the one I opened a few months ago?

#106

jacobtylerwalls · 2026-06-15T14:15:54Z

(Above, @p-r-a-v-i-n is using "spacesuit" as a colloquialism for "protocol".)

jacobtylerwalls · 2026-06-15T14:19:45Z

Just to make sure I understand. You mean that we would only support usage for annotation purpose in when building queries and not as a field that can be used when defining a model class?

I encouraged @p-r-a-v-i-n to defer that aspect for now, but we can re-prioritize it if you think it's important to include earlier. Can you say a little bit about the use cases for that?

charettes · 2026-06-15T17:00:51Z

@jacobtylerwalls

I encouraged @p-r-a-v-i-n to defer that aspect for now, but we can re-prioritize it if you think it's important to include earlier.

I think it's smart to defer the implementation as it should cut scope significantly. I think that making sure CompositePrimaryKey subclasses CompositeField which subclasses Field would be beneficial though. We could make a CompositeField.contribute_to_class override that makes sure to raise an exception if type(self) is CompositeField to disallow this usage for now.

Can you say a little bit about the use cases for that?

From my understanding that was the original intent behind #17279 / ticket-373 but we took a shortcut to focus solely on composite primary key at first (leaving us with the composite fk problem for example).

There was a SoC project for CompositeField 14 years ago and many discussions of the benefit of allowing virtual field composition to exist.

LilyFirefly · 2026-06-16T07:32:16Z

Hi! Hope you're doing well! Sorry if this is a silly question, but wouldn't this DEP conflict with the one I opened a few months ago?

#106

There's definitely some overlap, but this DEP is more about laying the groundwork for a wide collection of features, including CTEs. If this GSOC work goes well, you may be able to rewrite #106 to build on top of this work.

p-r-a-v-i-n · 2026-07-07T15:26:16Z

+Expression-like table sources
+-----------------------------
+After output columns can be represented, the ORM needs a way to explicitly register an expression or query as a table source in the ``FROM`` clause.
+We cannot be sure if subqueries return multi-column single row or multi-column multi-row.
+For example:
+
+.. code-block:: python
+
+   subquery = Post.objects.filter(user=first_user).values("title", "body")
+   User.objects.alias(posts=subquery).values("posts__title", "posts__body")
+
+In this example, the ORM cannot reliably know if the ``subquery`` returns exactly one row or multiple rows.
+To solve this, our approach requires the developer to explicitly declare when a result is a multi-row table source.
+The user will do this by wrapping the queryset in a ``TableExpression``.
+
+.. code-block:: python
+
+   User.objects.alias(posts=TableExpression(subquery))
+
+This explicit wrapper allows the ORM to cleanly separate the processing logic. When encountering a ``TableExpression``, the ORM knows to place the subquery inside the ``FROM`` clause (utilizing ``LATERAL`` joins if there are outer references) rather than resolving it as a scalar subquery in the ``SELECT`` clause.
+*(Note: This DEP uses ``TableExpression`` as a placeholder name. The final public API may use a different name).*


Please don't mind for writing very late and the reason is i'm being shy to write , I'm afraid of if I sound like "I'm forcing something or being very concrete on something".

I have try to add clarity in this section why do we might need TableExpression wrapper.

@jacobtylerwalls @LilyFirefly @charettes

Wouldn't the ~~scalar~~ single row case look like: (note the [:1])

subquery = Post.objects.filter(user=first_user).values("title", "body")[:1] User.objects.alias(posts=subquery).values("posts__title", "posts__body")

Or am I missing something here? (I might be!)

Please correct me, I'm assuming you doubt that i have used similar example with [:1] in scaler case.

below is just examples (my understanding) , so we both are on same page

subquery = Post.objects.filter(user=first_user).values("title")[:1]

This one is scaler.

subquery = Post.objects.filter(user=first_user).values("title", "body")[:1]

This is multi-column single row

subquery = Post.objects.filter(user=first_user).values("title", "body")

If first_user has created multiple posts the this will return multi-column multi-row

i used this so it represent the table-value result

Ah, so @LilyFirefly is your idea that we can inspect whether the queryset is sliced, and if not, treat it as multi-row, without the need for a wrapper?

I think it's consistent with how we've designed subqueries up to this point and I think it's worth exploring if it will work in practice or cause problems (maybe it's too subtle and people will get confused?).

I like it -- let's explore it.

p-r-a-v-i-n added 7 commits May 30, 2026 18:52

Ready. Set. Go.

08634f5

Renamed file to .rst so it can rendered properly

f569756

Formatting sub points

14f7626

Fixed python code block

e939129

Improved specification and link to my previous design

e4dff94

Added some clarity and Correct the Meta section

97a8b15

Edit the title

bdcca1c

p-r-a-v-i-n marked this pull request as draft June 1, 2026 17:18

p-r-a-v-i-n commented Jun 1, 2026

View reviewed changes

Comment thread draft/table_value_expression.rst Outdated

p-r-a-v-i-n marked this pull request as ready for review June 1, 2026 18:06

jacobtylerwalls approved these changes Jun 11, 2026

View reviewed changes

Appled suggestions received from feedback

32a2c14

p-r-a-v-i-n force-pushed the relation-api branch from a725312 to 32a2c14 Compare June 12, 2026 03:27

p-r-a-v-i-n commented Jun 12, 2026

View reviewed changes

Comment thread draft/table_value_expression.rst Outdated

charettes reviewed Jun 14, 2026

View reviewed changes

p-r-a-v-i-n added 2 commits June 15, 2026 11:50

Added detailing for the issue with inline table results

f25091a

Applied in-place suggestions

69f46f7

jacobtylerwalls reviewed Jun 15, 2026

View reviewed changes

Comment thread draft/table_value_expression.rst Outdated

LilyFirefly reviewed Jun 16, 2026

View reviewed changes

Comment thread draft/table_value_expression.rst Outdated

Removed the wrapping of Subquery and corrected the example

054acc1

p-r-a-v-i-n mentioned this pull request Jun 16, 2026

Refs #5929 -- Added a CompositeField expression to support multi-column subqueries. django/django#21496

Closed

11 tasks

LilyFirefly reviewed Jul 6, 2026

View reviewed changes

Comment thread draft/table_value_expression.rst

p-r-a-v-i-n added 2 commits July 7, 2026 20:13

Refactored DEP: Added clarity in the usage of TableExpression

6b85a24

fixed the sql syntax spacing

752376e

p-r-a-v-i-n added 2 commits July 7, 2026 20:28

Fixed spacing in other places

92c851f

fixed spelling

e3d6b3a

p-r-a-v-i-n commented Jul 7, 2026

View reviewed changes

p-r-a-v-i-n mentioned this pull request Jul 17, 2026

Refs #5929 -- Added a CompositeField expression to support multi-column subqueries. django/django#21636

Open

11 tasks

Revised the design and added more simplicity.

a771cf9

		For single-column table expressions, the final API may allow a shorter form such
		as ``series__gt=1``. That detail is still open.

Uh oh!

Conversation

p-r-a-v-i-n commented Jun 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

jacobtylerwalls left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

charettes left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

charettes Jun 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

p-r-a-v-i-n commented Jun 15, 2026

Uh oh!

Genarito commented Jun 15, 2026

Uh oh!

jacobtylerwalls commented Jun 15, 2026

Uh oh!

jacobtylerwalls commented Jun 15, 2026

Uh oh!

charettes commented Jun 15, 2026

Uh oh!

Uh oh!

LilyFirefly commented Jun 16, 2026

Uh oh!

Uh oh!

Uh oh!

p-r-a-v-i-n Jul 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

LilyFirefly Jul 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

p-r-a-v-i-n Jul 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

p-r-a-v-i-n commented Jun 1, 2026 •

edited

Loading

charettes Jun 14, 2026 •

edited

Loading

p-r-a-v-i-n Jul 7, 2026 •

edited

Loading

LilyFirefly Jul 7, 2026 •

edited

Loading

p-r-a-v-i-n Jul 7, 2026 •

edited

Loading