Skip to content

Support Yugabyte as a Postgres target - #1347

Open
brandur wants to merge 1 commit into
masterfrom
brandur-yugabyte-support
Open

Support Yugabyte as a Postgres target#1347
brandur wants to merge 1 commit into
masterfrom
brandur-yugabyte-support

Conversation

@brandur

@brandur brandur commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

This one's aimed at #1346, in which it might be possible for us to
support Yugabyte as a database target without a hugely inordinate amount
of work.

Yugabyte is currently targeting compatibility against Postgres 15 [1].
It doesn't support xmax which is what #1346 is about, but somewhat
surprisingly, we only use xmax in one place and don't use any other
Postgres 16+ features (as Postgres 15 is still a valid target in the
CI matrix).

The xmax trick to determine whether an upserted row is new or existing
is a little outdated anyway because Postgres 18 added the capability to
detect an existing row with OLD.id IS NOT NULL [2].

Long run, we should switch to that for everything. Shorter term,
Postgres 18 is still quite new, so I propose we do something like this:

  • If on Postgres 18+ (we should be getting Postgres 19 soon), use
    OLD.id IS NOT NULL.

  • If on Yugabyte, fall back to the same trick we use in SQLite by
    upserting rows with a unique nonce and checking whether the nonce was
    the one we inserted or not.

  • Otherwise, use the existing approach with xmax.

We do have to check which database we're on, but only once, after which
we can cache that information forever, so it shouldn't have any impact
on performance.

Fixes #1346.

[1] https://docs.yugabyte.com/stable/faq/compatibility/#what-is-the-extent-of-compatibility-with-postgresql
[2] https://www.crunchydata.com/blog/postgres-18-old-and-new-in-the-returning-clause


// UniqueInsertMetadataWithNonce returns metadata with nonce set under
// UniqueInsertMetadataKey.
func UniqueInsertMetadataWithNonce(metadata []byte, nonce string) ([]byte, error) {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This approach originally comes from SQLite, but extracted out to here to be reusable.

This one's aimed at #1346, in which it might be possible for us to
support Yugabyte as a database target without a hugely inordinate amount
of work.

Yugabyte is currently targeting compatibility against Postgres 15 [1].
It doesn't support `xmax` which is what #1346 is about, but somewhat
surprisingly, we only use `xmax` in one place and don't use any other
Postgres 16+ features (as Postgres 15 is still a valid target in the
CI matrix).

The `xmax` trick to determine whether an upserted row is new or existing
is a little outdated anyway because Postgres 18 added the capability to
detect an existing row with `OLD.id IS NOT NULL` [2].

Long run, we should switch to that for everything. Shorter term,
Postgres 18 is still quite new, so I propose we do something like this:

* If on Postgres 18+ (we should be getting Postgres 19 soon), use
  `OLD.id IS NOT NULL`.

* If on Yugabyte, fall back to the same trick we use in SQLite by
  upserting rows with a unique nonce and checking whether the nonce was
  the one we inserted or not.

* Otherwise, use the existing approach with `xmax`.

We do have to check which database we're on, but only once, after which
we can cache that information forever, so it shouldn't have any impact
on performance.

Fixes #1346.

[1] https://docs.yugabyte.com/stable/faq/compatibility/#what-is-the-extent-of-compatibility-with-postgresql
[2] https://www.crunchydata.com/blog/postgres-18-old-and-new-in-the-returning-clause
@brandur
brandur force-pushed the brandur-yugabyte-support branch from 8359011 to a524293 Compare August 11, 2026 20:20
@brandur

brandur commented Aug 11, 2026

Copy link
Copy Markdown
Contributor Author

@bgentry Thoughts on this?

I'm kind of thinking that we wouldn't be able to officially support Yugabyte since I really don't want to be testing against it, but we could have soft support that works as long as they commit to their stated PG 15 contract with a few very minor deviations (like the xmax thing which admittedly is a little bit abusive of Postgres anyway).

@brandur
brandur requested a review from bgentry August 11, 2026 20:21
@jqueuniet

Copy link
Copy Markdown

I tested this branch in our test environment and our service managed to both enqueue asynchronous tasks and run scheduled ones without any other error.

@brandur

brandur commented Aug 12, 2026

Copy link
Copy Markdown
Contributor Author

@jqueuniet Excellent! Thanks for checking. Next we'll see if it can stand the stress of a DB-based job queue ...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Yugabyte compatibility issues when inserting new tasks

2 participants