Skip to content

Add Postgres.load! to load a Tables.jl source into a table - #37

Merged
quinnj-bot[bot] merged 3 commits into
mainfrom
claude/load-tables
Oct 30, 2026
Merged

quinnj-bot[bot] merged 3 commits into
mainfrom
claude/load-tables

Conversation

@quinnj-bot

@quinnj-bot quinnj-bot Bot commented Oct 29, 2026

Copy link
Copy Markdown
Contributor

What

Postgres.load!(table, conn, name; append=true, coltypes=Dict{Symbol,String}(), quoteidentifiers=true) loads any Tables.jl source into a table, and table |> Postgres.load!(conn, name) does the same in a pipe. It is declared public, not exported.

  • If the table doesn't exist, it is created from Tables.schema: integers as smallint/integer/bigint (numeric beyond 64 bits), floats as real/double precision, Bool as boolean, Date/Time as date/time, DateTime as timestamp(3), Timestamps as timestamp(0|3|6), UUID as uuid, decimals as numeric(P, S), periods as interval, byte vectors as bytea, other vectors as arrays, and everything else as text. coltypes overrides a column (coltypes=Dict(:payload => "jsonb")).
  • Rows go through COPY ... FROM STDIN in text format, in 4 MiB batches. Each value is written as the same text the driver already sends for a bind parameter (_param), with COPY's backslash escapes. missing/nothing become \N.
  • The whole load is one transaction (a savepoint inside an open one). A failure adds no rows, and a table the call created is removed. append=false deletes the existing rows first.

Why

MySQL.jl (MySQL.load), SQLite.jl (SQLite.load!), ODBC.jl (ODBC.load) and LibPQ.jl (LibPQ.load!) can load a Tables.jl source. Postgres.jl had executemany, which needs an existing table and an INSERT, and raw copy_from, which needs hand-formatted COPY text. Today's equivalent of Postgres.load!(CSV.File("sales.csv"), conn, "sales") is a hand-written CREATE TABLE plus DBInterface.executemany(conn, raw"INSERT INTO sales (...) VALUES ($1, ...)", Tables.columntable(file)).

1M rows × (Int, Float64, String, Date), PostgreSQL 16 in Docker on the same Mac, Julia 1.12.6, second of two runs:

time allocated
Postgres.load! 2.6 s 756 MiB
DBInterface.executemany (pipelined INSERT) 20.9 s 1676 MiB

For review

  • Name: load! follows this package's own ! for calls with side effects (listen!, notify!, register_type!) and matches SQLite.jl and LibPQ.jl; MySQL.jl and ODBC.jl use load.
  • Created columns are nullable, as in MySQL.load, so later appends with missing work.
  • quoteidentifiers applies to the table name; column names are always quoted, since they come from the data and PostgreSQL folds unquoted names to lower case.
  • Types without a mapping go into a text column as their parameter text, string(x); a Dict is stored as its Julia string, not as JSON. Pass coltypes and JSON strings for jsonb.

Verification

  • New live testset "Load Tables" (33 assertions): every mapped type round-trips (strings with tabs, newlines, backslashes, quotes and braces; bytea containing a backslash byte; arrays with special characters; intervals; decimals; microsecond timestamps), and the declared column types match. Also: append and append=false, the curried form, an inexact nanosecond timestamp leaving an existing table unchanged and removing a table the call created, 60k rows across two COPY batches with a coltypes override, a server error in the second batch rolling back the first, a source without a schema, a schema-qualified name with quoteidentifiers=false, and an ArgumentError for a source with no columns.
  • Full suite with the Docker integration tests (POSTGRES_REQUIRE_INTEGRATION=true) on Julia 1.12.6: 9274/9274, including Aqua. The new testset fails on main, where Postgres.load! is undefined.

🤖 Generated with Claude Code

MySQL.jl, SQLite.jl, ODBC.jl and LibPQ.jl can load a Tables.jl source;
Postgres.jl needed a hand-written CREATE TABLE plus executemany, or
hand-formatted COPY text.

`Postgres.load!(table, conn, name; append, coltypes, quoteidentifiers)`
creates the table from the source schema when it is missing and sends
the rows with COPY FROM STDIN in text format, in 4 MiB batches, each
value as its bind-parameter text with COPY escapes. The load is one
transaction, so a failure adds no rows and removes a table it created.
1M rows x 4 columns: 2.6 s, against 20.9 s for pipelined executemany.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
quinnj and others added 2 commits October 30, 2026 08:23
JSON values read from json/jsonb columns, and JSON.JSONText, now get a
jsonb column instead of text when load! creates the table. Their COPY
text is the JSON itself (#39).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@quinnj-bot
quinnj-bot Bot marked this pull request as ready for review October 30, 2026 14:39
@quinnj-bot
quinnj-bot Bot merged commit 1b601d7 into main Oct 30, 2026
16 checks passed
@quinnj-bot quinnj-bot Bot mentioned this pull request Oct 30, 2026
quinnj referenced this pull request Oct 30, 2026
Co-authored-by: Jacob Quinn <quinn.jacobd@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant