Repository navigation
Add Postgres.load! to load a Tables.jl source into a table - #37
Merged
Merged
Conversation
MySQL.jl, SQLite.jl, ODBC.jl and LibPQ.jl can load a Tables.jl source; Postgres.jl needed a hand-written CREATE TABLE plus executemany, or hand-formatted COPY text. `Postgres.load!(table, conn, name; append, coltypes, quoteidentifiers)` creates the table from the source schema when it is missing and sends the rows with COPY FROM STDIN in text format, in 4 MiB batches, each value as its bind-parameter text with COPY escapes. The load is one transaction, so a failure adds no rows and removes a table it created. 1M rows x 4 columns: 2.6 s, against 20.9 s for pipelined executemany. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
JSON values read from json/jsonb columns, and JSON.JSONText, now get a jsonb column instead of text when load! creates the table. Their COPY text is the JSON itself (#39). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Merged
quinnj
referenced
this pull request
Oct 30, 2026
Co-authored-by: Jacob Quinn <quinn.jacobd@gmail.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Postgres.load!(table, conn, name; append=true, coltypes=Dict{Symbol,String}(), quoteidentifiers=true)loads any Tables.jl source into a table, andtable |> Postgres.load!(conn, name)does the same in a pipe. It is declaredpublic, not exported.Tables.schema: integers assmallint/integer/bigint(numericbeyond 64 bits), floats asreal/double precision,Boolasboolean,Date/Timeasdate/time,DateTimeastimestamp(3),Timestamps astimestamp(0|3|6),UUIDasuuid, decimals asnumeric(P, S), periods asinterval, byte vectors asbytea, other vectors as arrays, and everything else astext.coltypesoverrides a column (coltypes=Dict(:payload => "jsonb")).COPY ... FROM STDINin text format, in 4 MiB batches. Each value is written as the same text the driver already sends for a bind parameter (_param), with COPY's backslash escapes.missing/nothingbecome\N.append=falsedeletes the existing rows first.Why
MySQL.jl (
MySQL.load), SQLite.jl (SQLite.load!), ODBC.jl (ODBC.load) and LibPQ.jl (LibPQ.load!) can load a Tables.jl source. Postgres.jl hadexecutemany, which needs an existing table and anINSERT, and rawcopy_from, which needs hand-formatted COPY text. Today's equivalent ofPostgres.load!(CSV.File("sales.csv"), conn, "sales")is a hand-writtenCREATE TABLEplusDBInterface.executemany(conn, raw"INSERT INTO sales (...) VALUES ($1, ...)", Tables.columntable(file)).1M rows × (
Int,Float64,String,Date), PostgreSQL 16 in Docker on the same Mac, Julia 1.12.6, second of two runs:Postgres.load!DBInterface.executemany(pipelinedINSERT)For review
load!follows this package's own!for calls with side effects (listen!,notify!,register_type!) and matches SQLite.jl and LibPQ.jl; MySQL.jl and ODBC.jl useload.MySQL.load, so later appends withmissingwork.quoteidentifiersapplies to the table name; column names are always quoted, since they come from the data and PostgreSQL folds unquoted names to lower case.textcolumn as their parameter text,string(x); aDictis stored as its Juliastring, not as JSON. Passcoltypesand JSON strings forjsonb.Verification
byteacontaining a backslash byte; arrays with special characters; intervals; decimals; microsecond timestamps), and the declared column types match. Also: append andappend=false, the curried form, an inexact nanosecond timestamp leaving an existing table unchanged and removing a table the call created, 60k rows across two COPY batches with acoltypesoverride, a server error in the second batch rolling back the first, a source without a schema, a schema-qualified name withquoteidentifiers=false, and anArgumentErrorfor a source with no columns.POSTGRES_REQUIRE_INTEGRATION=true) on Julia 1.12.6: 9274/9274, including Aqua. The new testset fails onmain, wherePostgres.load!is undefined.🤖 Generated with Claude Code