Skip to content

Make typed reads statically compilable - #26

Merged
quinnj-bot[bot] merged 10 commits into
mainfrom
jq/trim-safe-typed-reads
Oct 27, 2026
Merged

quinnj-bot[bot] merged 10 commits into
mainfrom
jq/trim-safe-typed-reads

Conversation

@quinnj

@quinnj quinnj commented Oct 5, 2026 •

Copy link
Copy Markdown
Member

Typed reads with the default PostgresStyle now parse each field from the column text by its declared type instead of going through the OID decoder, which returns Any. Fields of the common types (strings, symbols, enums, integers, floats, Bool, Char, Date, Time, DateTime, UUID, bytea, lazy JSON, timestamps, decimals, vectors of these, and their Nothing/Missing unions) decode without runtime dispatch, so typed reads work in juliac --trim builds. Decoding is also faster: a 10-column typed row takes about a quarter of the time and half the allocations it did on main.

Conversions that worked before still work: integer fields read integral numeric text and float columns printed in exponent form (1e+15), a float4 read into Float64 keeps its Float32 value, Date and Time fields over timestamp columns get the UTC date or time of day, integer arrays convert into Vector{UInt8}, and a registered parser still applies when it returns the field's type. General unions and custom styles keep the OID decoder.

Behavior changes:

  • A String field receives any column's text, so json/jsonb columns read as JSON text (this threw before).
  • Numeric fields over bool columns throw (they read 1 or 0).
  • JSON null in a Union{Nothing, String} field reads as "null"; only SQL NULL becomes nothing.
  • A registered parser whose type does not fit the field is skipped, and the field parses the column text.

Also for --trim: the startup message and tuple parameters no longer need runtime dispatch, and Date/DateTime parameters use a static formatter with the same output as string(x).

test/trim_compile_tests.jl restores the JuliaC trim test. It compiles a workload of typed reads, prepared statements, transactions, and parameters, fails on any Postgres.jl verifier error, and runs the executable against PostgreSQL with trust authentication. Released Julia's Base64 (SCRAM) and Dates.CompoundPeriod (intervals) are not trim-safe; those errors are reported but not counted, so --trim=safe still rejects the build. docs/src/support.md says so.

Thanks to @PingoLee for the compatibility matrix that found the float, union, and Time regressions in the first version of this PR.

Version 2.4.0.

Co-authored by Codex

🤖 Generated with Claude Code

quinnj and others added 4 commits October 5, 2026 16:22
`juliac --trim=safe` rejected typed queries: an application with a few
hundred row structs hit 1,455 verifier errors, all in Postgres.jl. Typed
fields that are not timestamps or decimals were decoded by the column's OID
decoder, which returns `Any`, so every field's `StructUtils.make` call was a
runtime dispatch. Reading a json/jsonb (or any non-text) column into a
`String` field also failed with a MethodError, because the OID decoder
produced lazy JSON, integers, UUIDs and so on.

With the default `PostgresStyle`, fields whose type Postgres.jl parses from
text (strings, symbols, enums, integers, floats, Bool, Char, Date, Time,
DateTime, UUID, bytea, lazy JSON, timestamps, decimals, and vectors of these,
optionally unioned with Nothing or Missing) now decode from the column text
by their declared type. The source stays a `String`, so each read compiles
to static calls. A `String` field receives any column's text. Decoders keep
the conversions that worked before: integer fields accept integral numeric
text such as "12.00" and throw InexactError when out of range, and a Date
field reads the UTC date of a timestamptz. A parser registered for the
column still applies when its declared type fits the field. Custom styles
keep the OID decoder, so their lifts still receive decoded values; other
field types keep it too.

Also fixed while verifying under `--trim`:
- Protocol strings containing a NUL byte threw UndefVarError (the check
  referred to `Postgres.PostgresInterfaceError` from inside the API module)
  instead of PostgresInterfaceError.
- The startup message took an unused `statement_timeout` argument; with
  `options` passed as a String the call no longer needs runtime dispatch.
- `writemessages` and tuple parameters to `execute` iterated heterogeneous
  tuples in a loop; both now unroll.
- Date and DateTime parameters were formatted with `string(x)`, whose Dates
  printing is not trim-safe; they use an equivalent static formatter.

Behavior changes for typed fields read with the default style: an Int or
Float field over a bool column now errors (it previously read 1 or 0), and
a registered parser whose declared type does not fit the field is no longer
applied before converting; the field parses the column text instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The 1.0 release removed test/trim_compile_tests.jl, so later changes to
typed decoding regressed `--trim` builds without a failing test. The
restored test compiles test/postgres_trim_queries.jl with JuliaC, requires
zero verifier errors, and runs the executable against the test database
(trust auth). The workload covers typed reads of nullable strings, dates and
timestamps, numerics into Float64 and Int fields, enums, arrays, and jsonb
into a String field, plus prepared statements, transactions and parameters
of each bound type.

JuliaC needs Julia 1.12, so it lives in test/trim/Project.toml and the test
builds that environment only on released 1.12+ (64-bit, non-Windows), as
JSON.jl does. Released Julia's Base64 pipes and Dates.CompoundPeriod
constructor are not trim-safe; SASLAuth's SCRAM exchange and interval
decoding reach them. Errors raised inside those stdlib files are reported
and not counted, and the test compiles with `--trim=unsafe-warn` so the
executable still builds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Kerberos fixture's cleanup ran `DROP ROLE krbuser` right after closing
the last krbuser connection. That connection created a temp table, and
PostgreSQL drops a session's temp tables only as its backend exits, which
can trail `close!`. On a fast runner the DROP ROLE won the race and failed
with 2BP01 ("owner of table pg_temp_4.bulk_gss"). Poll pg_stat_activity
until the role has no sessions (up to 10s) before dropping it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@PingoLee

Copy link
Copy Markdown
Contributor

Thanks for this. I compared typed reads between main and this branch with a matrix: 38 field types (each scalar, Union{Nothing/Missing,…}, a few unions and vectors) × 87 column values (limits, NaN/Infinity, exponent-form floats, long numerics, BC and infinity dates, timestamptz with odd offsets, arrays with NULLs…) × 7 session settings (TimeZone, IntervalStyle, bytea_output, extra_float_digits), about 23,000 reads per side on PostgreSQL 18 and Julia 1.13. Everything under "Behavior changes" reproduced as described, and the Date/DateTime parameter formatter produced the same text as string(x) in every case, including 5-digit and negative years.

Leaving aside pairings that already produced nonsense on main (e.g. an int4 column into a Vector{Int64} field), three regressions aren't in that list:

1. An Int field over a float4/float8 column with a value of 1e15 or more. PostgreSQL prints these in exponent form (SELECT 1e15::float8 returns 1e+15), so pg_parse_integer falls through to Parsers.parse(Int64, "1e+15") and throws ArgumentError. On main the field reads 1000000000000000.

2. Union field types. _text_field(Union{Int64, Float64}) is true, because the union is a subtype of TextField, but nothing lifts text into the union. A Union{Int64, Float64} field over a float8 column now throws MethodError (main: the Float64), and a Union{Int64, String} field over an int8 column now gets a String (main: the Int64).

3. A Time field over a timestamp/timestamptz column. On main it reads the (UTC) time of day, e.g. Time(23, 59, 59, 999, 999); now pg_parse_time throws ArgumentError. It's the counterpart of the Date-from-timestamptz conversion this PR keeps.

A possible fix for all three, which I tried on this branch. With it, every one of these cases reads as on main across all 7 settings, and the full suite passes locally (Julia 1.13, Linux), including the restored trim test: 0 Postgres.jl verifier errors, and the compiled executable runs its workload against the database.

Patch
--- a/src/api/types.jl
+++ b/src/api/types.jl
@@ -409,6 +409,8 @@ end
 
 # timestamp(tz) column into a Date field: the UTC date, as for DateTime fields
 pg_parse_date_any(s::AbstractString)::Date = occursin(':', s) ? Date(pg_parse_datetime_any(s)) : pg_parse_date(s)
+# timestamp(tz) column into a Time field: the UTC time of day
+pg_parse_time_any(s::AbstractString)::Time = occursin(' ', s) ? convert(Time, pg_parse_timestamp(s)) : pg_parse_time(s)
 
 # Integer fields also read integral numeric text such as "12.00" (numeric
 # columns, or SUM over one). Out-of-range values throw InexactError, as when
@@ -423,7 +425,13 @@ function pg_parse_integer(::Type{T}, s::AbstractString) where {T <: Integer}
             wide = Parsers.tryparse(Int128, SubString(s, 1, dot - 1))
         end
     end
-    wide === nothing && return Parsers.parse(T, s)
+    if wide === nothing
+        # float4/float8 columns print large values in exponent form ("1e+16");
+        # numeric never does, so this conversion is exact
+        f = (occursin('e', s) || occursin('E', s)) ? Parsers.tryparse(Float64, s) : nothing
+        f === nothing && return Parsers.parse(T, s)
+        return convert(T, f)
+    end
     return convert(T, wide)
 end
 
@@ -1003,7 +1011,7 @@ const TextField = Union{String, Symbol, Bool, Char, Int8, Int16, Int32, Int64, C
     Date, Time, DateTime, UUID, Vector{UInt8}, JSONType, Vector{String}, Vector{Int16}, Vector{Int32},
     Vector{Int64}, Vector{Float32}, Vector{Float64}, Vector{Bool}, Vector{Date}, Vector{Time},
     Vector{DateTime}, Vector{UUID}, Vector{Char}}
-_text_field(::Type{S}) where {S} = S !== Union{} && (S <: Union{TextField, Enum} || S === IntervalType || _wire_field(S))
+_text_field(::Type{S}) where {S} = S === IntervalType || (S !== Union{} && !(S isa Union) && (S <: Union{TextField, Enum} || _wire_field(S)))
 
 function _field_source(::PostgresStyle, ::Type{T}, v::FieldValue) where {T}
     S = Base.nonnothingtype(Base.nonmissingtype(T))
@@ -1056,7 +1064,7 @@ StructUtils.lift(::AbstractPostgresStyle, ::Type{Cuint}, s::String) = pg_parse_i
 StructUtils.lift(::AbstractPostgresStyle, ::Type{Float32}, s::String) = Parsers.parse(Float32, s), nothing
 StructUtils.lift(::AbstractPostgresStyle, ::Type{Float64}, s::String) = Parsers.parse(Float64, s), nothing
 StructUtils.lift(::AbstractPostgresStyle, ::Type{Date}, s::String) = pg_parse_date_any(s), nothing
-StructUtils.lift(::AbstractPostgresStyle, ::Type{Time}, s::String) = pg_parse_time(s), nothing
+StructUtils.lift(::AbstractPostgresStyle, ::Type{Time}, s::String) = pg_parse_time_any(s), nothing
 StructUtils.lift(::AbstractPostgresStyle, ::Type{DateTime}, s::String) = pg_parse_datetime_any(s), nothing
 StructUtils.lift(::AbstractPostgresStyle, ::Type{UUID}, s::String) = UUID(s), nothing
 StructUtils.lift(::AbstractPostgresStyle, ::Type{IntervalType}, s::String) = parse_interval(s), nothing
--- a/src/array_parsing.jl
+++ b/src/array_parsing.jl
@@ -4,7 +4,7 @@ using Parsers, Dates, UUIDs
 import Durations, DataDecimals
 import ..parse_decimal
 import ..pg_parse_timestamp
-import ..pg_parse_date, ..pg_parse_date_any, ..pg_parse_time, ..pg_parse_datetime_any, ..pg_parse_char, ..pg_parse_integer
+import ..pg_parse_date, ..pg_parse_date_any, ..pg_parse_time, ..pg_parse_time_any, ..pg_parse_datetime_any, ..pg_parse_char, ..pg_parse_integer
 
 const BRACKET_OPEN = UInt8('[')
 const BRACKET_CLOSE = UInt8(']')
@@ -94,7 +94,7 @@ function parse_scalar(token::String, inner_type::Type{T}, quoted::Bool) where {T
     inner_type === Float32 && return Parsers.parse(Float32, token)
     inner_type === Float64 && return Parsers.parse(Float64, token)
     inner_type === Date && return pg_parse_date_any(token)
-    inner_type === Time && return pg_parse_time(token)
+    inner_type === Time && return pg_parse_time_any(token)
     inner_type <: Durations.Timestamp && return pg_parse_timestamp(token, inner_type)
     inner_type === DateTime && return pg_parse_datetime_any(token)
     inner_type === UUID && return UUID(token)

Smaller differences, probably fine either way:

  • An Int field over non-integral text (3.5::float8, 12.50::numeric) now throws ArgumentError from Parsers instead of InexactError.
  • A Float64 field over a float4 column now reads the printed value, 0.1, instead of the widened Float32, 0.10000000149011612. That matches the text the server sends, but it changes the value silently.
  • A Union{Nothing, String} field over a json 'null' now reads "null" instead of nothing. The column isn't SQL NULL, so this looks like a fix.

AI use: the edge-case matrix, the suggested fix and the test runs were done with Claude Code (Claude Opus 5.5). I reviewed the results and this comment before posting.

quinnj and others added 3 commits October 26, 2026 17:43
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Use the source floating precision for numeric conversions, preserve union selection, and extract UTC times from timestamps. Cover scalar, nullable, and array reads against PostgreSQL and in the compiled workload.

Recognize current and legacy trim diagnostics, check their reported totals, and capture the complete compiler output through a pipe. Keep the existing stdlib exemptions and verify the executable against a live database.
@quinnj-bot quinnj-bot Bot changed the title Make typed reads statically compilable under juliac --trim Preserve typed conversions in statically compiled reads Oct 27, 2026
quinnj and others added 2 commits October 27, 2026 12:23
A Vector{UInt8} field over an integer array column decoded the array
text as bytea, so {1,2} read as the bytes of "{1,2}". It converts
element by element again, as on main.

docs/src/support.md still said --trim was unsupported; it now says what
the trim test checks and why --trim=safe still fails on stock Julia.
Date parameters pad with string(n; pad), the trim test runner uses a
Timer instead of a polling loop, and the live typed-read test drops
session settings that do not affect its values. Version 2.4.0: main
already released 2.3.1, and typed reads change behavior.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@quinnj-bot quinnj-bot Bot changed the title Preserve typed conversions in statically compiled reads Make typed reads statically compilable Oct 27, 2026
@quinnj-bot
quinnj-bot Bot merged commit 85cd44a into main Oct 27, 2026
16 checks passed
@quinnj-bot

quinnj-bot Bot commented Oct 27, 2026

Copy link
Copy Markdown
Contributor

@PingoLee thanks for the comparison matrix. The three regressions you found are fixed in the merged version, with tests against a live server: float columns parse at their own precision before converting, general unions keep the OID decoder, and Time fields read the UTC time of day from timestamp columns. On your smaller notes: float4 into Float64 keeps the widened Float32 value, JSON null reads as "null", and a non-integral value into an Int field throws InexactError from float columns and ArgumentError from numeric text.

Review found one more case: an integer array column read into a Vector{UInt8} field decoded the array text as bytea. It converts element by element again. This ships in 2.4.0.

AI disclosure: Claude Code reviewed and fixed the PR and drafted this comment.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants