Repository navigation
Make typed reads statically compilable - #26
Conversation
`juliac --trim=safe` rejected typed queries: an application with a few hundred row structs hit 1,455 verifier errors, all in Postgres.jl. Typed fields that are not timestamps or decimals were decoded by the column's OID decoder, which returns `Any`, so every field's `StructUtils.make` call was a runtime dispatch. Reading a json/jsonb (or any non-text) column into a `String` field also failed with a MethodError, because the OID decoder produced lazy JSON, integers, UUIDs and so on. With the default `PostgresStyle`, fields whose type Postgres.jl parses from text (strings, symbols, enums, integers, floats, Bool, Char, Date, Time, DateTime, UUID, bytea, lazy JSON, timestamps, decimals, and vectors of these, optionally unioned with Nothing or Missing) now decode from the column text by their declared type. The source stays a `String`, so each read compiles to static calls. A `String` field receives any column's text. Decoders keep the conversions that worked before: integer fields accept integral numeric text such as "12.00" and throw InexactError when out of range, and a Date field reads the UTC date of a timestamptz. A parser registered for the column still applies when its declared type fits the field. Custom styles keep the OID decoder, so their lifts still receive decoded values; other field types keep it too. Also fixed while verifying under `--trim`: - Protocol strings containing a NUL byte threw UndefVarError (the check referred to `Postgres.PostgresInterfaceError` from inside the API module) instead of PostgresInterfaceError. - The startup message took an unused `statement_timeout` argument; with `options` passed as a String the call no longer needs runtime dispatch. - `writemessages` and tuple parameters to `execute` iterated heterogeneous tuples in a loop; both now unroll. - Date and DateTime parameters were formatted with `string(x)`, whose Dates printing is not trim-safe; they use an equivalent static formatter. Behavior changes for typed fields read with the default style: an Int or Float field over a bool column now errors (it previously read 1 or 0), and a registered parser whose declared type does not fit the field is no longer applied before converting; the field parses the column text instead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The 1.0 release removed test/trim_compile_tests.jl, so later changes to typed decoding regressed `--trim` builds without a failing test. The restored test compiles test/postgres_trim_queries.jl with JuliaC, requires zero verifier errors, and runs the executable against the test database (trust auth). The workload covers typed reads of nullable strings, dates and timestamps, numerics into Float64 and Int fields, enums, arrays, and jsonb into a String field, plus prepared statements, transactions and parameters of each bound type. JuliaC needs Julia 1.12, so it lives in test/trim/Project.toml and the test builds that environment only on released 1.12+ (64-bit, non-Windows), as JSON.jl does. Released Julia's Base64 pipes and Dates.CompoundPeriod constructor are not trim-safe; SASLAuth's SCRAM exchange and interval decoding reach them. Errors raised inside those stdlib files are reported and not counted, and the test compiles with `--trim=unsafe-warn` so the executable still builds. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Kerberos fixture's cleanup ran `DROP ROLE krbuser` right after closing
the last krbuser connection. That connection created a temp table, and
PostgreSQL drops a session's temp tables only as its backend exits, which
can trail `close!`. On a fast runner the DROP ROLE won the race and failed
with 2BP01 ("owner of table pg_temp_4.bulk_gss"). Poll pg_stat_activity
until the role has no sessions (up to 10s) before dropping it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Thanks for this. I compared typed reads between Leaving aside pairings that already produced nonsense on 1. An 2. Union field types. 3. A A possible fix for all three, which I tried on this branch. With it, every one of these cases reads as on Patch--- a/src/api/types.jl
+++ b/src/api/types.jl
@@ -409,6 +409,8 @@ end
# timestamp(tz) column into a Date field: the UTC date, as for DateTime fields
pg_parse_date_any(s::AbstractString)::Date = occursin(':', s) ? Date(pg_parse_datetime_any(s)) : pg_parse_date(s)
+# timestamp(tz) column into a Time field: the UTC time of day
+pg_parse_time_any(s::AbstractString)::Time = occursin(' ', s) ? convert(Time, pg_parse_timestamp(s)) : pg_parse_time(s)
# Integer fields also read integral numeric text such as "12.00" (numeric
# columns, or SUM over one). Out-of-range values throw InexactError, as when
@@ -423,7 +425,13 @@ function pg_parse_integer(::Type{T}, s::AbstractString) where {T <: Integer}
wide = Parsers.tryparse(Int128, SubString(s, 1, dot - 1))
end
end
- wide === nothing && return Parsers.parse(T, s)
+ if wide === nothing
+ # float4/float8 columns print large values in exponent form ("1e+16");
+ # numeric never does, so this conversion is exact
+ f = (occursin('e', s) || occursin('E', s)) ? Parsers.tryparse(Float64, s) : nothing
+ f === nothing && return Parsers.parse(T, s)
+ return convert(T, f)
+ end
return convert(T, wide)
end
@@ -1003,7 +1011,7 @@ const TextField = Union{String, Symbol, Bool, Char, Int8, Int16, Int32, Int64, C
Date, Time, DateTime, UUID, Vector{UInt8}, JSONType, Vector{String}, Vector{Int16}, Vector{Int32},
Vector{Int64}, Vector{Float32}, Vector{Float64}, Vector{Bool}, Vector{Date}, Vector{Time},
Vector{DateTime}, Vector{UUID}, Vector{Char}}
-_text_field(::Type{S}) where {S} = S !== Union{} && (S <: Union{TextField, Enum} || S === IntervalType || _wire_field(S))
+_text_field(::Type{S}) where {S} = S === IntervalType || (S !== Union{} && !(S isa Union) && (S <: Union{TextField, Enum} || _wire_field(S)))
function _field_source(::PostgresStyle, ::Type{T}, v::FieldValue) where {T}
S = Base.nonnothingtype(Base.nonmissingtype(T))
@@ -1056,7 +1064,7 @@ StructUtils.lift(::AbstractPostgresStyle, ::Type{Cuint}, s::String) = pg_parse_i
StructUtils.lift(::AbstractPostgresStyle, ::Type{Float32}, s::String) = Parsers.parse(Float32, s), nothing
StructUtils.lift(::AbstractPostgresStyle, ::Type{Float64}, s::String) = Parsers.parse(Float64, s), nothing
StructUtils.lift(::AbstractPostgresStyle, ::Type{Date}, s::String) = pg_parse_date_any(s), nothing
-StructUtils.lift(::AbstractPostgresStyle, ::Type{Time}, s::String) = pg_parse_time(s), nothing
+StructUtils.lift(::AbstractPostgresStyle, ::Type{Time}, s::String) = pg_parse_time_any(s), nothing
StructUtils.lift(::AbstractPostgresStyle, ::Type{DateTime}, s::String) = pg_parse_datetime_any(s), nothing
StructUtils.lift(::AbstractPostgresStyle, ::Type{UUID}, s::String) = UUID(s), nothing
StructUtils.lift(::AbstractPostgresStyle, ::Type{IntervalType}, s::String) = parse_interval(s), nothing
--- a/src/array_parsing.jl
+++ b/src/array_parsing.jl
@@ -4,7 +4,7 @@ using Parsers, Dates, UUIDs
import Durations, DataDecimals
import ..parse_decimal
import ..pg_parse_timestamp
-import ..pg_parse_date, ..pg_parse_date_any, ..pg_parse_time, ..pg_parse_datetime_any, ..pg_parse_char, ..pg_parse_integer
+import ..pg_parse_date, ..pg_parse_date_any, ..pg_parse_time, ..pg_parse_time_any, ..pg_parse_datetime_any, ..pg_parse_char, ..pg_parse_integer
const BRACKET_OPEN = UInt8('[')
const BRACKET_CLOSE = UInt8(']')
@@ -94,7 +94,7 @@ function parse_scalar(token::String, inner_type::Type{T}, quoted::Bool) where {T
inner_type === Float32 && return Parsers.parse(Float32, token)
inner_type === Float64 && return Parsers.parse(Float64, token)
inner_type === Date && return pg_parse_date_any(token)
- inner_type === Time && return pg_parse_time(token)
+ inner_type === Time && return pg_parse_time_any(token)
inner_type <: Durations.Timestamp && return pg_parse_timestamp(token, inner_type)
inner_type === DateTime && return pg_parse_datetime_any(token)
inner_type === UUID && return UUID(token)Smaller differences, probably fine either way:
AI use: the edge-case matrix, the suggested fix and the test runs were done with Claude Code (Claude Opus 5.5). I reviewed the results and this comment before posting. |
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Use the source floating precision for numeric conversions, preserve union selection, and extract UTC times from timestamps. Cover scalar, nullable, and array reads against PostgreSQL and in the compiled workload. Recognize current and legacy trim diagnostics, check their reported totals, and capture the complete compiler output through a pipe. Keep the existing stdlib exemptions and verify the executable against a live database.
A Vector{UInt8} field over an integer array column decoded the array
text as bytea, so {1,2} read as the bytes of "{1,2}". It converts
element by element again, as on main.
docs/src/support.md still said --trim was unsupported; it now says what
the trim test checks and why --trim=safe still fails on stock Julia.
Date parameters pad with string(n; pad), the trim test runner uses a
Timer instead of a polling loop, and the live typed-read test drops
session settings that do not affect its values. Version 2.4.0: main
already released 2.3.1, and typed reads change behavior.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts: # Project.toml
|
@PingoLee thanks for the comparison matrix. The three regressions you found are fixed in the merged version, with tests against a live server: float columns parse at their own precision before converting, general unions keep the OID decoder, and Review found one more case: an integer array column read into a AI disclosure: Claude Code reviewed and fixed the PR and drafted this comment. |
Typed reads with the default
PostgresStylenow parse each field from the column text by its declared type instead of going through the OID decoder, which returnsAny. Fields of the common types (strings, symbols, enums, integers, floats,Bool,Char,Date,Time,DateTime,UUID, bytea, lazy JSON, timestamps, decimals, vectors of these, and theirNothing/Missingunions) decode without runtime dispatch, so typed reads work injuliac --trimbuilds. Decoding is also faster: a 10-column typed row takes about a quarter of the time and half the allocations it did onmain.Conversions that worked before still work: integer fields read integral
numerictext and float columns printed in exponent form (1e+15), afloat4read intoFloat64keeps itsFloat32value,DateandTimefields over timestamp columns get the UTC date or time of day, integer arrays convert intoVector{UInt8}, and a registered parser still applies when it returns the field's type. General unions and custom styles keep the OID decoder.Behavior changes:
Stringfield receives any column's text, sojson/jsonbcolumns read as JSON text (this threw before).boolcolumns throw (they read 1 or 0).nullin aUnion{Nothing, String}field reads as"null"; only SQL NULL becomesnothing.Also for
--trim: the startup message and tuple parameters no longer need runtime dispatch, andDate/DateTimeparameters use a static formatter with the same output asstring(x).test/trim_compile_tests.jlrestores the JuliaC trim test. It compiles a workload of typed reads, prepared statements, transactions, and parameters, fails on any Postgres.jl verifier error, and runs the executable against PostgreSQL with trust authentication. Released Julia's Base64 (SCRAM) andDates.CompoundPeriod(intervals) are not trim-safe; those errors are reported but not counted, so--trim=safestill rejects the build.docs/src/support.mdsays so.Thanks to @PingoLee for the compatibility matrix that found the float, union, and
Timeregressions in the first version of this PR.Version 2.4.0.
Co-authored by Codex
🤖 Generated with Claude Code