Skip to content

support saving Strings of variable length - #603

Open
bjarthur wants to merge 1 commit into
JuliaDataCubes:mainfrom
bjarthur:bja/string
Open

support saving Strings of variable length#603
bjarthur wants to merge 1 commit into
JuliaDataCubes:mainfrom
bjarthur:bja/string

Conversation

@bjarthur

Copy link
Copy Markdown

claude did the heavy lifting here. let me know what you think. works in practice for me and very useful.

Summary

Saving a cube whose element type has no definite size (e.g. String) currently
crashes:

ERROR: Type String does not have a definite size.
Stacktrace:
[1] sizeof(x::Type) @ Base ./essentials.jl:783
[2] get_copy_buffer_size(...) @ YAXArrays.Cubes .../Cubes/Rechunker.jl:55
[3] copy_diskarray(...) @ YAXArrays.Cubes .../Cubes/Rechunker.jl:108
[4] copydataset!(...) @ YAXArrays.Datasets .../DatasetAPI/Datasets.jl:635
...
[8] savecube(...) @ YAXArrays.Datasets .../DatasetAPI/Datasets.jl:813

The backend variable is created fine — the only blocker is the buffer-size
heuristic in the rechunker, which calls sizeof(eltype(incube)) directly.
sizeof throws for any non-isbitstype element type, since such values are
stored by reference rather than inline.

Changes

  • Add Cubes._elsize(T): like sizeof, but returns a pointer-sized fallback
    (sizeof(Ptr{Cvoid})) for element types without a definite size. The copy
    buffer holds element references, so a pointer-sized footprint is the right
    estimate for cache budgeting, and it's safely bounded by the existing
    min(maxbuf, length) / maxbuf > prod(insize) guards in
    get_copy_buffer_size.
  • Use _elsize in get_copy_buffer_size (the reported crash) and in
    cubesize, which had the same latent sizeof(T) issue.
  • Unify the pre-existing ad-hoc workarounds in DAT.jl to call _elsize:
    • mysizeof (previously a special-cased mysizeof(::Type{String}) = 1)
    • getbackend, where elementtype = Union{oc.outtype,Missing} would also
      throw for a string output type
    • getCacheSizes (sizeof(C.outtype))

Tests

Adds a "Saving variable-length strings" testset that round-trips a
YAXArray{String} through both the Zarr and NetCDF backends, plus unit checks
on _elsize itself. Verified passing locally.

Notes

  • The show_after display path is intentionally left unchanged — it already
    branches on isbitstype/Union and uses Base.summarysize for
    variable-length element types, which is more accurate for display than an
    estimate.

@bjarthur

Copy link
Copy Markdown
Author

@mkitti

@mkitti

mkitti commented Jun 25, 2026

Copy link
Copy Markdown

I think the purpose of the code is merely to produce an estimate to try to optimize the size of the copy buffer. While the current method is an improvement over an error, it is not really estimating the actual size of what needs to be copied.

@mkitti

mkitti commented Jun 25, 2026

Copy link
Copy Markdown

I sent a pull request to your branch to try to get an actual estimate of the string sizes:
bjarthur#1

@bjarthur

Copy link
Copy Markdown
Author

yuck, ok, uglier than i expected.

so, maybe it's easier then to just advise the user to extend YAXArrays. this works for me. maybe document somewhere instead of merging this PR?

using YAXArrays.Cubes

struct StringArrayProxy{T, N, A<:AbstractArray} <: AbstractArray{T, N}
    parent::A
end
Base.size(p::StringArrayProxy) = size(p.parent)

function Cubes.get_copy_buffer_size(
    incube::AbstractArray{<:Union{Missing,AbstractString}},
    outcube;
    writefac = 4.0,
    maxbuf = YAXArrays.YAXDefaults.max_cache[],
    align_output = true
)

    string_length = 256  # hard-coded average length of your data
    proxy = StringArrayProxy{NTuple{string_length , UInt8}, ndims(incube), typeof(incube)}(incube)

    return Cubes.get_copy_buffer_size(proxy, outcube; writefac, maxbuf, align_output)
end

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants