support saving Strings of variable length - #603
Open
bjarthur wants to merge 1 commit into
Open
Conversation
Author
|
I think the purpose of the code is merely to produce an estimate to try to optimize the size of the copy buffer. While the current method is an improvement over an error, it is not really estimating the actual size of what needs to be copied. |
|
I sent a pull request to your branch to try to get an actual estimate of the string sizes: |
Author
|
yuck, ok, uglier than i expected. so, maybe it's easier then to just advise the user to extend YAXArrays. this works for me. maybe document somewhere instead of merging this PR? |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
claude did the heavy lifting here. let me know what you think. works in practice for me and very useful.
Summary
Saving a cube whose element type has no definite size (e.g.
String) currentlycrashes:
ERROR: Type String does not have a definite size.
Stacktrace:
[1] sizeof(x::Type) @ Base ./essentials.jl:783
[2] get_copy_buffer_size(...) @ YAXArrays.Cubes .../Cubes/Rechunker.jl:55
[3] copy_diskarray(...) @ YAXArrays.Cubes .../Cubes/Rechunker.jl:108
[4] copydataset!(...) @ YAXArrays.Datasets .../DatasetAPI/Datasets.jl:635
...
[8] savecube(...) @ YAXArrays.Datasets .../DatasetAPI/Datasets.jl:813
The backend variable is created fine — the only blocker is the buffer-size
heuristic in the rechunker, which calls
sizeof(eltype(incube))directly.sizeofthrows for any non-isbitstypeelement type, since such values arestored by reference rather than inline.
Changes
Cubes._elsize(T): likesizeof, but returns a pointer-sized fallback(
sizeof(Ptr{Cvoid})) for element types without a definite size. The copybuffer holds element references, so a pointer-sized footprint is the right
estimate for cache budgeting, and it's safely bounded by the existing
min(maxbuf, length)/maxbuf > prod(insize)guards inget_copy_buffer_size._elsizeinget_copy_buffer_size(the reported crash) and incubesize, which had the same latentsizeof(T)issue.DAT.jlto call_elsize:mysizeof(previously a special-casedmysizeof(::Type{String}) = 1)getbackend, whereelementtype = Union{oc.outtype,Missing}would alsothrow for a string output type
getCacheSizes(sizeof(C.outtype))Tests
Adds a
"Saving variable-length strings"testset that round-trips aYAXArray{String}through both the Zarr and NetCDF backends, plus unit checkson
_elsizeitself. Verified passing locally.Notes
show_afterdisplay path is intentionally left unchanged — it alreadybranches on
isbitstype/Unionand usesBase.summarysizeforvariable-length element types, which is more accurate for display than an
estimate.