Skip to content

perf(results): pack typed columns while the driver streams rows (#1552) - #1719

Merged
ZhuchkaTriplesix merged 1 commit into
devfrom
perf/redis-tree-sort-decorated
Oct 18, 2026
Merged

ZhuchkaTriplesix merged 1 commit into
devfrom
perf/redis-tree-sort-decorated

Conversation

@ZhuchkaTriplesix

Copy link
Copy Markdown
Member

Closes #1552

PostgreSQL copied every row with row.toList(), SQLite built a list per row, MySQL kept all ResultSetRows and then read each with colAt; CompactResultDataset.fromRawRows then walked the whole list once for the kinds and once per column to pack.

ResultColumnsBuilder (core/database/result_columns_builder.dart) takes the rows as they arrive (addRow, or addCell + endRow for drivers whose rows are not lists) and learns each column's kind on the way. Rules unchanged: all ints int64, all numbers float64, all strings string, all NULL string, else generic; the first 60 non-null cells still go through the driver formatter (#1548). A column can be demoted late (ints that meet a double, anything that meets a string), so ints and doubles are kept in separate buffers: a column that ends up generic still holds its ints as ints, exactly as before. fromRawRows now feeds the builder, so every existing dataset test covers it. MySQL uses a new forEachThenDrain (the drain-after-cap behaviour of takeThenDrain stays, which is built on it).

Tests: kinds and raw values for mixed, late-demoted and all-NULL columns, short rows, cell-by-cell vs whole rows, the formatter case, and 150 random columns checked against the kind rules (some beyond the 256-row first buffer). All of test/core/database, mysql, postgresql, sqlite and workspace tests pass locally.

Measured (JIT test runner, 50k x 20 mixed columns, 3 runs each; rows already in memory as the driver holds them): old path (copy rows, then pack) 124-135 ms, peak RSS over baseline 26-40 MiB; builder 118-121 ms, 21-25 MiB. 5k rows: 16-20 ms vs 17-22 ms, no difference outside noise. So the gain is the intermediate list's memory (about half the peak at 50k) and a small time saving; I did not measure against live servers.

…nstead of collecting List<List<Object?>> first (#1552)
@github-actions github-actions Bot added enhancement New feature or request performance Theme parser epic label: performance core Core library logic and services backend Backend database driver execution and queries P3 Low priority / Polish & Enhancements labels Oct 18, 2026
@ZhuchkaTriplesix
ZhuchkaTriplesix merged commit 868c7b1 into dev Oct 18, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backend Backend database driver execution and queries core Core library logic and services enhancement New feature or request P3 Low priority / Polish & Enhancements performance Theme parser epic label: performance

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant