Skip to content

perf(arrow/ipc): release dictionary payloads eagerly - #1259

Open
fallintoplace wants to merge 2 commits into
apache:mainfrom
fallintoplace:perf/arrow-ipc-dictionary-payload-lifetime
Open

perf(arrow/ipc): release dictionary payloads eagerly#1259
fallintoplace wants to merge 2 commits into
apache:mainfrom
fallintoplace:perf/arrow-ipc-dictionary-payload-lifetime

Conversation

@fallintoplace

Copy link
Copy Markdown
Contributor

What

  • Release each dictionary payload as soon as WritePayload returns.
  • Keep the optional dictionary delta slice in the same short-lived scope.
  • This keeps encoded dictionary payloads from accumulating until writeDictionaryPayloads returns.

Benchmark

The benchmark uses a tracking allocator and reports peak bytes held by dictionary payloads beyond the input record.

Apple M1 Pro, Go 1.26.3, 1 CPU, 150 ms, 7 runs:

Dictionary columns Before After
8 1,536 B 192 B
64 12,288 B 192 B
256 49,152 B 192 B

Tests

  • go test ./arrow/ipc
  • PARQUET_TEST_DATA="$(pwd)/parquet-testing/data" go test ./...
  • go vet ./arrow/ipc

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant