Streaming memory issues and allocation hotspots (loadChunk quadratic growth, GetSnippet bug, Stream.String allocations, token retention)

[bzick]

Follow-up to the hot-path optimizations in #46. CPU/memory profiling (pprof + runtime.ReadMemStats, Apple M1 Max, go1.26, benchmarks from bench_test.go) surfaced four remaining problems outside the parse hot path.

1. Quadratic allocations in loadChunk when a single token exceeds the chunk size

Where: parser.go, loadChunk — chunk := make([]byte, len(p.str)+p.chunkSize) + copy(chunk, p.str).

What's wrong: in the normal case checkPoint() trims p.str after every emitted token, so each loadChunk copies only a small tail. But while the parser is inside one token (e.g. a long quoted string arriving via ParseStream), checkPoint never runs: p.str keeps growing and every chunk load reallocates and copies the entire accumulated buffer. Total allocations are O(N²/2C) for a token of size N and chunk size C.

Measured (chunk 4096, one quoted string in a stream):

Token size	Total allocated	Overhead
512 KB	34.6 MB	66×
1 MB	136 MB	130×
2 MB	541 MB	258×

(2MB)²/(2·4096) = 537 MB — matches the model.

Suggested fix: grow the buffer geometrically (e.g. make([]byte, len(p.str)+max(p.chunkSize, len(p.str)))) → amortized O(N). Note: reusing the buffer instead of allocating is not an option — emitted tokens alias p.str (token.value, token.indent), and buffer reuse corrupts already-emitted tokens (verified experimentally: breaks TestInfStream).

2. Bug: GetSnippet fills all "before" tokens with the same token

Where: stream.go, GetSnippet, first loop:

for p := ptr; p != nil; p, before = ptr.prev, before-1 {
    segment[before] = Token{
        id:  ptr.id,
        key: ptr.key,
        ...

What's wrong: the loop declares p but both the post-statement (ptr.prev) and the body read ptr, which never advances. Every "before" element of the snippet is a copy of the same (current) token instead of walking backwards through the list. The "after" loop below is correct (p.next, reads p).

Not a performance issue — an actual correctness bug; needs a fix plus a regression test (current tests don't catch it).

3. Stream.String / Token.String are allocation-heavy

Where: stream.go String() (strconv.Itoa(ptr.id)+": "+ptr.String() per token, then strings.Join), token.go String() (fmt.Sprintf with 6 interface-boxed args).

Measured: ~6 allocations per token; on a ~440-token stream one Stream.String() call = 2 649 allocs / 150 KB. Each output byte is allocated 3–4 times (Sprintf → concat → Join).

Suggested fix: build the whole output with a single strings.Builder + strconv.AppendInt; expected ~orders-of-magnitude fewer allocations. Debug-oriented API, but cheap to fix.

4. Unbounded token retention without SetHistorySize + sync.Pool overhead on Close

Where: stream.go (GoNext/Close), tokenizer.go (allocToken/freeToken, sync.Pool).

What's wrong (measured on a 10 MB / 2.3M-token stream, chunk 4096):

• Without SetHistorySize, all Token structs stay alive until Close: live heap = 269 MB, ~27× the source size (2.3M × 112 B per Token + ~12 MB of pinned chunk buffers). With SetHistorySize(10) live heap is only 0.13 MB.
• Close() then mass-returns millions of tokens to sync.Pool: poolChain.pushHead alone is 33.5% of all alloc_space in the no-history stream benchmark, and live memory temporarily grows after Close (269 → 324 MB) until the GC drains the pool.
• The GC periodically empties the pool anyway, so pool.New still accounts for ~48% of allocated objects in that scenario — the pool churns instead of amortizing.

Suggested directions:

• replace sync.Pool with an intrusive free-list on the Tokenizer (Token already has a next pointer): Close() becomes O(1) — link the whole list into the free-list, no per-token Put;
• optionally allocate tokens in slabs ([]Token batches) to reduce object count and improve locality (a prototype showed ~9-11% CPU gain but needs slab-aware freeing to avoid losing pool reuse);
• at minimum, document that without SetHistorySize memory retention is ~27× the input size.

---

All measurements are reproducible with make bench plus -cpuprofile/-memprofile; benchmark sources live in bench_test.go (added in #46).