fix(ENGKNOW-3933): MAP/MULTIMAP cache key, single shared load and lower memory

[gmagnu]

Summary

Changes to MapAndListUtilities (the loaders behind MAP, MULTIMAP and INSET):

1. Cache-key bug (correctness). The per-request cache key was built as "map"+filename+ic+oc.mkString(",")+asSet (and the same without asSet for multimap).
   • It left out caseInsensitive, so MAP -cis and a plain MAP of the same file in one request shared one cached map, and one of them returned wrong results.
   • It left out skipEmpty, so MAP -e and a plain MAP of the same file shared one cached map, although -e changes how values of duplicate keys are merged.
   • It had no separator between ic and oc, so ic=1, oc=[23] and ic=12, oc=[3] collided.
   • The key is now delimited and includes every parameter that affects the loaded map.
2. Single load per session cache. Concurrent pipelines of a request share the session cache (for example pgor partitions), and each one missed the cache and built its own full copy of the map. Now the first caller registers a CompletableFuture per (session cache, key) and loads; the others close their own iterator and wait for that future.
   • Waiting is interruptible, so a cancelled query does not wait for the load to finish.
   • A failed load fails all of its waiters with the same error, instead of each retrying it in turn. The entry is removed afterwards, so a later call loads again.
   • Loads are keyed on the session cache instance, not the request id, so unrelated sessions that share a request id (e.g. -1) do not wait for each other.
3. Value dedup. Equal value strings share one instance while a map loads. MAP value columns repeat heavily. The pool is capped at 100k distinct values, so maps with mostly unique values don't pay for a large pool. Values merged from duplicate keys are not pooled, as later duplicates replace them.
4. MULTIMAP peak. Entries are moved into a presized output map instead of copied, so the ListBuffer map and the Array map are never both fully held.

Measurements

Heap retained after GC. Synthetic 1M-line files with values drawn from 50 distinct strings.

	before	after
Single map, 1M keys	160.8 MB	98.4 MB (−39%)
Multimap, 200k keys × 5 values, retained	90.4 MB	28.0 MB (−69%)
Multimap, peak during load	350.1 MB	305.9 MB (−13%)

Tests

• New UTestMapAndListUtilities, 10 tests:
  • These fail on main and pass with this change:
    • -cis vs plain map and multimap caching
    • -e (skipEmpty) vs plain map caching
    • the ic/oc key collision
    • concurrent loads reading the file once: main read it 4,000 times with 4 threads instead of 1,000
    • a failed load failing its waiters once: 4 load attempts instead of 1 before the shared future
    • value string sharing for map and multimap, with multimap value order kept
  • These cover the shared load: a waiting caller can be interrupted, and sessions with the same request id but separate caches don't wait for each other. Both failed with the earlier per-request lock in this PR.
• Map tests pass: UTestMapAndListUtilities, UTestGorMapMultimap, UTestDAGMap, UTestMultiMapLookup and the *UTestMapCommand* tests (91 tests). The new class passed 5 repeated runs.
• :model:test passed on the first commit: 1,548 tests, 17 skipped.

🤖 Generated with Claude Code

[github-actions]

Junit Tests - Summary

4 863 tests  +12   4 692 ✅ +13   18m 33s ⏱️ - 2m 39s
  506 suites + 1     171 💤  -  1 
  506 files   + 1       0 ❌ ± 0 

Results for commit 7aaba85. ± Comparison against base commit 9536c2f.

♻️ This comment has been updated with latest results.

[cesarvp-gdx]

Nice!

[cesarvp-gdx]

LGTM!