Skip to content

Commit 3353560

Browse files
zzylolclaude
andcommitted
doc: document accumulator lifecycle and ownership in worker section
Adds an 'Accumulator lifecycle and ownership' subsection to §3.4 (Worker) explaining the three-level lazy initialisation: configs copied to all workers at startup, AggregationState created on first sample per series, AccumulatorUpdater created on first sample per pane. Includes the full ownership hierarchy diagram and the deterministic-hash ownership guarantee. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
1 parent e2f5f46 commit 3353560

1 file changed

Lines changed: 43 additions & 0 deletions

File tree

asap-query-engine/src/precompute_engine/precompute_engine_design_doc.md

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -150,6 +150,49 @@ struct AggregationState {
150150
}
151151
```
152152

153+
#### Accumulator lifecycle and ownership
154+
155+
Accumulators are not pre-assigned — they are created **lazily** at three nested levels:
156+
157+
**1. At engine startup** (`engine.rs`): every worker receives a full copy of all `AggregationConfig`s. All workers are symmetric; none is pre-assigned to any series or config.
158+
159+
```rust
160+
let agg_configs = streaming_config.get_all_aggregation_configs().clone();
161+
for (id, rx) in receivers {
162+
Worker::new(id, rx, sink.clone(), agg_configs.clone(), ...)
163+
}
164+
```
165+
166+
**2. On first sample for a series** (`get_or_create_series_state`): the worker calls `matching_agg_configs(series_key)` to filter the config map by metric name, then creates one `AggregationState` per match (a `WindowManager` + empty pane map). No accumulators exist yet.
167+
168+
```rust
169+
let aggregations = matching_agg_configs(series_key).map(|(_, config)| AggregationState {
170+
window_manager: WindowManager::new(config.window_size, config.slide_interval),
171+
config: config.clone(),
172+
active_panes: BTreeMap::new(), // ← empty; no memory allocated for sketches yet
173+
}).collect();
174+
```
175+
176+
**3. On first sample in a pane** (`process_samples`): the accumulator is created the moment a sample falls into a pane that does not yet exist in `active_panes`.
177+
178+
```rust
179+
let updater = agg_state.active_panes
180+
.entry(pane_start)
181+
.or_insert_with(|| create_accumulator_updater(&agg_state.config));
182+
```
183+
184+
**Ownership hierarchy:**
185+
186+
```
187+
Worker
188+
└── series_map[series_key] one entry per series this worker owns
189+
└── aggregations[i] one AggregationState per matching config
190+
└── active_panes[pane_start] one AccumulatorUpdater per open pane
191+
└── Box<dyn AggregateCore> the actual sketch / sum / minmax / etc.
192+
```
193+
194+
Because `xxhash64(series_key) % N` is deterministic, a series always lands on the same worker. Its accumulators live in exactly one worker with no sharing and no locking. Workers that never receive a series never allocate any state for it.
195+
153196
#### Pane-Based Sliding Window Optimization
154197

155198
The worker uses **pane-based incremental computation** to reduce per-sample

0 commit comments

Comments
 (0)