feat: memory + on-disk size reporting#155
Draft
tellet-q wants to merge 1 commit into
Draft
Conversation
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adds post-indexing memory and on-disk size reporting by querying Qdrant REST endpoints (/collections/{c}/memory and /telemetry), printing a human-readable summary, and embedding the sampled data into the benchmark results JSON.
Changes:
- Introduces a new
memorymodule with REST wire types, fetching logic, printing, and a parsing fixture-backed test. - Extends results JSON (
PhaseResults) to optionally include amemoryreport, and populates it after indexing in upload/legacy benchmark flows. - Documents the new reporting in
README.mdand updates developer docs to include the new module.
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/fixtures/collection_memory.json | Adds a real-server fixture used to validate REST response parsing for memory reporting. |
| src/memory.rs | New module: defines report structs, fetches REST endpoints, prints summary, and tests parsing against fixture. |
| src/main.rs | Fetches and attaches memory report after indexing (best-effort, REST-only extras). |
| src/results.rs | Adds results.memory: Option<MemoryReport> to the serialized results schema and updates tests. |
| src/args/mod.rs | Updates --skip-server-stats help text to include memory/disk reporting. |
| README.md | Documents the new “Memory & disk reporting” output and JSON location. |
| DEVELOPMENT.md | Lists the new memory.rs module in the source layout. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+148
to
+155
| let (rest_uri, collection) = (rest_uri(args), args.collection_name.clone()); | ||
| let api_key = std::env::var("QDRANT_API_KEY").ok(); | ||
|
|
||
| // `ureq` is blocking; keep it off the async worker threads. | ||
| let fetched = tokio::task::spawn_blocking(move || { | ||
| memory::fetch(&rest_uri, &collection, api_key.as_deref()) | ||
| }) | ||
| .await; |
Comment on lines
+148
to
+155
| pub fn fetch(rest_url: &str, collection: &str, api_key: Option<&str>) -> Result<MemoryReport> { | ||
| let base = rest_url.trim_end_matches('/'); | ||
| let memory: CollectionMemory = | ||
| get(&format!("{base}/collections/{collection}/memory"), api_key)?; | ||
| let process = get::<Telemetry>(&format!("{base}/telemetry?details_level=1"), api_key) | ||
| .ok() | ||
| .and_then(|telemetry| telemetry.memory); | ||
|
|
Comment on lines
+167
to
+184
| fn get<T: serde::de::DeserializeOwned>(url: &str, api_key: Option<&str>) -> Result<T> { | ||
| let agent = ureq::Agent::new_with_defaults(); | ||
| let mut request = agent.get(url); | ||
| if let Some(key) = api_key { | ||
| request = request.header("api-key", key); | ||
| } | ||
|
|
||
| let body = request | ||
| .call() | ||
| .with_context(|| format!("failed to GET {url}"))? | ||
| .into_body() | ||
| .read_to_string() | ||
| .with_context(|| format!("failed to read response from {url}"))?; | ||
|
|
||
| let envelope: RestEnvelope<T> = | ||
| serde_json::from_str(&body).with_context(|| format!("failed to parse {url}"))?; | ||
| Ok(envelope.result) | ||
| } |
Comment on lines
+74
to
+87
| for vector in self.vectors.iter().chain(&self.sparse_vectors) { | ||
| let name = if vector.name.is_empty() { | ||
| "(default)" | ||
| } else { | ||
| &vector.name | ||
| }; | ||
| println!( | ||
| "vector {name}: storage {} disk / {} ram, index {} disk / {} ram", | ||
| mib(vector.storage.disk_bytes), | ||
| mib(vector.storage.ram_bytes), | ||
| mib(vector.index.disk_bytes), | ||
| mib(vector.index.ram_bytes), | ||
| ); | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Note: base is not dev! Only merge after #152
Sample Qdrant's memory/disk telemetry after indexing (REST-only, like the optimization stages) and report it under results.memory in --json:
cachedis what the page cache holds of the on-disk bytes vs. the expected a component wants cached to serve queries without hitting disk — the gap shows how cold the cache is. Runs automatically in bfb upload and legacy runs; gated off by the existing --skip-server-stats. Parsing is fixture-tested against a real server response.