Count Firestore query reads in the document read probe - #12281
Open
Git-on-my-level wants to merge 1 commit into
Open
Count Firestore query reads in the document read probe#12281Git-on-my-level wants to merge 1 commit into
Git-on-my-level wants to merge 1 commit into
Conversation
The probe wrapped only DocumentReference.get and Client.get_all, so it covered the LOOKUP path -- 8.6% of the read bill. QUERY is 91.4%, and ~428M docs/day of it is issued by code no metric names. Wrap Query.stream and AggregationQuery.stream instead of annotating ~307 call sites. Query.get and CollectionReference.get/stream funnel through Query.stream, so wrapping them too would double count. Aggregations bill one read per batch of up to 1000 index entries, not one per matched document, so charge the batches. Streams stay lazy and labels still reduce through collection_pattern(), so cardinality and caller memory profiles are unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
kodjima33
approved these changes
Aug 27, 2026
kodjima33
left a comment
Collaborator
There was a problem hiding this comment.
Solid instrumentation work, but this is an attribution enabler (feature), not a bug fix, so per policy it's approve-only regardless of CI.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Extends the existing Firestore read probe to count query reads, not just
single-document lookups. No call sites change.
Why
Cloud Firestore Read Opsis ~$538/day, and 91.4% of it isQUERYtype(
1,039M docs/day). The instrument that covers QUERY today accounts for 596.5M$210/day) is issued by code no metric names.**docs/day, and 587.1M of that is a single already-owned family. Netting out,
roughly **428M docs/day (
The probe merged in
firestore_document_probe.pywraps exactlyDocumentReference.getandClient.get_all— the LOOKUP / BatchGetDocumentspath. Its "structural rather than remembered" coverage claim therefore holds
only over the cheap 8.6% slice; the expensive 91.4% stays dark. There are ~307
.stream()call sites inbackend/**.pyand 5 record a family, so annotatingby hand is not a plan.
This is an attribution enabler, not a cut. It sizes and locates a cut. Precedent:
action_items_listwas invisible until instrumented and turned out to be$278/day.
SDK surface wrapped
Verified against the installed
google-cloud-firestore2.20.0 in the venv ratherthan assumed:
Query.streamAggregationQuery.stream.count()is used across conversations, action_items, chat, apps, folders, daily_summaries, x_posts..get()materialises.stream().Deliberately not wrapped, to avoid double counting:
Query.get— already callsself.stream()thenlist().CollectionReference.get/.stream— both funnel throughQuery.stream.BaseCollectionReference.get/streamareNotImplementedErrorstubs.AsyncQuery/AsyncAggregationQuery— zero uses inbackend/.VectorQuery—find_nearest()is never called here.CollectionGroupsubclassesQuery, so collection-group streams are covered.Transaction.get(query)callsquery.stream(), so it is covered too.Aggregations are billed by index-entry batch, not by matched document
Firestore charges an aggregation one read per batch of up to 1000 index
entries, not one read per matched document. Counting matched documents would
overstate a large
count()by up to 1000x and let cheap aggregations dominate acounter whose entire purpose is attributing the billed read line. The probe
charges
ceil(matched / 1000), floored at 1, and the tests pin that at the1000/1001 boundary and at 2.5M matched.
Invariants held
collection_pattern()/_KNOWN_PATTERNS/other,so the label space is unchanged and no uid, document id, or query text can
reach a label.
a list, so a caller's memory profile on a large result set is unchanged. This
follows the existing
get_allprecedent.cannot break a read.
_installedand still no-ops onImportErrorfor the unit-test stubs.Known type change
As with the existing
get_allwrapper, a wrapped.stream()returns a plaingenerator rather than
StreamGenerator. Iteration is unchanged;get_explain_metrics()would not survive, and this codebase never passesexplain_options.Testing
backend/tests/unit/test_firestore_document_probe.py— 11 passed. Covers:per-document counting with the right label, laziness (the underlying generator is
not consumed before the caller pulls), unknown collection reducing to
other, araising recorder not propagating, aggregation batch billing, and idempotent
install.
Neighbouring suites were run file-by-file and match
origin/mainexactly:test_firestore_read_site_attribution11 passed,test_firestore_query_stream_retry7 passed,
test_firestore_di_seam5 passed.test_bounded_firestore_list_reads(2 failed) and
test_firestore_query_contract(collection error) fail identicallyon
origin/mainand are untouched by this change. Note that running the whole-k "firestore or probe"selection in one process segfaults onorigin/maintoo;that is pre-existing and unrelated.
Failure-Class: none