fix(cli): enrich-agencies reports distinct rows, not upsert counts (#142) - #202
Merged
Conversation
) `tb enrich-agencies` printed the number of upserts ISSUED as though it were the table's contents. The tables dedup on `(buyer_id, native_ref)` and the award line key -- a report and its amendment collapse into one row -- so a reader who saw "107 solicitations" went looking and found 74. The leading figures are now distinct ROWS, from two real row-count queries. #177 already made the DELTA honest this way; #142 is the same correction applied to the totals beside it. The upsert counts are kept, in a trailing bracket that names them: "0 upserts" and "107 upserts that all deduped" are very different runs and otherwise look identical from a flat (+0). `_source_row_counts` gains solicitations (it counted awards/bids only), so all three figures come from the same source-scoped query. Verified against a copy of the production store -- every printed figure matches the DB, checked independently: trca stored : 225 solicitations (+0), 439 awards (+0), 408 bids (+0) [upserts 446/693/527] zoo stored : 46 solicitations (+0), 53 awards (+0) [upserts 58/58] ep stored : 74 solicitations (+0), 88 awards (+0), 115 bids (+0) [upserts 107/107/115] EP is the 107-vs-74 case from the issue. The gap turned out wider than reported: TRCA was claiming 446 solicitations for a table holding 225. 778 tests passing (2 new, 4 updated to the widened 3-tuple contract). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W53WHx8mm2UHuLFAQWeF62
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #142.
The bug
tb enrich-agenciesprinted the number of upserts issued as though it were the table'scontents. The tables dedup on
(buyer_id, native_ref)and the award line key — a report and itsamendment, or two items sharing a fallback ref, collapse into one row — so a reader who saw
107 solicitationswent looking in the DB and found 74.The dedup is correct and intended. Only the messaging was wrong.
The fix
Took the issue's first option: report distinct rows. The leading figures now come from two
real row-count queries. #177 already made the delta honest this way; this is the same
correction applied to the totals sitting beside it.
The upsert counts are kept, in a trailing bracket that names them for what they are —
0 upsertsand107 upserts that all dedupedare very different runs, and a flat(+0)alonecannot tell them apart. That diagnostic would have been lost by simply swapping the numbers out.
_source_row_countsgains solicitations (it counted awards/bids only), so all three figures comefrom the same source-scoped query rather than mixing sources.
Verification
Run against a copy of the production store, with every figure checked independently against the
DB:
Exact match on all nine figures. EP is the 107-vs-74 case from the issue — and the gap turned
out wider than reported: TRCA was claiming 446 solicitations for a table holding 225.
778 tests passing — 2 new (distinct-rows-not-upserts; the dedup that causes the gap, pinned
directly by upserting two rows under one
native_refand asserting one row results), and 4updated to the widened 3-tuple
_source_row_countscontract.🤖 Generated with Claude Code
https://claude.ai/code/session_01W53WHx8mm2UHuLFAQWeF62