Feature/sc 46788/modify abstractmongoset to have a method - #3649
Draft
stevekaplan123 wants to merge 22 commits into
Draft
Feature/sc 46788/modify abstractmongoset to have a method#3649stevekaplan123 wants to merge 22 commits into
stevekaplan123 wants to merge 22 commits into
Conversation
Replies to yitzhakc's two review comments on PR #3648: * Route every line the audit script emits through a module logger instead of bare print(). Bare %(message)s formatter (the report is a fixed-width table), propagate=False so Django's LOGGING config doesn't duplicate or file the lines, and a --log-level flag. Progress and the report are INFO; cases that behaved differently than predicted, and harness cleanup failures, are WARNING — so --log-level WARNING reduces a run to just what went wrong. * Document what a "site" is and what "S1" means: a SITE IDS section in the module docstring (which is also the --help text), a note on the case table, and the site/operation/corruption params in case()'s docstring. The ids are this script's own labels; `operation` is the string that identifies the real guard. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…e/sc-46788/modify-abstractmongoset-to-have-a-method
…e/sc-46788/modify-abstractmongoset-to-have-a-method
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
DO NOT MERGE UNTIL #3648 is merged
Description
This builds on previous work to handle bad data on server start and reset of cache. The startup guards wrap the body of each loop in startup/reset of cache, but AbstractMongoSet._read_records() builds every record in one pass the moment the set is touched:
for i in IndexSet(): # ← all 6,584 Index objects built HERE
with skip_bad_record(...): # ← guard only starts HERE
So one bad document aborts the whole set and the loop body never runs even once.
Code Changes
The new function, AbstractMongoSet.with_skip_guard(), stores a guard on the set and returns self; The existing
_read_records()is now modified so that it guards each record as it's constructed. Applied at the five guarded build sites that iterate a set. The solution is to do this:Note:
_read_records()is called implicitly when iterated and is called in 9 other ways (array(), len(), contents(), …).I didn't use an init flag, because pathway/operation are required, not a bare flag: they group the Slack summary and key the skip-tracking breakers.
Note 2: Two new hooks: _instantiate_record() and _post_read_records() are both called from
_read_recordsand they make record-building overridable. LexiconEntrySet and TopicSet override these methods so that they can modify the way records are loaded and do post-processing.