Skip to content

Feature/sc 46788/modify abstractmongoset to have a method - #3649

Draft
stevekaplan123 wants to merge 22 commits into
masterfrom
feature/sc-46788/modify-abstractmongoset-to-have-a-method
Draft

Feature/sc 46788/modify abstractmongoset to have a method#3649
stevekaplan123 wants to merge 22 commits into
masterfrom
feature/sc-46788/modify-abstractmongoset-to-have-a-method

Conversation

@stevekaplan123

@stevekaplan123 stevekaplan123 commented Aug 21, 2026

Copy link
Copy Markdown
Member

DO NOT MERGE UNTIL #3648 is merged

Description

This builds on previous work to handle bad data on server start and reset of cache. The startup guards wrap the body of each loop in startup/reset of cache, but AbstractMongoSet._read_records() builds every record in one pass the moment the set is touched:

for i in IndexSet(): # ← all 6,584 Index objects built HERE
with skip_bad_record(...): # ← guard only starts HERE
So one bad document aborts the whole set and the loop body never runs even once.

Code Changes

The new function, AbstractMongoSet.with_skip_guard(), stores a guard on the set and returns self; The existing _read_records() is now modified so that it guards each record as it's constructed. Applied at the five guarded build sites that iterate a set. The solution is to do this:

for i in IndexSet().with_skip_guard(skip_bad_record, "reset_cache,startup",
                                    "_build_index_maps index record", level="error"):
    with skip_bad_record(...):   # still needed — guards USING the record, not building it

Note: _read_records() is called implicitly when iterated and is called in 9 other ways (array(), len(), contents(), …).
I didn't use an init flag, because pathway/operation are required, not a bare flag: they group the Slack summary and key the skip-tracking breakers.

Note 2: Two new hooks: _instantiate_record() and _post_read_records() are both called from _read_records and they make record-building overridable. LexiconEntrySet and TopicSet override these methods so that they can modify the way records are loaded and do post-processing.

stevekaplan123 and others added 22 commits August 5, 2026 10:01
Replies to yitzhakc's two review comments on PR #3648:

* Route every line the audit script emits through a module logger instead
  of bare print(). Bare %(message)s formatter (the report is a fixed-width
  table), propagate=False so Django's LOGGING config doesn't duplicate or
  file the lines, and a --log-level flag. Progress and the report are INFO;
  cases that behaved differently than predicted, and harness cleanup
  failures, are WARNING — so --log-level WARNING reduces a run to just what
  went wrong.

* Document what a "site" is and what "S1" means: a SITE IDS section in the
  module docstring (which is also the --help text), a note on the case
  table, and the site/operation/corruption params in case()'s docstring.
  The ids are this script's own labels; `operation` is the string that
  identifies the real guard.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…e/sc-46788/modify-abstractmongoset-to-have-a-method
…e/sc-46788/modify-abstractmongoset-to-have-a-method
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant