Servos backfill - #392
Merged
Merged
Servos backfill#392
Conversation
Servos only ever affected attributes indexed after they were enabled. This adds the missing half: re-running the ingest chain over what is already there. It is chain-wide rather than per-servo, which is not what the original plan assumed. `_update_by_query` re-runs the index's *final* pipeline whatever pipeline is named on the request -- verified on OpenSearch 3.4 -- and on misp-attributes that pipeline is geoip plus every enabled servo. Running one servo in isolation is therefore not possible, so the API and the audit entry say chain-wide instead of promising isolation they cannot deliver. A Lucene filter scopes which attributes are touched. That same ordering is what makes the run reportable. A named pipeline runs *before* the final one, so the request names a new repo-managed pipeline whose only job is to remove expanded.servo_errors. The field is an append, so without it every backfill would stack another copy of the same error; with it each run reports only its own failures. Confirmed by backfilling twice against a deliberately failing servo and finding one error entry, not two. Runs are recorded because a backfill rewrites live documents and "what did we touch, and when" is the question asked afterwards. OpenSearch executes the update asynchronously, so the row carries its task id and the Celery task polls it -- but reading a run reconciles it against OpenSearch too, which makes the engine the source of truth and the worker's polling an optimisation. A run left unclaimed past five minutes is called failed rather than sitting at "queued" for ever, which is what happens when the worker is down; found by hitting it. Gated on a new servos:run scope, separate from servos:update, so rewriting the index is not implied by being allowed to edit a servo. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A third tab on the servos page: a Lucene filter, a preview of how many attributes it matches and which servos would run, a confirm step that repeats both, and the run history. The preview is not decoration. A backfill rewrites live documents in place and cannot be undone, so the count is shown before the operator commits rather than discovered afterwards, and the confirm dialog says "EVERY attribute in the index" when the filter is empty rather than quietly meaning it. A servo's red error badge is now a link into that tab, prefilled with `expanded.servo_errors:servo_<slug>*` -- re-run the chain over just the documents that servo failed on. That is the one useful way to scope a backfill to a servo, given the chain always runs whole. Tests cover the parts that were reasoned about rather than obvious: that the request names the reset pipeline and does not block, that a running task reads its counters from task.status and a finished one from response, that a task OpenSearch has forgotten completes instead of polling for ever, that a filter OpenSearch cannot parse is a 422 and queues nothing, and that servos:update does not grant servos:run. servo_runs is cleared in the test teardown: its user FK is ON DELETE SET NULL, so the rows outlive the user wipe and would leak between test classes. The openapi spec is regenerated for the three new endpoints, matching the committed file's formatting so the diff is the 307 lines that changed rather than the whole document. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
All five servo screenshots are regenerated so the tab strip shows Backfill alongside the other two, and the new capture shows the panel doing its job: a filter checked, the count and servo list an operator sees before confirming, and a run history with one running, one succeeded and one failed run rather than only the happy path. Taking the capture found the Backfill badge showing 5 when there were 3 runs: an earlier edit to tabCount never applied -- eslint had rewrapped the line the patch anchored on -- so the tab fell through to the system-pipeline count. The spec now asserts the badge text, so it cannot go wrong again quietly. Adding a second table to the page also broke two specs that counted `table tbody tr` globally: the panels are v-show, so both tables are in the DOM at once and the count was servos plus runs. The panels now carry data-tab-panel, matching the event view, and the locators name the panel they mean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
OpenAPI changesAdded endpoints
|
The preview says how many attributes a filter would rewrite; it could not say which. The link hands the same Lucene filter to Explore (`/explore?q=`, the contract the hunt view's "Open in Explore" already uses) in a new tab, so the documents can be inspected before they are rewritten in place. An empty filter — every attribute — maps to `q=*`. The capture suite also stubs the unread-notifications count now: the nav badge reflected whatever the instance held, and a demo-seed count leaked into the regenerated screenshot. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #392 +/- ##
==========================================
- Coverage 84.43% 84.33% -0.10%
==========================================
Files 208 208
Lines 19138 19391 +253
==========================================
+ Hits 16159 16354 +195
- Misses 2979 3037 +58 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.