Skip to content

Offer a rerun on the jobs a request can be faithfully rebuilt from (#55) - #133

Open
r0ny123 wants to merge 14 commits into
familiary:masterfrom
r0ny123:fix/55-rerun-job
Open

r0ny123 wants to merge 14 commits into
familiary:masterfrom
r0ny123:fix/55-rerun-job

Conversation

@r0ny123

@r0ny123 r0ny123 commented Aug 30, 2026 •

Copy link
Copy Markdown
Collaborator

Mirrors r0ny123/mcritweb#30 from a triage series run against df53db9. Bare #N in the text below are upstream issue numbers; links point at the sibling PRs of the series.

Closes #55. Both boxes are implemented now - rerun with the same configuration, and a link to the configuration page with the job's samples preselected.

With the backend's own rerun (familiary/mcrit#311)

The rebuild from parameters below can only resend three requests. mcrit can't be asked for a query against a stored reference, for getMatchesForSampleVsGroup, or for unique blocks with force_recalculation. familiary/mcrit#311 adds POST /jobs/<job_id>/rerun and McritClient.requestJobRerun, which queue a job again from the payload the backend stored, input files included.

Change (b63a060, bfce32b).

  • Where the installed McritClient has requestJobRerun (server_reruns(), asked of the class, so the button and the route agree), the route asks the backend to rerun the job and lands on the new job. When the backend answers None (a job that is queued or running, isn't an analysis, or whose input file is gone), the page stays on the job and says so. Without the method, everything below applies unchanged.
  • The button is then offered on every finished, failed or terminated analysis: 1vsN, 1vs1, vs-group, cross compare, the three query methods and unique blocks.
  • A query is held to the upload cap. A rerun resends no upload, so QUERY_UPLOAD_LIMITS has nothing to measure, but the backend matches the stored query again, and disassembles a binary query again. A role with a cap (by default a visitor, at 1 MiB) may only rerun queries it submitted itself; other roles may rerun any. The job's recorded username decides, and a job that names nobody belongs to nobody. Every other analysis reruns for a visitor as before, since it repeats what the analyze pages already let a visitor run.

The review's two points (68bb176, a41ded6, on master merged in 530453f).

Tests. tests/testJobRerun.py:

  • with a client that has requestJobRerun: the backend is asked for every analysis kind; a refusal stays on the job and says so; a malformed id never reaches it; the button shows on every analysis and on a terminated job, but not on a running job or a collection change;
  • without the method, the rebuild runs as before;
  • query reruns: a visitor reruns their own query; a visitor can't rerun someone else's query, or one with no recorded owner (no button, flash, nothing asked); a visitor still reruns someone else's analysis of the corpus; a visitor whose role has no cap reruns any query; an unknown query job is reported before the backend is asked.

The crafted-id cases, the server-rerun tests and the query refusals fail with the route reverted. The guards that must keep working pass both ways. Full suite with Chromium on bfce32b: 2254 passed. ruff check . is clean.

Against a current backend (mcrit 1.13 on)

Until b642028 neither control was ever offered by a current backend. From mcrit 1.13 on, the backend fills in every matching knob before it queues a job (resolveMatchingParams in mcrit's index/MatchingParameters.py), so every 1-vs-N job records minhash_threshold, pichash_size, band_matches_required, band_df_cutoff and shortlist_size, and a 1-vs-1 or vs-group job the same without the shortlist. The rerun knew only the first three and withheld itself from all of them; the configuration link wanted band_matches_required alone. The captured fixtures come from mcrit 1.4.3 and record only that one, which is why the tests passed.

Change.

  • The rerun carries band_df_cutoff, and shortlist_size where the client method takes it (METHOD_MATCHING_PARAMS, mcritweb/views/data.py:2016): requestMatchesForSample does, requestMatchesForSampleVs and requestMatchesCross do not.
  • shortlist_unavailable is left out (DERIVED_MATCHING_PARAMS, mcritweb/views/data.py:2026). The backend derives it and no client call takes it; it marks the new job again if the shortlist still cannot be applied.
  • A cross compare's 1-vs-corpus children carry the shortlist_size=0 that getMatchesCross gives them, which requestMatchesCross would refuse. It is dropped there; a child with another size is not rerun (mcritweb/views/data.py:2146).
  • requirements.txt allows mcrit 1.12, whose client takes neither new knob, and would raise a TypeError on them. client_takes() (mcritweb/views/data.py:2155) offers the rerun only when the installed McritClient method takes every keyword it would pass.
  • The configuration link accepts the knobs the form has no control for when they are at mcrit's defaults (FORM_DEFAULT_MATCHING_PARAMS, mcritweb/views/data.py:2031, read from mcrit's MinHashConfig and StorageConfig, the same from 1.12 to 1.16). The analyze routes send only band_matches_required (mcritweb/views/analyze.py:322, :390, :447), so submitting the form leaves the others to the backend, which gives them those defaults unless its deployment configures other values. A job off the defaults still gets no link (mcritweb/views/data.py:2259). I took this over showing only what the form can show, since that would link a job run at threshold 40 to a form that resubmits it at 50.
  • The rerun route reads a cross compare's children in batches through dependency_jobs, as the job page does, instead of one getJobData each (mcritweb/views/data.py:2319).
  • Master merged in (c7a688d, then edef81f and 5ba68c1 for batch C). Its batched read goes through getQueueData(job_ids=...), so the two tests that delete a child now take it out of the fake's queue too (remove_job in tests/fixtureData.py). The job page passes the rerun and the link next to Poll a running job's status instead of reloading its overview #214's polling state (mcritweb/views/data.py:1873); a finished job's page does not poll, and the rerun lands on the new job's page with refresh=3, which Poll a running job's status instead of reloading its overview #214 polls (mcritweb/views/data.py:2332).

Verification. On the testbed (mcrit 1.13.0 backend), the old head (e4d66f2) and this head (b642028) as side instances, GET only, three finished jobs of the live queue:

job params recorded e4d66f2 b642028
1-vs-N, sample 8 all five knobs, shortlist_size 0 no rerun, no link rerun, link /analyze/compare?query=8&selected=8&minhashBandRange=2
1-vs-1, samples 8 and 12 four knobs no rerun, no link rerun, link /analyze/compare_versus?query_a=8&selected_a=8&query_b=12&selected_b=12&minhashBandRange=2
cross compare children {"0": 14}, queued before the backend recorded knobs rerun, no link rerun, no link

No rerun was posted to the live backend. check_corpus.py: as in the snapshot.

Tests. Jobs shaped like those of mcrit 1.16 (CURRENT_KNOBS in tests/fixtureData.py, the params from the review):

  • tests/testJobRerun.py: a 1-vs-N and a 1-vs-1 job are rerun with every knob; shortlist_unavailable is left out; a cross compare of either child method is rerun without a shortlist; a cross compare's rerun reads its children in one batch; a 1-vs-1 job carrying a shortlist, cross children run with a shortlist, and an mcrit 1.12 client are not rerun.
  • tests/testJobConfiguration.py: those jobs link to their form, and a job with any of the four knobs off the default does not.

Removing the shortlist drop, the client check, the derived-knob rule, the defaults check or the batched read each fails at least one of them. The 1-vs-1-with-a-shortlist test is held by both the per-method list and the client check; removing either one alone does not fail it. The three job test files also pass with mcrit 1.16.0's client on the path. ruff check . clean.

The rest of this description is from before the merge and the knob change.

Implements the first half of #55.

What's broken

A finished or failed analysis has no way back. To run it again you have to remember
which samples it compared and which MinHash setting it used, then rebuild the request by
hand on the compare page. grep -rn -i "rerun\|re-run\|resubmit" across *.py,
*.html and *.js returns exactly one hit, and it's a prose line in a test docstring.

How to see it yourself

Clean checkout, no backend needed:

git clone https://github.com/fkie-cad/mcritweb && cd mcritweb
python -m venv .venv && . .venv/bin/activate
make init

cat > tests/probe55.py <<'EOF'
import pytest
from fixtureData import job_id_of


class Recorder:
    def __init__(self, inner):
        self._inner, self.submitted = inner, []

    def __getattr__(self, name):
        if name.startswith("request"):
            def record(*a, **k):
                self.submitted.append((name, a, k))
                return "beefbeefbeefbeefbeefbeef"
            return record
        return getattr(self._inner, name)


@pytest.fixture
def fake_mcrit(corpus_mcrit):
    return Recorder(corpus_mcrit)


@pytest.mark.parametrize("report", ["matches_for_sample", "matches_for_sample_vs",
                                    "cross_compare", "matches_for_query", "unique_blocks"])
def test_probe(client, as_role, fake_mcrit, report):
    as_role("visitor")
    job_id = job_id_of(report)
    page = client.get(f"/data/jobs/{job_id}")
    has_button = f"/data/jobs/{job_id}/rerun" in page.get_data(as_text=True)
    r = client.post(f"/data/jobs/{job_id}/rerun")
    print(f"\n  {report:22s} page={page.status_code} button={has_button} post={r.status_code}")
    print(f"      submitted: {fake_mcrit.submitted}")
EOF

python -m pytest tests/probe55.py -q -s

Before

Every line reads button=False and submitted: [] — the route doesn't exist.

After

  matches_for_sample     page=200 button=True post=302
      submitted: [('requestMatchesForSample', (0,), {'force_recalculation': True, 'band_matches_required': 2})]
  matches_for_sample_vs  page=200 button=True post=302
      submitted: [('requestMatchesForSampleVs', (1, 3), {'force_recalculation': True, 'band_matches_required': 2})]
  cross_compare          page=200 button=True post=302
      submitted: [('requestMatchesCross', ([0, 2, 4, 6, 1],), {'force_recalculation': True, 'band_matches_required': 2, 'sample_group_only': True})]
  matches_for_query      page=200 button=False post=302
      submitted: []
  unique_blocks          page=200 button=False post=302
      submitted: []

Cross-check those against the captured job payloads and they match argument for argument
— e.g. matches_for_sample_vs stores {"band_matches_required": 2, "0": 1, "1": 3}.

Why this fix

The governing rule is: reconstructing the wrong request is worse than offering no
button.
A user who clicks "rerun" believes they re-ran what is on screen. If the
rebuilt request quietly differs, they'll compare the two results and draw a conclusion
from a difference that isn't real. So every step refuses rather than guesses — a
parameter that can't be read as an int, an unknown named parameter, or an unexpected
positional key disqualifies the whole rerun instead of being dropped from it. A future
mcrit that records one more matching parameter makes the button disappear rather than
submit a different comparison.

Only three methods are rerunnable, and the exclusions are the interesting part:

excluded why
getMatchesForSmdaReport, getMatchesForMappedBinary, getMatchesForUnmappedBinary the binary lives in the backend's GridFS; the job holds a reference, not bytes, and McritClient can't submit from a reference. mcritweb's own instance/temp/uploads/ copy is not a substitute — it's pruned, unshared across hosts, absent for jobs submitted by the IDA plugin, and for the SMDA branch it's named by smda_report.sha256 while the descriptor hashes the canonicalised JSON, so it can't even be located reliably.
getUniqueBlocks neither request method takes force_recalculation, so a repeat is answered from mcrit's descriptor cache with the very job being looked at. A "rerun" that hands back the job you're already on is worse than no button.
getMatchesForSampleVsGroup McritClient has no requestMatchesForSampleVsGroup.
addBinarySample, deleteSample, modifySample, deleteFamily, modifyFamily destructive, or no input to resend.
minhashing / maintenance not analyses; they already have their own POST buttons on the admin page.

force_recalculation=True, always. QueueRemoteCalls consumes that flag before the
payload is written, so no job records whether it was forced — and without it
get_cached_job_id(descriptor) returns the job being rerun. This is the deliberate
opposite of the analyze routes, which must stay cacheable because they write on GET
(#97).

Cross-compare is rebuilt from its children, and that inference is checked against
mcrit's source rather than assumed.
combineMatchesToCross stores only
{sample_id: child_job_id}; sample_group_only survives only as the choice between
the two child methods, and the matching parameters live on the children.
MinHashIndex.getMatchesCross is where that choice is made:

def getMatchesCross(self, sample_ids, sample_group_only=False, force_recalculation=False, **params):
    for id in sample_ids:
        if sample_group_only:
            job_id = self.getMatchesForSampleVsGroup(id, [...], ...)
        else:
            job_id = self.getMatchesForSample(id, ...)

So the child method is the record of that flag. The rerun is offered exactly when every
child named in the mapping resolves, has a known method, and they all agree — otherwise
no button, and the route refuses. It costs no extra round trips, because job_by_id
already resolves the dependencies.

POST-only, visitor_required. Rendered as a data-post button through the existing
post_action.js, following the precedent job deletion set in #84 — a rerun is a state
change and must not be reachable by GET. Visitor because the identical three requests,
rematch included, are already reachable at visitor level from analyze.compare_all,
compare_vs and start_cross_compare; the button grants no new capability. CSRF is the
standard mcritweb/csrf.py before_request, and a POST without the token was verified
to get a 400.

Offered only once a job has finished or failed. Forcing a recalculation of a running
job queues the same work twice; failed is the case where it's most wanted.

Parsing reads payload["params"] directly rather than going through Job.arguments,
which raises on a malformed payload — deciding whether to offer a rerun must not be
able to take the job page down.

Overlap with #18 — read before merging either

job_by_id had to stop 500ing on a deleted dependency before the refusal path was
reachable at all (a cross compare whose children are gone is exactly the case where the
button must be withheld). That is the same defect
#18 fixes.

I took #18's code verbatim, including its missing_children count and the warning
banner, rather than shipping a competing version that silently drops the missing entries.
So the two branches now contain textually identical hunks there — but they will still
conflict, and whichever merges second needs a trivial resolution (keep one copy).

What I didn't do

  • On job page add "Rerun job" button #55's second bullet — a "modify this job" link to a preselected configuration page —
    is not implemented. It's not a one-liner: analyze.cross_compare?samples=… genuinely
    preselects, but analyze.compare?selected=… only highlights a row if that sample
    happens to be on the current search page
    , so the link would usually appear to do
    nothing. Worth its own issue rather than a half-working link.
  • No rerun control on the jobs list rows — the issue asks for the job page.
  • Nothing was exercised against a real MCRIT backend (no MongoDB here). The parameter
    mapping was derived by reading mcrit/queue/QueueRemoteCalls.py, mcrit/Worker.py,
    mcrit/index/MinHashIndex.py and mcrit/server/utils.py::getMatchingParams, and
    checked against the five captured job payloads. The sample_group_only inference in
    particular is derived from the source quoted above, not observed on a live rerun.
  • is_rerunnable reads job_info.finished_at / is_failed unguarded. Safe because
    job_by_id dereferences both earlier in the same request, but it isn't defensive in
    isolation.
  • A malformed job payload still 500s job_by_id at job_info.parameters, before any
    of this code runs. Pre-existing — job_corrupted.html exists but is wired to nothing.
    I asserted the rerun route refuses such payloads instead of widening scope to fix the
    page; worth a separate issue.
  • A very fast double-click could queue two forced jobs. Same shape as the existing
    delete control; not mitigated.
  • A cross-compare rerun costs N+1 forced backend jobs. Intended, but the button gives
    no warning about the size of what it starts.
  • tests/fixtureData.py gained one thing: CorpusMcritClient.getJobData now also serves
    jobs from the captured queue.json, as a real backend does. Without it the cross
    compare's job page couldn't render under the corpus fixture at all.

Full suite: 269 passed (baseline on master is 239; +30 new tests, none changed or
removed). ruff check . clean.

Audit note: bullets above that no longer hold

Every What I didn't do / Risks section in this series was re-checked against the current code and against mcrit v1.8.1. The bullets below are the ones from this PR that a reviewer should not act on as written.

  • Two bullets were overtaken. The "modify this job" link is implemented, and each link carries the sample id so the row is actually on the page — which also answers the objection the bullet raised against it. And is_rerunnable no longer reads finished_at/is_failed unguarded.

  • The malformed-payload 500 this PR flagged as pre-existing is confirmed and is being fixed, on the job-search branch. Related: a rerun correctly refuses such a job and then redirects to the page that 500s, so the refusal did not survive the redirect.

A finished or failed analysis had no way back: to run it again you had to
remember which samples it compared and with which MinHash setting, and rebuild
the request by hand on the compare page.

A job records the call that produced it - payload["params"] is the arguments
the backend method was called with, positional ones keyed by index and the
rest by name - so the request can be rebuilt from the job itself. But only
where "the same request" is still a well-defined thing to ask for, so the set
is deliberately short: getMatchesForSample, getMatchesForSampleVs and
combineMatchesToCross.

Left out, with reasons: the three query methods, because the binary they
matched lives in the backend's GridFS and the job holds a reference rather
than the bytes, which McritClient cannot resubmit from; getUniqueBlocks,
because neither of its request methods takes force_recalculation, so a repeat
is answered from the cache with the very job being looked at; the collection
and minhashing methods, because they are not analyses to repeat.

Rebuilding the wrong request would be worse than offering no button - the user
would believe they had rerun what is on screen - so every step refuses rather
than guesses. A parameter that cannot be read as an int, an unknown named
parameter, or an unexpected positional key disqualifies the whole rerun rather
than being silently dropped from it; a backend that grows a parameter makes
the button disappear rather than run a different comparison.

force_recalculation is always set. QueueRemoteCalls consumes that flag before
the payload is written, so no job records whether it was forced - and without
it get_cached_job_id answers with the job being rerun.

A cross compare stores only {sample_id: child_job_id}. sample_group_only
survives only as the choice between the two child methods, which
MinHashIndex.getMatchesCross makes, and the matching parameters live on the
children - so the rerun is offered exactly when every child named in the
mapping resolves, has a known method, and they all agree.

POST-only through the existing postTo() helper, as job deletion is since familiary#84.
visitor_required because the identical requests are already reachable at
visitor level from analyze.compare_all, compare_vs and start_cross_compare.
Offered only once a job has finished or failed - forcing a recalculation of a
running job queues the same work twice.

job_by_id also had to stop 500ing on a deleted dependency to reach the
refusal path at all; that fix is taken verbatim from the branch of the
job-overview crash so the two do not disagree.

Refs #55.
`init` still ran `pip install -r requirements.txt` alone, so a fresh checkout
that follows the README lands without pytest and `make test` fails with
"No module named pytest" - the same failure the CI change on this branch fixes
for the runner, left unfixed for a contributor.

pytest and pytest-cov are not runtime dependencies and are not in
requirements.txt; mcrit declares them under its `dev` extra, which nothing
here requests, so they do not arrive transitively either. Install them from
`init`, pinned to the versions CI uses, and say so once above the target
rather than in a comment further down that describes a step that never ran.

Reported by Codex review.
… button

is_rerunnable() gated the button on a job having run its course, but
rerun_job_by_id did not repeat the check. Withholding a button only hides the
door: the POST is reachable by hand, or from a page rendered a moment before
the job started, and force_recalculation=True then queues the same work
alongside the run already in progress - which is precisely what the check
exists to prevent.

Lifted the condition into has_run_its_course() and applied it in both places.
It also stops raising for a job record that does not carry finished_at or
is_failed, which the route reaches for a job the page never rendered.

Reported by Codex review.
#55's second bullet: from a job, a way to reach the page it was submitted
from with its inputs already filled in, so the comparison can be changed and
run again rather than retyped. The rerun button covers repeating a job
unchanged; this covers everything else the issue asked for.

The arguments come from the rerun's own recovery (rerun_request), so it is
the same three methods and there is one place that reads a job's parameters,
not two. What is new is the requirement that the *form* be able to show them:

  * analyze.cross_compare preselects from `samples=` outright, and its two
    checkboxes cover sample_group_only.
  * analyze.compare and analyze.compare_versus highlight `selected` only when
    the sample is among the search results in front of them, and compare.html
    falls back to the first row when it is not - which is why PR familiary#133 left
    this bullet out. So each link also carries the search that puts the sample
    on the page: mcrit answers a sample id with that sample (id_match)
    wherever it would otherwise fall in the paging. That is what makes the
    preselection real rather than decorative, and it is what
    test_the_link_survives_the_sample_being_off_the_search_page pins down.
  * The Minhash Matching slider is the only matching control these forms have,
    so a job carrying minhash_threshold or pichash_size, or recording no band
    matching at all, or a band value with no slider position, gets no link
    rather than a form that quietly describes a different comparison.

`force_recalculation` is not preselected: QueueRemoteCalls consumes it before
a payload is written, so no job records whether it was forced.

The slider's position/value table moves to a module constant in params.py and
gains the reverse lookup, so the two directions cannot drift apart.

No new route - the link points at existing analyze pages - so routePolicy.py
is unchanged. Nothing was exercised against a real MCRIT backend; the
preselection is asserted by rendering the linked pages under the captured
corpus.

`altered_job` and `inject_job` move from testJobRerun.py to fixtureData.py so
both job-page test modules share them.
r0ny123 added a commit to r0ny123/mcritweb that referenced this pull request Aug 30, 2026
#55's second bullet: from a job, a way to reach the page it was submitted
from with its inputs already filled in, so the comparison can be changed and
run again rather than retyped. The rerun button covers repeating a job
unchanged; this covers everything else the issue asked for.

The arguments come from the rerun's own recovery (rerun_request), so it is
the same three methods and there is one place that reads a job's parameters,
not two. What is new is the requirement that the *form* be able to show them:

  * analyze.cross_compare preselects from `samples=` outright, and its two
    checkboxes cover sample_group_only.
  * analyze.compare and analyze.compare_versus highlight `selected` only when
    the sample is among the search results in front of them, and compare.html
    falls back to the first row when it is not - which is why PR familiary#133 left
    this bullet out. So each link also carries the search that puts the sample
    on the page: mcrit answers a sample id with that sample (id_match)
    wherever it would otherwise fall in the paging. That is what makes the
    preselection real rather than decorative, and it is what
    test_the_link_survives_the_sample_being_off_the_search_page pins down.
  * The Minhash Matching slider is the only matching control these forms have,
    so a job carrying minhash_threshold or pichash_size, or recording no band
    matching at all, or a band value with no slider position, gets no link
    rather than a form that quietly describes a different comparison.

`force_recalculation` is not preselected: QueueRemoteCalls consumes it before
a payload is written, so no job records whether it was forced.

The slider's position/value table moves to a module constant in params.py and
gains the reverse lookup, so the two directions cannot drift apart.

No new route - the link points at existing analyze pages - so routePolicy.py
is unchanged. Nothing was exercised against a real MCRIT backend; the
preselection is asserted by rendering the linked pages under the captured
corpus.

`altered_job` and `inject_job` move from testJobRerun.py to fixtureData.py so
both job-page test modules share them.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JNG3bNU5qpFwvtgeDEDc9K
r0ny123 added a commit to r0ny123/mcritweb that referenced this pull request Sep 6, 2026
#55's second bullet: from a job, a way to reach the page it was submitted
from with its inputs already filled in, so the comparison can be changed and
run again rather than retyped. The rerun button covers repeating a job
unchanged; this covers everything else the issue asked for.

The arguments come from the rerun's own recovery (rerun_request), so it is
the same three methods and there is one place that reads a job's parameters,
not two. What is new is the requirement that the *form* be able to show them:

  * analyze.cross_compare preselects from `samples=` outright, and its two
    checkboxes cover sample_group_only.
  * analyze.compare and analyze.compare_versus highlight `selected` only when
    the sample is among the search results in front of them, and compare.html
    falls back to the first row when it is not - which is why PR familiary#133 left
    this bullet out. So each link also carries the search that puts the sample
    on the page: mcrit answers a sample id with that sample (id_match)
    wherever it would otherwise fall in the paging. That is what makes the
    preselection real rather than decorative, and it is what
    test_the_link_survives_the_sample_being_off_the_search_page pins down.
  * The Minhash Matching slider is the only matching control these forms have,
    so a job carrying minhash_threshold or pichash_size, or recording no band
    matching at all, or a band value with no slider position, gets no link
    rather than a form that quietly describes a different comparison.

`force_recalculation` is not preselected: QueueRemoteCalls consumes it before
a payload is written, so no job records whether it was forced.

The slider's position/value table moves to a module constant in params.py and
gains the reverse lookup, so the two directions cannot drift apart.

No new route - the link points at existing analyze pages - so routePolicy.py
is unchanged. Nothing was exercised against a real MCRIT backend; the
preselection is asserted by rendering the linked pages under the captured
corpus.

`altered_job` and `inject_job` move from testJobRerun.py to fixtureData.py so
both job-page test modules share them.
This branch and master each lifted parse_band_range's slider table into a
module constant, under different names: BAND_RANGE_BY_SLIDER_POSITION
here, with slider_position_for_band_range() to read it backwards for the
rerun links; BAND_RANGE_ARG_TO_VALUE on master (#32), with the labels a
finished job's setting is shown by. One table now, master's name, read in
both directions - the inverse lookup stays and walks master's table.

The job overview passes can_rerun / configuration_url beside master's
missing_children, and the fake's altered_job / inject_job helpers sit
beside master's MATCHED_SAMPLE_ID.

One duplication git produced without a conflict: this branch carried its
own copy of the missing-sub-jobs alert, placed a few lines from master's,
so job_overview.html rendered it twice. master's
test_the_page_says_that_children_are_missing exists for exactly this and
failed on it; one copy is gone.

1040 passed, ruff clean.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011EAW1DRkBwmjZtGzQ5pgDA
r0ny123 added a commit to r0ny123/mcritweb that referenced this pull request Sep 23, 2026
Both poll the job's status instead of reloading its overview, and both cut
the backend calls alike: 850 against master's 1,230 while watching a
66-sample cross compare and a 1vN queued behind it. The familiary
session's version (fix/183-poll-job-status) also polls jobs without
sub-jobs and updates their progress in place, where this session's kept
reloading them; it reads the unfinished-dependency list defensively; and
its Chromium tests run the script itself and fail with the fix reverted.
An adversarial review recommended it over the other.

It conflicts with familiary#115, familiary#133 and familiary#149 where both sides extend the same
lines; its PR text gives each resolution, including the one that keeps
familiary#149's sub-job average in the progress cell. fix/183-job-page-poll stays
on the fork, unused.
@danielplohmann

Copy link
Copy Markdown
Collaborator

Reviewed for the batch C integration (#298, #299). The rerun is built safely: it's POST-only with CSRF and a data-post button, it resubmits only the stored job with a real force_recalculation=True, a running job is refused, and it gives a visitor nothing that analyze.compare* / start_cross_compare don't already allow. configuration_url is built from url_for with integers only. One thing keeps it out of the batch: against a current backend, the button never appears.

Against mcrit ≥ 1.13, no job is rerunnable and no configuration link is offered. mcrit resolves every matching knob before it queues a job (index/MatchingParameters.py resolveMatchingParams, called from MinHashIndex._submitMatchingJob and server/utils.py). So each job's params always carry minhash_threshold, pichash_size, band_matches_required and band_df_cutoff, and 1-vs-N jobs also carry shortlist_size (and shortlist_unavailable when the shortlist couldn't be applied). matching_kwargs rejects any key outside RERUN_MATCHING_PARAMS = ("minhash_threshold", "pichash_size", "band_matches_required"), so rerun_request returns None for every 1.16 job. configuration_url likewise requires the settings to be exactly {band_matches_required}.

The tests pass because the captured fixtures come from mcrit 1.4.3 (tests/fixtures/version.json) and record only band_matches_required. A getMatchesForSample job carrying {"minhash_threshold": 50, "pichash_size": 10, "band_matches_required": 2, "band_df_cutoff": 0, "shortlist_size": 0, "0": 0} gives rerun_request -> None; without the last two knobs, it gives a valid request.

What it would take:

  • carry band_df_cutoff and shortlist_size through the rerun, since the client accepts both;
  • drop shortlist_unavailable, because the backend derives it and the client can't send it;
  • for cross-compare children, drop the shortlist_size=0 that getMatchesCross injects, because requestMatchesCross refuses a shortlist;
  • let configuration_url accept knobs that equal the backend defaults, or only those the form can show;
  • add fixtures shaped like a 1.16 job, so the tests see what a real backend sends.

Smaller: rerun_job_by_id reads a cross compare's children with one getJobData each, up to 250. job_by_id now reads them in batches through dependency_jobs, which the rerun can reuse.

Also needs master merged in (#249). Master reads children through getQueueData(job_ids=...), and the two missing-child tests must remove the child from the fake's _queue as well as from _queued_by_id, or the child stays visible.

🤖 Generated with Claude Code

https://claude.ai/code/session_01UCjRSPdSunUm9rFRrQqGsG

Conflicts in mcritweb/views/data.py: master parses the base address and
the bitness in one call, and reads a job's dependencies in batches
(dependency_jobs) and their samples and families through
get_sample_entries / get_family_entries. The job page keeps both and
passes the rerun and configuration link on as before.

Master's batched read goes through getQueueData(job_ids=...), so the
two tests that delete a cross compare's child now take it out of the
fake's queue as well (remove_job), not only out of its getJobData map.
From mcrit 1.13 on, the backend fills in every matching knob before it
queues a job (resolveMatchingParams), so every job records
minhash_threshold, pichash_size, band_matches_required and
band_df_cutoff, a 1-vs-N job also shortlist_size, and
shortlist_unavailable where the shortlist could not be applied. The
rerun only knew the first three and withheld itself from all of them,
and the configuration link wanted band_matches_required alone. Against
a current backend neither was ever offered.

The rerun now carries band_df_cutoff, and shortlist_size where the
client method takes one (requestMatchesForSample). It leaves out
shortlist_unavailable, which the backend derives and no client call
takes. A cross compare's 1-vs-corpus children carry the shortlist_size=0
that getMatchesCross gives them, which requestMatchesCross refuses, so
it is dropped there, and a child with another size is not rerun.
requirements.txt still allows mcrit 1.12, whose client takes neither
new knob, so the rerun is offered only when the installed McritClient
takes every keyword it would pass.

The configuration link accepts the knobs the form has no control for
when they are at mcrit's defaults, which is what the form, leaving them
to the backend, gets again.

The rerun route reads a cross compare's children in batches through
dependency_jobs, as the job page does, instead of one getJobData each.

The tests get jobs shaped like those of mcrit 1.16, which the captured
fixtures (mcrit 1.4.3) are not.
@r0ny123

r0ny123 commented Oct 19, 2026 •

Copy link
Copy Markdown
Collaborator Author

Thanks. Master is merged in (c7a688d) and the knobs are handled in b642028; the description has a new section at the top with the details and a live check.

  • band_df_cutoff and shortlist_size are carried through the rerun. shortlist_size only where the client method takes it: requestMatchesForSample does, requestMatchesForSampleVs and requestMatchesCross don't, so a 1-vs-1 job that carries one is not rerun.
  • shortlist_unavailable is left out. The backend marks the new job again if the shortlist still can't be applied.
  • Cross-compare children: the shortlist_size=0 from getMatchesCross is dropped before requestMatchesCross. A child with any other size is not rerun, since a cross compare can't ask for it.
  • configuration_url accepts the knobs the form has no control for when they are at mcrit's defaults (read from MinHashConfig / StorageConfig, the same from 1.12 to 1.16), and still gives no link for a job run with another value. I picked that over "only what the form can show", because the latter would link a job run at threshold 40 to a form that resubmits it at 50. On a deployment that configures non-default knobs, its jobs get no link. That direction is the safe one.
  • Batched child reads: the rerun route now uses dependency_jobs, as the job page does.
  • Fixtures: CURRENT_KNOBS in tests/fixtureData.py has the params from your comment, and the new tests use them for 1-vs-N, 1-vs-1 and both kinds of cross compare. The 10 that expect a rerun, a link or a batched read fail on the old head. The refusals pass there too, because the old head refused every such job, so they are guards. The 1.12-client test is a guard as well; on the old head it errors only because that module imports no McritClient to patch.
  • Missing-child tests: they now remove the child from _queue as well (remove_job).

One more, found while checking the client signatures: requirements.txt still allows mcrit 1.12, whose request methods take neither new knob and would raise a TypeError. The rerun is now offered only when the installed McritClient method takes every keyword it would pass.

On the testbed (mcrit 1.13.0 backend, GET only), the live queue's 1-vs-N and 1-vs-1 jobs had no rerun and no link on the old head. Both now get them.

@r0ny123

r0ny123 commented Oct 19, 2026 •

Copy link
Copy Markdown
Collaborator Author

CI is red only in tests/testSubmitMetadata.py: 8 failed, 1427 passed on b642028, in a file this PR doesn't touch. CI now installs mcrit 1.16.0, whose McritClient uses its own requests.Session subclass, and the test stub doesn't provide it. #297 fixes that. I left the fix out of this PR so the two don't overlap. Once #297 is merged, a rerun here should be green.

@danielplohmann

Copy link
Copy Markdown
Collaborator

Two things since batch E (#318) landed on master:

  1. POST /data/jobs/<job_id>/rerun hands the id to the client without Check job ids from the URL before they reach the backend client #279's check. Please gate it with
    JOB_ID_PATTERN.fullmatch(job_id) like the other job routes. Otherwise a crafted id such as %2E%2E makes the
    client ask the backend for another path.
  2. Three tests in testJobConfiguration.py expect single-quoted URLs in the handlers. Since Keep query values inside the URL strings of the pagination handlers #275 they render with
    double quotes.

POST /data/jobs/<job_id>/rerun handed the id from the URL to getJobData
unchecked. McritClient puts it into its request path unquoted, so an id
such as %2E%2E made it ask the backend for another path. It is now
checked with JOB_ID_PATTERN, as familiary#279 checks the other job routes, and an
id that is not one goes back to the jobs list without a backend call.
Since familiary#275 the compare forms write `selected`, `selected_a` and
`selected_b` through tojson, so the three tests that looked for
single-quoted values failed.
familiary/mcrit#311 adds POST /jobs/<job_id>/rerun and
McritClient.requestJobRerun, which queue a finished, failed or terminated
analysis again from the payload it stored, input files included. Where the
installed McritClient has it, the rerun route asks for that, after the
JOB_ID_PATTERN check, and the button is offered on every such analysis:
the queries, unique blocks and 1-vs-group comparisons too, which cannot be
rebuilt from their parameters here. A rerun the backend refuses stays on
the job's page with a message. Without it, the request is rebuilt as
before. Both ask server_reruns(), so the button and the route agree.
The backend's rerun resends no upload, so QUERY_UPLOAD_LIMITS has
nothing to measure, but it matches the stored query again and
disassembles a binary query again. A visitor could so have any stored
query rerun, however much larger than the 1 MiB they may upload
themselves.

A query job is now rerun for a role without a cap, and for a capped
role only when the job is its own: the backend records who submitted
each job, and that upload passed the cap. The button follows the same
rule, and a job whose submitter is not recorded is nobody's. Every
other analysis reruns as before, since it repeats what the analyze
pages already let a visitor run.
@r0ny123

r0ny123 commented Oct 25, 2026 •

Copy link
Copy Markdown
Collaborator Author

Both fixed, on master merged in (530453f).

Two more commits, now that familiary/mcrit#311 adds requestJobRerun:

  • b63a060 has the backend rerun the job itself wherever the installed client has that method. The button then covers every finished, failed or terminated analysis, queries and unique blocks included.
  • bfce32b holds a query rerun to the upload cap. A role with a cap, by default a visitor, may only rerun its own queries, since a stored query may be larger than that role could have uploaded.

The description has the details. Full suite: 2254 passed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

On job page add "Rerun job" button

2 participants