Skip to content

Show how long a job with dependencies really took (#46) - #149

Open
r0ny123 wants to merge 14 commits into
familiary:masterfrom
r0ny123:fix/46-cross-job-duration
Open

r0ny123 wants to merge 14 commits into
familiary:masterfrom
r0ny123:fix/46-cross-job-duration

Conversation

@r0ny123

@r0ny123 r0ny123 commented Aug 30, 2026 •

Copy link
Copy Markdown
Collaborator

Closes #46. A cross compare does not start until every child has finished, so its Duration (finished_at - started_at of the parent) times only the assembly of the matrix and reads about 0, and its Progress is the parent's own counter, which reads 0 for the whole time the children run. The job tables now show the total since the job was queued, and the overview shows the progress across the dependencies.

Change.

  • total_duration() (mcritweb/views/data.py:1673), a template filter, is finished_at - created_at for a finished job with dependencies. Both stamps come from the same clock. The job tables show it as "Total (since queued)" (mcritweb/templates/table/column_table.html:292), next to Duration, which keeps its meaning. A job without dependencies gets no row.
  • While the job runs, the far end of that span is the latest time the backend wrote on the job or on the dependencies the page fetched (latest_job_timestamp(), data.py:1604), not this server's clock. mcrit's queue stamps its jobs with datetime.now(), the backend's local time: in mcrit 1.13.0's libs/mongoqueue.py, created_at at line 365, started_at and locked_at at 418, locked_at again on every progress update at 1069, finished_at at 1029. The REST API sends those with a "Z" (json_util.dumps in server/utils.py). Measured against our UTC clock, a running job's total was off by the backend's UTC offset. East of UTC it came out negative and the row was hidden; west of UTC it showed hours too many. Now it is a lower bound, short by the time since the latest stamp, but never off by a time zone. The page counts on from there once a second (column_table.html:312), in the format the server renders a timedelta in. A failed or terminated job has no total while unfinished.
  • The in-progress page reads the dependencies for this through master's dependency_jobs() (render_job_in_progress(), data.py:2051): one queue read per 100 dependencies, only for an unfinished job that has dependencies. A page that has not read them shows no running total, since the parent's own stamps would only give the time between its children's creation and its own.
  • dependency_progress() (data.py:1629) averages the children's progress for the overview's Progress row (column_table.html:325). A finished child counts as 1.0, a child the backend no longer has is left out, and the count beside the average is the same set: "(across 4 dependencies)" when one of five is gone. The overview already says how many are gone, in master's warning.
  • That cell carries data-progress-of="dependencies" (column_table.html:329). Since Poll a running job's status instead of reloading its overview #214 the overview polls data.job_status and reloads only when the job's own state changes, which it does not while the sub-jobs run, so the average used to freeze until one of them settled, and once the parent started the next tick wrote the parent's own counter over it, from 100% down to about 0%.
    • job_status (data.py:2029) now also reads the sub-jobs of an unfinished job that has them, through dependency_jobs(), one queue read per 100 (data.py:2040), and job_overview_state() (data.py:1991) adds their average and how many it covers, as dependency_progress and dependencies. It still reads none of the samples or families they name, which are most of what a reload costs (Job pages re-fetch samples, families and dependencies from the backend on every poll #183). Neither value is a reload trigger.
    • The page's script (mcritweb/templates/job_overview.html:41) updates a cell with that flag from dependency_progress, keeps it as it is when no sub-job could be read (:44), and updates any other cell from the job's own progress, as Poll a running job's status instead of reloading its overview #214 does.
  • CorpusMcritClient.getJobData falls back to the captured queue for jobs without a captured result, as a real backend does.

Verification. In the test client and in Chromium on the captured corpus. The captured cross compare was queued at 10:46:10.490, its last child finished at 10:46:18.545 and the parent at 10:46:18.572:

state Duration Total (since queued) Progress
finished 0:00:00 0:00:08 Results available
rewound to before the parent started, server clock two hours behind the backend's This job hasn't finished yet 0:00:08 and counting 100.00% (across 5 dependencies)
the same, one child deleted 100.00% (across 4 dependencies), and "1 of this job's 5 sub-jobs is no longer in the system"

A job without dependencies renders as on master. On the testbed (mcrit 1.13.0), master on :5000 against a side instance of this branch, GET only: the finished 20-sample cross compare 6ab3e95a3b24190e88fedbc0 shows Duration 0:00:00 on both and "Total (since queued)" 0:00:08 on the branch; the repairMinHashes job 6abcb1f3394fadeaa0630ab1, which has no dependencies, shows the same rows on both. The testbed's backend runs on UTC and has no running job with dependencies, so the running total is checked on the captured corpus only.

Tests.

  • tests/testJobDuration.py: the finished and running totals on the overview, the result page and the in-progress page; the running total with this server's clock two hours behind the backend's stamps; the backend's latest stamp over every field and timestamp shape; no running total without the dependencies; the progress average, its count with a child gone, and the cell's flag; every timestamp shape a job document carries, and the shapes that give no number.
  • tests/testJobDurationBrowser.py: the running total counts on, from 0:00:08 to 0:00:10 after two seconds and to "1 day, 1:00:08" after a day and an hour, on Playwright's clock. It skips without Playwright, as in CI.
  • tests/testJobOverview.py, Poll a running job's status instead of reloading its overview #214's tests: the status of a running cross compare carries the sub-jobs' average and count, read in one queue request and with no sample or family lookups (this replaces Poll a running job's status instead of reloading its overview #214's test that the status reads the job alone); a job without sub-jobs is still read alone; a finished job's status reads no sub-jobs.
  • tests/testJobOverviewBrowser.py: a cross compare's cell follows dependency_progress in place, from 62.50% to 75.00% "(across 2 dependencies)", while the job's own counter says 0, without a reload; it keeps 62.50% when a tick read no sub-job; Poll a running job's status instead of reloading its overview #214's in-place test now uses a job without sub-jobs.

38 of the 41 new tests fail on master (6564602). Three are guards and pass there too: test_a_job_without_dependencies_reports_only_its_own_duration, test_the_status_of_a_job_without_sub_jobs_is_read_off_the_job_alone and test_the_status_of_a_finished_job_does_not_read_its_sub_jobs. Measuring against this server's clock again fails the four running-total unit tests and the two running-page tests; counting all of all_dependencies fails the count test; dropping the flag fails its test; dropping the script fails the browser test. For the polling: leaving out the flag check in the script fails both new browser tests, dropping the check for a number fails the one where no sub-job was read, a status that reads no sub-jobs fails the status test and the test that the page and the endpoint agree, and reading them for a finished job fails its test.

Relation to open PRs.

A cross compare does not start until every child it waits on has finished, and
Job.duration is finished_at - started_at of the parent alone, so it times the
assembly of the matrix and nothing of the matching that fills it. The captured
cross compare job was created at 10:46:10.490 and finished at 10:46:18.572 - eight
seconds, five children - and its page reported "Duration: 0:00:00". On a real
corpus the children dominate by orders of magnitude, so that number is always ~0.

created_at -> finished_at brackets the children as well, and both timestamps are
already on the parent's job document, so this needs no extra backend call. It is
shown as a second row beside the existing duration rather than replacing it: the
parent's own runtime is still the honest answer for a job that waited on nothing,
and every job without dependencies renders exactly as before.

The subtraction is done in Python because created_at arrives as a {"$date": ...}
dict on some paths and a string on others, and LocalQueue hands back a datetime on
a third; total_duration reads all three and returns None - no row - for anything it
cannot turn into a number.

Not done here: rolling the children's own progress up into the parent's. That needs
one getJobData per dependency and the job page auto-refreshes, so a 500-sample
cross compare would issue 500 calls per poll.

tests/fixtureData.py: getJobData now answers from the captured queue as well as
from the five reports with results, the way a real backend does - without it the
cross compare job page cannot be rendered offline at all, since its five children
live only in queue.json.

Refs #46
pytest and ruff used to arrive as transitive dependencies of mcrit and no
longer do, so a clean checkout could not run `make test` or `make lint`.
Install them explicitly in both the workflow and the target that claims to
set the environment up.
"Total (with dependencies)" reads as "including the time the dependencies
took", and it very nearly is - but a job that fans out is queued *after* the
children it waits on, so the number starts a little late. Measured in the
captured queue: the five children were created at .480, .482, .485, .487 and
.489, and the parent at .490 - so 10ms of a 8.06s job is outside the window,
and that interval grows with the number of children.

The exact lower bound is the earliest dependency's created_at. That is free on
the job overview page, which already fetches every child, and costs one
getJobData per child on the result pages, which fetch none - and paying that
on a page that auto-refreshes is the cost this issue's "wait" label is about.
Using it on one page and not the other would make two views of the same job
disagree, which is worse than a bound that is honest about being one.

So the number is unchanged and the label now says what it measures, with a
hint giving the reason and the size of the gap.

Found by Codex on the PR.
The "Total (since queued)" row only appeared once a job had finished, which
left the state issue #46 actually describes - children running, parent with no
duration of its own - with no number at all, on a page that auto-refreshes into
it and stays there for the whole of the work.

The determinism objection the row was held back for is a clock, so give it one:
`data.utc_now` is a function of its own precisely so a test can freeze it, and
nothing else about these rows varies. A job that ran out of attempts or was
terminated never gets a finished_at either, so it keeps showing nothing rather
than a total that grows for as long as the document lives.

The row also moves out of `show_duration`. The page that switches the duration
row off - job_in_progress.html, what /data/result renders for an unfinished job
- is exactly the one where this is the only elapsed time there is.

`job_timestamp_as_datetime` now truncates a datetime to whole seconds, as the
string path already did by construction: an in-process LocalQueue hands over
real datetimes and the clock carries microseconds, so without it a running
total rendered as 0:01:29.510000.

The "unfinished" case in testJobDuration is rewritten rather than removed - it
asserted that an unfinished job has no total, which is the behaviour this
commit changes. What it was really covering, "a shape that cannot produce a
number renders as no row", is now covered by the out-of-attempts, terminated
and missing-field cases.
The other half of #46. A job that waits on dependencies does not start until the
last child is done, so `Job.progress` reads 0 for the entire time the work is
running - the number the issue calls meaningless, on the page a reader watches
refresh.

The children each carry a real progress and `job_by_id` has already fetched them,
so averaging costs no additional backend call and needs no private field. mcrit's
Job exposes `all_dependencies` but no `unfinished_dependencies` accessor, and the
count only lives in `job_info._data`, which nothing in mcritweb touches - so this
takes the public route instead.

Three cases that would otherwise read wrong, each pinned by a test: a finished
child counts as done whatever its counter says, or a worker that never ticks to
1.0 holds the parent below 100% forever; a deleted dependency is skipped rather
than counted as zero, which would make the number go backwards when a dependency
is cleaned up; and a counter outside 0..1 is clamped, since a worker writes it.

Passed only from job_by_id, the one view that has the children. The in-progress
result page is left on the parent's counter deliberately: matching it there would
mean one getJobData per dependency on every poll of an auto-refreshing page.
@r0ny123
r0ny123 force-pushed the fix/46-cross-job-duration branch from e30f892 to 2d2a335 Compare September 6, 2026 07:51
r0ny123 added a commit to r0ny123/mcritweb that referenced this pull request Sep 23, 2026
Both poll the job's status instead of reloading its overview, and both cut
the backend calls alike: 850 against master's 1,230 while watching a
66-sample cross compare and a 1vN queued behind it. The familiary
session's version (fix/183-poll-job-status) also polls jobs without
sub-jobs and updates their progress in place, where this session's kept
reloading them; it reads the unfinished-dependency list defensively; and
its Chromium tests run the script itself and fail with the fix reverted.
An adversarial review recommended it over the other.

It conflicts with familiary#115, familiary#133 and familiary#149 where both sides extend the same
lines; its PR text gives each resolution, including the one that keeps
familiary#149's sub-job average in the progress cell. fix/183-job-page-poll stays
on the fork, unused.
@danielplohmann

Copy link
Copy Markdown
Collaborator

Reviewed for the batch C integration (#298, #299). The arithmetic holds up: dependency_progress can't divide by zero, it skips missing children and counts finished ones as 1.0, a failed or terminated job gets no "and counting", and mixed naive and aware times are caught. Two things keep it out of the batch for now.

1. "Total (since queued)" is off by the backend's UTC offset while a job runs. mcrit's queue stamps created_at / finished_at with datetime.now(), the server's local time (libs/mongoqueue.py:390, 1067); mcrit corrects for that itself at :311. job_clock_now() uses UTC, and its docstring says the job times are UTC, which they aren't. On a backend running UTC+2 (CEST), a running job's span is negative, so the row is hidden for its first two hours. West of UTC it shows hours too much. Finished jobs are fine, since both ends come from the same clock. Either measure a running job against the backend's clock rather than ours, or state plainly that the backend must run on UTC.

2. It no longer updates once #214 lands. #214 (in this batch) replaces the meta refresh with polling job_status, and reloads only when the parent job's own state changes. While the children run, the parent stays not-started, at 0% progress, with the same pending sub-job count. So the children's average progress and the running total this PR adds freeze until a child settles, which can be many minutes for one large sample. And after the reload on started, the next poll overwrites the cell (job_overview.html:41) with the parent's own progress: it drops from "100.00% (across N dependencies)" to about 0%. Options:

  • have job_status also return the children's average and update it in place;
  • or keep reloading on the timer while there are unfinished sub-jobs, and skip the in-place update when the cell shows the children's average (a data- flag on it would do);
  • either way, tick the elapsed time in the browser.

This will need master merged in as well (#249). Master reads the children in batches (dependency_jobs) and drops missing ones into missing_children. Note that "(across N dependencies)" counts the missing ones, while the average leaves them out.

🤖 Generated with Claude Code

https://claude.ai/code/session_01UCjRSPdSunUm9rFRrQqGsG

Conflicts in mcritweb/views/data.py: the job overview passes both
child_progress and master's missing_children, and reads the samples and
families of the dependencies through master's batch lookups.
mcrit's queue stamps created_at, started_at, locked_at and finished_at with
datetime.now(), the backend's local time, and the REST API sends them with
a "Z" all the same. Subtracting them from this server's UTC clock was off
by the backend's UTC offset while the job ran: east of UTC the span came out
negative and the row was hidden, west of UTC it showed hours too many.

A running job is now measured up to the latest stamp the backend wrote on
it or on the dependencies the page fetched, locked_at being refreshed on
every progress update. That is a lower bound, short by the time since that
stamp, but never off by a time zone. The in-progress page fetches the
dependencies for this too; a page without them shows no running total.

The progress row now counts the dependencies it averages, so a deleted
one is left out of both; the overview already says how many are gone.
Between refreshes the running total stood still. The page now counts on
from the number it was rendered with, once a second, in the format the
server renders a timedelta in. Starting from the backend's latest stamp, it
stays a lower bound.

The progress cell of a job waiting on dependencies carries
data-progress-of="dependencies", so a script that updates the cell from the
parent's own counter, as the polling in familiary#214 does, can leave the
dependencies' average alone instead of overwriting 100% with 0%.
Importing them by name made the module fail to import on master, so none
of its tests could show what master gets wrong.
@r0ny123
r0ny123 force-pushed the fix/46-cross-job-duration branch from 3c1371d to 034dabb Compare October 19, 2026 17:50
@r0ny123

r0ny123 commented Oct 19, 2026 •

Copy link
Copy Markdown
Collaborator Author

Thanks. Master is merged in (ffd783c), and the fixes are in c911879 and ab794d6. The merge is rebuilt on 2d2a335, so the branch was force-pushed and the old merge 3c1371d is gone from it.

1. Clock. c911879. You're right: mcrit stamps created_at, started_at, locked_at and finished_at with the backend's local datetime.now() (libs/mongoqueue.py:365, 418, 1029, 1069 in 1.13.0), and the REST API still sends them with a Z. Nothing the backend sends carries its current time, so a running job can't be measured against its clock directly. A running job is now measured up to the latest time the backend wrote on the job or its sub-jobs (latest_job_timestamp, data.py:1168). That is a lower bound, but it is never off by a time zone. job_clock_now and its docstring are gone. The in-progress page reads the sub-jobs for this, one queue read per 100; a page that hasn't read them shows no running total.

Count. "(across N dependencies)" now counts only the sub-jobs it averages. The page already says separately how many are missing.

2. #214. ab794d6. #214 isn't on master yet, so this keeps the meta refresh. The progress cell carries data-progress-of="dependencies", so #214's in-place update can leave a cell showing the children's average alone. The running total also counts on in the browser once a second, from the last value the page got. The description lists what the overview needs when the two are merged together: keep reloading on the timer while there are unfinished sub-jobs, since job_status reads only the parent, and skip the in-place update on a cell with that flag.

034dabb makes the tests reach the new helpers through the data module. Imported by name, the module failed to import on master, so none of its tests could fail there.

Gate on 034dabb: full suite, 1416 passed. 35 of the 36 new tests fail on master; the one that passes is a guard and is named as one. Undoing each fix fails its test. On the testbed (GET only), a finished 20-sample cross compare shows "Total (since queued)" 0:00:08, and a job without sub-jobs looks the same as on master.

@r0ny123

r0ny123 commented Oct 19, 2026 •

Copy link
Copy Markdown
Collaborator Author

CI is red only in tests/testSubmitMetadata.py: 8 failed, 1390 passed on 034dabb, in a file this PR doesn't touch. CI now installs mcrit 1.16.0, whose McritClient uses its own requests.Session subclass, and the test stub doesn't provide it. #297 fixes that. I left the fix out of this PR so the two don't overlap. Once #297 is merged, a rerun here should be green.

Since familiary#214 the overview polls data.job_status and reloads only when the
job's own state changes. A cross compare's parent does not start until its
last sub-job is done, so while they run that state stays the same, and the
progress cell, which shows the sub-jobs' average (#46), froze until one of
them settled. Once the parent started, the next tick wrote its own counter
over the cell, which dropped from 100% to about 0%.

job_status now also reads the sub-jobs of an unfinished job that has them,
one queue read per JOB_IDS_PER_REQUEST, and answers their average and how
many it covers. It still reads none of the samples or families they name,
which are most of what a reload costs. The page updates the cell marked
data-progress-of="dependencies" from that average and leaves it as it is
when no sub-job could be read; a job without sub-jobs is updated from its
own progress as before. Neither value triggers a reload.
@r0ny123

r0ny123 commented Oct 20, 2026 •

Copy link
Copy Markdown
Collaborator Author

Master is merged in again now that batch C has landed (e498077), and 8e45907 settles how this works with #214's polling. I took your first option.

  • job_status (data.py:2029) now also reads the sub-jobs of an unfinished job that has them, through dependency_jobs, one queue read per 100. It answers their average and how many it covers as dependency_progress and dependencies. It still reads none of the samples or families they name, which are most of what a reload costs, and neither value is a reload trigger.
  • The script (job_overview.html:41) updates the cell marked data-progress-of="dependencies" from that average. When a tick read no sub-job, it leaves the cell as it is rather than showing the parent's own 0. Any other cell is updated from the job's own progress, as before. So the average keeps moving between reloads, and the 100% no longer drops to about 0% once the parent starts.
  • The running total still counts on in the browser from the value the page was rendered with.

#214's test that a status tick reads the job alone now covers jobs without sub-jobs. For a cross compare it checks the one extra queue read and that no sample or family is looked up. Its in-place browser test uses a job without sub-jobs. New browser tests check that the cell follows the average while the parent's counter says 0, and that it keeps its value when no sub-job could be read. Taking out each guard fails at least one of them.

@danielplohmann

Copy link
Copy Markdown
Collaborator

Since #279 is on master, routes refuse job ids that aren't hex. The new tests use "solo", "done" and "parent", so
their pages answer "not found". Please give them 24-hex ids, as #279 did for testJobOverview.

The status endpoint keeps both: master's job id check before getJobData,
and this branch's sub-job read for an unfinished job waiting on them.

Since familiary#279 the job routes refuse ids that are not hex, so the new status
tests' "solo", "done" and "parent" jobs answered "not found". They get
24-hex ids like the tests familiary#279 changed, and the finished-job test checks
the state it gets back is the finished one.
@r0ny123

r0ny123 commented Oct 25, 2026 •

Copy link
Copy Markdown
Collaborator Author

Done in 9ef2af3, the master merge.

  • The new tests use 24-hex ids now: a new SOLO_JOB_ID, plus master's PARENT_JOB_ID and DONE_JOB_ID. That includes the browser test's route patterns.
  • The finished-job test also checks finished is True.
  • With the old ids, the two status tests fail.
  • job_status keeps master's hex check and this PR's sub-job read.

811 tests pass across the changed and related modules.

The hidden tooltip of the new "Total (since queued)" label is 962px of one
line (white-space: pre). hint.css hides a tooltip it does not show with
opacity and visibility alone, and a hidden box still takes its room, so the
cross compare result went from 916px to 1136px wide in a phone or a tablet.
A tooltip is now laid out only on hover or keyboard focus. A browser test
compares the page's width with and without the label's tooltip.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

cross job duration/progress is meaningless

2 participants