Repository navigation
Conversation
A cross compare does not start until every child it waits on has finished, and
Job.duration is finished_at - started_at of the parent alone, so it times the
assembly of the matrix and nothing of the matching that fills it. The captured
cross compare job was created at 10:46:10.490 and finished at 10:46:18.572 - eight
seconds, five children - and its page reported "Duration: 0:00:00". On a real
corpus the children dominate by orders of magnitude, so that number is always ~0.
created_at -> finished_at brackets the children as well, and both timestamps are
already on the parent's job document, so this needs no extra backend call. It is
shown as a second row beside the existing duration rather than replacing it: the
parent's own runtime is still the honest answer for a job that waited on nothing,
and every job without dependencies renders exactly as before.
The subtraction is done in Python because created_at arrives as a {"$date": ...}
dict on some paths and a string on others, and LocalQueue hands back a datetime on
a third; total_duration reads all three and returns None - no row - for anything it
cannot turn into a number.
Not done here: rolling the children's own progress up into the parent's. That needs
one getJobData per dependency and the job page auto-refreshes, so a 500-sample
cross compare would issue 500 calls per poll.
tests/fixtureData.py: getJobData now answers from the captured queue as well as
from the five reports with results, the way a real backend does - without it the
cross compare job page cannot be rendered offline at all, since its five children
live only in queue.json.
Refs #46
pytest and ruff used to arrive as transitive dependencies of mcrit and no longer do, so a clean checkout could not run `make test` or `make lint`. Install them explicitly in both the workflow and the target that claims to set the environment up.
"Total (with dependencies)" reads as "including the time the dependencies took", and it very nearly is - but a job that fans out is queued *after* the children it waits on, so the number starts a little late. Measured in the captured queue: the five children were created at .480, .482, .485, .487 and .489, and the parent at .490 - so 10ms of a 8.06s job is outside the window, and that interval grows with the number of children. The exact lower bound is the earliest dependency's created_at. That is free on the job overview page, which already fetches every child, and costs one getJobData per child on the result pages, which fetch none - and paying that on a page that auto-refreshes is the cost this issue's "wait" label is about. Using it on one page and not the other would make two views of the same job disagree, which is worse than a bound that is honest about being one. So the number is unchanged and the label now says what it measures, with a hint giving the reason and the size of the gap. Found by Codex on the PR.
The "Total (since queued)" row only appeared once a job had finished, which left the state issue #46 actually describes - children running, parent with no duration of its own - with no number at all, on a page that auto-refreshes into it and stays there for the whole of the work. The determinism objection the row was held back for is a clock, so give it one: `data.utc_now` is a function of its own precisely so a test can freeze it, and nothing else about these rows varies. A job that ran out of attempts or was terminated never gets a finished_at either, so it keeps showing nothing rather than a total that grows for as long as the document lives. The row also moves out of `show_duration`. The page that switches the duration row off - job_in_progress.html, what /data/result renders for an unfinished job - is exactly the one where this is the only elapsed time there is. `job_timestamp_as_datetime` now truncates a datetime to whole seconds, as the string path already did by construction: an in-process LocalQueue hands over real datetimes and the clock carries microseconds, so without it a running total rendered as 0:01:29.510000. The "unfinished" case in testJobDuration is rewritten rather than removed - it asserted that an unfinished job has no total, which is the behaviour this commit changes. What it was really covering, "a shape that cannot produce a number renders as no row", is now covered by the out-of-attempts, terminated and missing-field cases.
The other half of #46. A job that waits on dependencies does not start until the last child is done, so `Job.progress` reads 0 for the entire time the work is running - the number the issue calls meaningless, on the page a reader watches refresh. The children each carry a real progress and `job_by_id` has already fetched them, so averaging costs no additional backend call and needs no private field. mcrit's Job exposes `all_dependencies` but no `unfinished_dependencies` accessor, and the count only lives in `job_info._data`, which nothing in mcritweb touches - so this takes the public route instead. Three cases that would otherwise read wrong, each pinned by a test: a finished child counts as done whatever its counter says, or a worker that never ticks to 1.0 holds the parent below 100% forever; a deleted dependency is skipped rather than counted as zero, which would make the number go backwards when a dependency is cleaned up; and a counter outside 0..1 is clamped, since a worker writes it. Passed only from job_by_id, the one view that has the children. The in-progress result page is left on the parent's counter deliberately: matching it there would mean one getJobData per dependency on every poll of an auto-refreshing page.
e30f892 to
2d2a335
Compare
Both poll the job's status instead of reloading its overview, and both cut the backend calls alike: 850 against master's 1,230 while watching a 66-sample cross compare and a 1vN queued behind it. The familiary session's version (fix/183-poll-job-status) also polls jobs without sub-jobs and updates their progress in place, where this session's kept reloading them; it reads the unfinished-dependency list defensively; and its Chromium tests run the script itself and fail with the fix reverted. An adversarial review recommended it over the other. It conflicts with familiary#115, familiary#133 and familiary#149 where both sides extend the same lines; its PR text gives each resolution, including the one that keeps familiary#149's sub-job average in the progress cell. fix/183-job-page-poll stays on the fork, unused.
|
Reviewed for the batch C integration (#298, #299). The arithmetic holds up: 1. "Total (since queued)" is off by the backend's UTC offset while a job runs. mcrit's queue stamps 2. It no longer updates once #214 lands. #214 (in this batch) replaces the meta refresh with polling
This will need master merged in as well (#249). Master reads the children in batches ( 🤖 Generated with Claude Code |
Conflicts in mcritweb/views/data.py: the job overview passes both child_progress and master's missing_children, and reads the samples and families of the dependencies through master's batch lookups.
mcrit's queue stamps created_at, started_at, locked_at and finished_at with datetime.now(), the backend's local time, and the REST API sends them with a "Z" all the same. Subtracting them from this server's UTC clock was off by the backend's UTC offset while the job ran: east of UTC the span came out negative and the row was hidden, west of UTC it showed hours too many. A running job is now measured up to the latest stamp the backend wrote on it or on the dependencies the page fetched, locked_at being refreshed on every progress update. That is a lower bound, short by the time since that stamp, but never off by a time zone. The in-progress page fetches the dependencies for this too; a page without them shows no running total. The progress row now counts the dependencies it averages, so a deleted one is left out of both; the overview already says how many are gone.
Between refreshes the running total stood still. The page now counts on from the number it was rendered with, once a second, in the format the server renders a timedelta in. Starting from the backend's latest stamp, it stays a lower bound. The progress cell of a job waiting on dependencies carries data-progress-of="dependencies", so a script that updates the cell from the parent's own counter, as the polling in familiary#214 does, can leave the dependencies' average alone instead of overwriting 100% with 0%.
Importing them by name made the module fail to import on master, so none of its tests could show what master gets wrong.
3c1371d to
034dabb
Compare
|
Thanks. Master is merged in (ffd783c), and the fixes are in c911879 and ab794d6. The merge is rebuilt on 2d2a335, so the branch was force-pushed and the old merge 3c1371d is gone from it. 1. Clock. c911879. You're right: mcrit stamps Count. "(across N dependencies)" now counts only the sub-jobs it averages. The page already says separately how many are missing. 2. #214. ab794d6. #214 isn't on master yet, so this keeps the meta refresh. The progress cell carries 034dabb makes the tests reach the new helpers through the Gate on 034dabb: full suite, 1416 passed. 35 of the 36 new tests fail on master; the one that passes is a guard and is named as one. Undoing each fix fails its test. On the testbed (GET only), a finished 20-sample cross compare shows "Total (since queued)" 0:00:08, and a job without sub-jobs looks the same as on master. |
|
CI is red only in |
Since familiary#214 the overview polls data.job_status and reloads only when the job's own state changes. A cross compare's parent does not start until its last sub-job is done, so while they run that state stays the same, and the progress cell, which shows the sub-jobs' average (#46), froze until one of them settled. Once the parent started, the next tick wrote its own counter over the cell, which dropped from 100% to about 0%. job_status now also reads the sub-jobs of an unfinished job that has them, one queue read per JOB_IDS_PER_REQUEST, and answers their average and how many it covers. It still reads none of the samples or families they name, which are most of what a reload costs. The page updates the cell marked data-progress-of="dependencies" from that average and leaves it as it is when no sub-job could be read; a job without sub-jobs is updated from its own progress as before. Neither value triggers a reload.
|
Master is merged in again now that batch C has landed (e498077), and 8e45907 settles how this works with #214's polling. I took your first option.
#214's test that a status tick reads the job alone now covers jobs without sub-jobs. For a cross compare it checks the one extra queue read and that no sample or family is looked up. Its in-place browser test uses a job without sub-jobs. New browser tests check that the cell follows the average while the parent's counter says 0, and that it keeps its value when no sub-job could be read. Taking out each guard fails at least one of them. |
The status endpoint keeps both: master's job id check before getJobData, and this branch's sub-job read for an unfinished job waiting on them. Since familiary#279 the job routes refuse ids that are not hex, so the new status tests' "solo", "done" and "parent" jobs answered "not found". They get 24-hex ids like the tests familiary#279 changed, and the finished-job test checks the state it gets back is the finished one.
|
Done in 9ef2af3, the master merge.
811 tests pass across the changed and related modules. |
The hidden tooltip of the new "Total (since queued)" label is 962px of one line (white-space: pre). hint.css hides a tooltip it does not show with opacity and visibility alone, and a hidden box still takes its room, so the cross compare result went from 916px to 1136px wide in a phone or a tablet. A tooltip is now laid out only on hover or keyboard focus. A browser test compares the page's width with and without the label's tooltip.
Closes #46. A cross compare does not start until every child has finished, so its
Duration(finished_at - started_atof the parent) times only the assembly of the matrix and reads about 0, and itsProgressis the parent's own counter, which reads 0 for the whole time the children run. The job tables now show the total since the job was queued, and the overview shows the progress across the dependencies.Change.
total_duration()(mcritweb/views/data.py:1673), a template filter, isfinished_at - created_atfor a finished job with dependencies. Both stamps come from the same clock. The job tables show it as "Total (since queued)" (mcritweb/templates/table/column_table.html:292), next toDuration, which keeps its meaning. A job without dependencies gets no row.latest_job_timestamp(),data.py:1604), not this server's clock. mcrit's queue stamps its jobs withdatetime.now(), the backend's local time: in mcrit 1.13.0'slibs/mongoqueue.py,created_atat line 365,started_atandlocked_atat 418,locked_atagain on every progress update at 1069,finished_atat 1029. The REST API sends those with a "Z" (json_util.dumpsinserver/utils.py). Measured against our UTC clock, a running job's total was off by the backend's UTC offset. East of UTC it came out negative and the row was hidden; west of UTC it showed hours too many. Now it is a lower bound, short by the time since the latest stamp, but never off by a time zone. The page counts on from there once a second (column_table.html:312), in the format the server renders a timedelta in. A failed or terminated job has no total while unfinished.dependency_jobs()(render_job_in_progress(),data.py:2051): one queue read per 100 dependencies, only for an unfinished job that has dependencies. A page that has not read them shows no running total, since the parent's own stamps would only give the time between its children's creation and its own.dependency_progress()(data.py:1629) averages the children's progress for the overview'sProgressrow (column_table.html:325). A finished child counts as 1.0, a child the backend no longer has is left out, and the count beside the average is the same set: "(across 4 dependencies)" when one of five is gone. The overview already says how many are gone, in master's warning.data-progress-of="dependencies"(column_table.html:329). Since Poll a running job's status instead of reloading its overview #214 the overview pollsdata.job_statusand reloads only when the job's own state changes, which it does not while the sub-jobs run, so the average used to freeze until one of them settled, and once the parent started the next tick wrote the parent's own counter over it, from 100% down to about 0%.job_status(data.py:2029) now also reads the sub-jobs of an unfinished job that has them, throughdependency_jobs(), one queue read per 100 (data.py:2040), andjob_overview_state()(data.py:1991) adds their average and how many it covers, asdependency_progressanddependencies. It still reads none of the samples or families they name, which are most of what a reload costs (Job pages re-fetch samples, families and dependencies from the backend on every poll #183). Neither value is a reload trigger.mcritweb/templates/job_overview.html:41) updates a cell with that flag fromdependency_progress, keeps it as it is when no sub-job could be read (:44), and updates any other cell from the job's own progress, as Poll a running job's status instead of reloading its overview #214 does.CorpusMcritClient.getJobDatafalls back to the captured queue for jobs without a captured result, as a real backend does.Verification. In the test client and in Chromium on the captured corpus. The captured cross compare was queued at 10:46:10.490, its last child finished at 10:46:18.545 and the parent at 10:46:18.572:
A job without dependencies renders as on master. On the testbed (mcrit 1.13.0), master on :5000 against a side instance of this branch, GET only: the finished 20-sample cross compare
6ab3e95a3b24190e88fedbc0showsDuration0:00:00 on both and "Total (since queued)" 0:00:08 on the branch; therepairMinHashesjob6abcb1f3394fadeaa0630ab1, which has no dependencies, shows the same rows on both. The testbed's backend runs on UTC and has no running job with dependencies, so the running total is checked on the captured corpus only.Tests.
tests/testJobDuration.py: the finished and running totals on the overview, the result page and the in-progress page; the running total with this server's clock two hours behind the backend's stamps; the backend's latest stamp over every field and timestamp shape; no running total without the dependencies; the progress average, its count with a child gone, and the cell's flag; every timestamp shape a job document carries, and the shapes that give no number.tests/testJobDurationBrowser.py: the running total counts on, from 0:00:08 to 0:00:10 after two seconds and to "1 day, 1:00:08" after a day and an hour, on Playwright's clock. It skips without Playwright, as in CI.tests/testJobOverview.py, Poll a running job's status instead of reloading its overview #214's tests: the status of a running cross compare carries the sub-jobs' average and count, read in one queue request and with no sample or family lookups (this replaces Poll a running job's status instead of reloading its overview #214's test that the status reads the job alone); a job without sub-jobs is still read alone; a finished job's status reads no sub-jobs.tests/testJobOverviewBrowser.py: a cross compare's cell followsdependency_progressin place, from 62.50% to 75.00% "(across 2 dependencies)", while the job's own counter says 0, without a reload; it keeps 62.50% when a tick read no sub-job; Poll a running job's status instead of reloading its overview #214's in-place test now uses a job without sub-jobs.38 of the 41 new tests fail on master (6564602). Three are guards and pass there too:
test_a_job_without_dependencies_reports_only_its_own_duration,test_the_status_of_a_job_without_sub_jobs_is_read_off_the_job_aloneandtest_the_status_of_a_finished_job_does_not_read_its_sub_jobs. Measuring against this server's clock again fails the four running-total unit tests and the two running-page tests; counting all ofall_dependenciesfails the count test; dropping the flag fails its test; dropping the script fails the browser test. For the polling: leaving out the flag check in the script fails both new browser tests, dropping the check for a number fails the one where no sub-job was read, a status that reads no sub-jobs fails the status test and the test that the page and the endpoint agree, and reading them for a finished job fails its test.Relation to open PRs.