[anth-volk]

Fixes #26

Its stack (#25: #21 → #22 → #23 → #24) has merged, so the base is main. It implements the emitter side of the collector's artifact intake: PolicyEngine/microcosm-collector#6 (the contract) and PolicyEngine/microcosm-collector#12 (the endpoints and worker).

Update 2026-10-27: status checks and receipts

Why. The submission contract, merged as microcosm-collector's docs/artifact-submissions.md, says that once completion is accepted the client polls GET /v1/submissions/{submission_id} until the status is final, then reads GET /v1/submissions/{submission_id}/receipt. This pull request treated a 200 or 202 answer to completion as the end of its job. Decision (Anthony, 2026-10-27, tracker 5.2 · 1): the emitter checks each submission's status until it is final and fetches its receipt, in its background queue, so builds never wait for it. collector#12, which serves both endpoints, merged on 2026-10-27 and deploys as 0.5.0 with intake switched off.

What changed.

• Accepted submissions stay queued. Completion answering 202 (queued or processing) or 200 (a final status) no longer settles the hand-over's collector row in the hand-over SQLite database. Each submission becomes awaiting_final_status in the row's receipt, and the row stays pending, so the existing leases (two minutes, renewed every 30 seconds), attempts and retries drive the checks.
• Backoff and time limit. The first check is 10 seconds after acceptance, each later wait twice the last, up to ten minutes; at once when completion already answered a final status. A submission still not final a day after acceptance becomes timed_out (FINAL_STATUS_TIMED_OUT, or COLLECTOR_REFUSED_403 when the last check was refused because the token belongs to another Hugging Face user than the one who submitted it). The last check falls on the limit itself, and an emitter that starts days later checks once more before applying it. The collector's recovery brings every job to a final status within three attempts or dispatches of about an hour each.
• Final statuses, with the contract's names: succeeded, rejected and failed. On any of them the client reads the receipt, keeps it only when it names the submission and its receipt_sha256 matches (SHA-256 of the contract's canonical JSON), and the row stores it with the collector's rejection (code, message).
• Other answers follow the contract's table. A 404 submission_not_found from the status or the receipt creates the submission again, from the copies kept at hand-over (they stay until the row settles). A 409 submission_not_final from the receipt (an operator returned it to queued) keeps checking. Outages, 401, a 403 without a code and a collector without intake are handled as for submitting. A refused Hugging Face token, or a 403 submission_owned_by_another_user (the next emitter on the machine runs with another Hugging Face user's token), keeps the checks going until the time limit, since the submission is already in. Status checks read no file.
• When the emitter exits first, for example at the end of a build, the row stays pending in the hand-over database, and the next emitter started with that database resumes the checks (tested by closing one module and starting another on the same database). An emitter killed during a check frees the row within one lease, as before.
• Builds never wait. The checks run on the collector's own worker thread, added earlier in this pull request. stage_dataset_bundle and every receipt query answer from the database at once, and at shutdown a check in progress gets at most the existing 2 seconds.
• Interface: EmitterClient.collector_submission_receipt(handover_id), with the new fixture huggingface-collector-submission-receipt.json (digest recorded; released fixtures unchanged), and the TypedDicts CollectorSubmissionReceipt, CollectorSubmission and CollectorRejection in microcosm_emitter_interface.submissions. The call reports failures under its own kind, collector_submission, so an older emitter that refuses the query does not silence warnings for staging calls.
• HandoverQueue.finish_job now returns whether it recorded the attempt, so only the holder of a row's lease reports its outcome.
• Still off by default. No collector address handling was added (see "Not in this PR").

How a rejected submission shows up.

• To the build: client.collector_submission_receipt(handover_id) answers at once. A rejected bundle has status: "rejected" and error_code set to the collector's code, such as content_mismatch, stored_object_mismatch or object_missing; the matching entry in submissions has status: "rejected", rejection: {"code": ..., "message": ...} and the collector's receipt. The hand-over's status is rejected if any submission was rejected, else failed (for example worker_error), else timed_out, else succeeded. An entry that is failed without a receipt is one the emitter could not submit, with the emitter's own code.
• In the logs: when the hand-over settles, the emitter writes one message to standard error for each submission that ended rejected, failed or timed_out, for example Microcosm collector submission rejected: calibration-diagnostics for hand-over <id> (submission sub-…), content_mismatch. The hand-over's collector_submission_receipt has the details. The build starts the emitter host as a child process that inherits its standard error, so the text appears in the build's log when the build is still running; otherwise it appears in the log of the next build whose emitter finishes the checks.

Commits (red, then green). 40a419c first merges main (the 0.7.0 release commit) with a merge commit.

Commits (red → green)	Change	Red (tests first)	Green
96109fd → 20e6dc7	Check accepted submissions until final, and store their receipts	tests/submissions: 44 failed, 63 passed (no check_status; receipts still said submitted)	107 passed, three runs; tests/huggingface, tests/interface, tests/test_boundaries.py: 374 passed
5add52b → e940cf7	Report each submission that ends rejected or failed on standard error	tests/submissions: 3 failed (nothing written), 108 passed; the guard that a success writes nothing passes before and after	111 passed, three runs; tests/huggingface, tests/test_queue_compatibility.py: 168 passed
81b4938 → 53c1769	Stop checking a submission's status a day after acceptance	tests/submissions: 2 failed (checks continued, the last at 87,240 s instead of 86,400 s), 112 passed; the guard that a final status found days later is still stored passes before and after	114 passed, three runs
37d4f19 → 33ea2ac	Let a build read its collector outcome through the interface	tests/interface, tests/submissions: 4 failed (no such call), 310 passed	314 passed, two runs; tests/test_boundaries.py, tests/test_package_names.py, tests/huggingface: 181 passed
351eaaf	Describe status checks, receipts and how a rejection shows (README, interface README, changelog fragment)	Documentation only	—
beb09c3 → 7671af0 (review)	Keep checking a submission whose status another user's token cannot read: a status or receipt request answered 403 submission_owned_by_another_user had recorded the submission as failed (COLLECTOR_REFUSED_403) and stopped checking, although the collector holds it and may have finished it	tests/submissions/test_client.py: 2 failed, 66 passed	68 passed; tests/submissions: 118 passed
5bf5351 → b09dfa6 (re-review)	Keep that refusal as the code of a submission that times out: when every check until the one-day limit was answered 403 submission_owned_by_another_user, the submission became timed_out with FINAL_STATUS_TIMED_OUT, so the stored record and the standard-error message no longer showed that another user's token was used. It now keeps COLLECTOR_REFUSED_403	tests/submissions/test_delivery.py: 2 failed (FINAL_STATUS_TIMED_OUT instead of COLLECTOR_REFUSED_403), 44 passed	46 passed; tests/submissions: 120 passed

Testing. At the head: tests/submissions, tests/interface, tests/huggingface, tests/test_boundaries.py, tests/test_queue_compatibility.py and tests/test_package_names.py, 501 passed at b09dfa6. tools/check-quality.sh (ruff check, ruff format --check), tools/check-changelog.sh and tools/check-artifacts.sh pass; the repository defines no type checker. The fake collector (tests/support/fake_collector.py) gained both GET endpoints, answering as collector#12's intake_api.py does, and finish(), which plays the worker. No test reaches a real collector or the network beyond loopback.

Cross-check against collector#12's code (not committed). A scratch script served microcosm-collector main's create_app over loopback, with the collector's own test stand-ins for its database, bucket and worker queue, and ran its real worker.run_job. Through this client: a submission was accepted as queued, then succeeded after the worker ran, with a receipt whose receipt_sha256 matched and whose result was {"processed_by": "intake-only", "verified_files": 2}; bytes changed after completion gave rejected with content_mismatch; a stored object with another recorded SHA-256 gave completion 200 rejected, then a receipt with stored_object_mismatch and job: null; an unknown ID gave SUBMISSION_NOT_FOUND; and a collector with intake switched off gave COLLECTOR_INTAKE_NOT_ENABLED.

Deployment verification. No workflow, Dockerfile, Terraform, packaging or release script changed, and no new code reads an environment variable. Unverified, and a gap on Modal: the hand-over database defaults to ~/.cache/microcosm/telemetry/events.handovers.sqlite3 (or under XDG_CACHE_HOME). On Modal that is inside the container unless it points at a Volume, and I found no Microcosm code that does so. A status check still pending when a Modal container stops is then lost with the container's database; the collector still holds the submission and its receipt.

Summary

The huggingface module now also sends what a staged dataset bundle names to the collector, after the bundle's Hugging Face commit. Microcosm needs no separate call for it. Off by default.

• The interface describes the build (CollectorProvenance). A bundle hand-over doesn't say which country or build it is, or whether its data is licensed, and the collector needs all three. StageDatasetBundle therefore takes an optional collector=CollectorProvenance(country_code, build_id, licensed_data, release=None), checked with the collector contract's rules.
  • The message carries it only when given, so existing bundles keep their message and hand-over ID.
  • A bundle without it goes only to Hugging Face, as today.
• What is submitted (submissions/planning.py, by role only; files are never read or interpreted):
  • A calibration_diagnostics file gives a calibration-diagnostics submission, with any build_manifest, release_manifest and demographics file.
  • orrery_schema plus execution_overlay give an orrery-inputs submission, with any orrery_upstream and orrery_evidence files, mapped to the collector's role names. The interface exports these bundle role names.
  • Dataset files and sidecars are never submitted.
• After the Hugging Face commit. The provenance names the repo, the commit that staged the bundle and its folder. So the hand-over's collector row in handover_jobs (one row per hand-over and destination, so no new table) waits for its huggingface row (WAITING_FOR_HUGGING_FACE). A bundle Hugging Face refuses is skipped.
• The steps (submissions/client.py):
  • Exchange the build's Hugging Face token for an artifacts:submit credential. CollectorSession gains an optional scope; telemetry sessions still send {}. A scoped session uses a token only when the answer says "scope": "artifacts:submit".
  • Create the submission, PUT each file the collector still lacks to its signed URL with exactly its headers, including Host (412 means already stored), then complete it. An instruction whose Host differs from its URL's host is refused.
  • Files copied when the bundle was handed over (the Hugging Face module now copies a bundle's small files, because Microcosm rewrites its build manifest right after staging) are submitted as they were then; the other files are re-hashed first.
  • Uploads go only to https://storage.googleapis.com, or to loopback when a development collector is configured.
• Outcomes (the contract, now merged as microcosm-collector's docs/artifact-submissions.md; updated on 2026-10-27, see the next section):
  • pending, retried with the queue's backoff: outages, 408/429/5xx, a 409 at completion, and a 404 submission_not_found at completion (the next attempt creates the submission again).
  • pending for an hour, with no request at all to that collector meanwhile (COLLECTOR_INTAKE_NOT_ENABLED): an exchange answered 200 without the artifacts:submit scope (a collector released before intake) or 404, and any other 404 from a submission path, including one to create. The token is discarded.
  • A 403 without a code (the token's scope does not allow the request) is followed by one fresh exchange and retry; only a second such refusal is final.
  • failed without a receipt: the emitter's own refusals (including 403 submission_owned_by_another_user) and changed bytes, with the emitter's code.
  • skipped: a bundle Hugging Face refused, or whose folder is longer than the collector's 512 characters (COLLECTOR_FOLDER_TOO_LONG).
  • After completion accepts a submission: awaiting_final_status while the emitter checks its status, then the collector's final succeeded, rejected or failed with its receipt, or timed_out after a day. This replaces the earlier submitted. Read it through EmitterClient.collector_submission_receipt.
• Off by default. Nothing is submitted unless the module configuration sets collector_submissions: true, which no interface release sends. development_collector_url points at a loopback collector. Switching production builds on is a separate emitter change that flips this default, so this release changes nothing for running builds.
• A new protocol fixture (huggingface-stage-dataset-bundle-with-collector.json) carries a described bundle. Released fixtures are unchanged.

Not in this PR

• Releases (publish_release). The same mechanism would submit a release's calibration diagnostics with release provenance, which the calibration-tree worker needs to record each release publication. It is a follow-up, kept out to keep this bounded.
• The existing Orrery hand-over (hand_over_orrery_inputs, the graph publication queue to microcosm-runs) is unchanged. Its role set has no execution overlay, so the collector gets the Orrery inputs from the bundle instead. O1 (microcosm#1295) already registers graph.execution.json as a build output.
• Submission through emitter#29's trusted collector addresses. Whichever of Submit staged bundles' calibration diagnostics and Orrery inputs to the collector #27 and Choose the collector origin in one place, from a fixed set of trusted origins #29 merges second makes collector submissions use it.
• Retry-After and the 1 MiB response cap, as before.

Commits (red, then green)

1. Let a bundle hand-over describe its build for the collector: red 11 failed against a stub. The 12th test, that a bundle without provenance keeps its message, already passed. Green 12 passed.
2. Plan the collector submissions a staged bundle names: red 6 failed; green 6 passed.
3. Submit a planned submission through the collector's intake: red 23 failed; green 23 passed against a loopback fake collector (tests/support/fake_collector.py).
4. Submit described bundles to the collector after their Hugging Face commit: red 10 errored (the module took no collector settings). Adding the new protocol fixture then failed the released-digest test (1 failed, 14 passed) until its digest was recorded. Green 10 and 15 passed.

Each commit's own test files were run again after committing.

Testing

• Every test file this PR adds or changes, together: tests/interface/test_collector_provenance.py, tests/submissions/ and tests/interface/test_protocol_fixtures.py, 66 passed. They use temporary migrated databases, the in-memory Hub, synthetic credentials and loopback servers only.
• tools/check-quality.sh and tools/check-changelog.sh pass.
• I didn't run the rest of the suite locally, including the existing tests for the files I changed (core/auth.py, huggingface/service/module.py, requests.py, bundles.py). CI covers them.
• Cross-check against the real collector code, not committed. A scratch script ran the collector's app from microcosm-collector#12 over loopback, with its bucket and queue replaced by local stand-ins, and drove this client with a bundle naming both kinds. The real endpoints accepted both submissions, and the uploads carried the signed headers. Repeating each submission returned the same ID, with the job enqueued once.

Deployment verification

No workflow, Dockerfile or release script changes. New and reused settings:

Setting	Provider	Wiring
collector_submissions (module configuration)	none; absent means off	The interface's HuggingFaceSettings.configuration sends only handover_database_path, so it is off in every build.
development_collector_url (module configuration)	none; development and tests only	Must be a loopback origin (the existing _development_collector_url check).
Collector origin	the existing PRODUCTION_COLLECTOR_URL constant	Used only when switched on without a development URL.
HF_TOKEN, HUGGINGFACE_TOKEN or the saved hf auth login token	the build machine's environment, as for the existing modules	Read by the existing lookup, now public as huggingface_token (#24), and sent only to the collector's token exchange.

Coordination

• This PR edits files Add the typed microcosm-emitter-interface package and release it with the emitter #22–Publish release builds side by side with releases.json as the only record of what is current #24 added (bundles.py, __init__.py, huggingface/service/module.py, requests.py, tests/interface/test_protocol_fixtures.py). It was restacked on their review fixes on 2026-10-20 (below).
• Per AGENTS.md, I won't merge this or monitor queued checks.

Restack and collector contract fixes (2026-10-20)

Restacked on the review fixes to #21–#24, then changed to match collector#6 (contract, 673df3d) and collector#12 (intake, 30dbd1b). Each commit red, then green; only the test files this pull request adds or changes were run, with loopback services only.

Commit	Change	Red (tests first)	Green
4598ed7 Use the shared hand-over codes and public credential lookup after the restack	#24 now defines the hand-over error codes and file hashing once and renamed the credential lookup; the module did not import at all on the restacked branch.	tests/test_boundaries.py, tests/submissions: 2 failed, 10 errors, 38 passed	49 passed (the one remaining failure is the next commit's)
9c5447f Submit the bundle files copied at hand-over to the collector	With #23's copying, the collector row still re-hashed every file, so every Microcosm calibration-diagnostics submission (which includes the rewritten build manifest) would fail. Copies are kept until both rows settle.	tests/submissions: 1 failed, 39 passed	68 passed with test_bundles.py
b7c7422 Send the described-bundle fixture through a typed call in the tests	The CI failure in test_every_fixture_has_a_typed_call_that_produces_it.	test_client_messages.py: 1 failed, 28 passed (as in CI)	30 passed
906cd4e Use a submit token only when the exchange grants that scope	Item 1: token scope, intake not available, hourly retry.	tests/submissions: 21 failed, 23 passed (most from the client's new clock argument)	73 passed with tests/telemetry/test_delivery.py
42b65d2 Create a submission again only for a 404 submission_not_found	Item 2.	test_client.py: 5 failed, 26 passed	tests/submissions: 49 passed
517d1b6 Fail a submission whose completion answers rejected or failed	Item 3; the fake answers stored_object_mismatch with 200 rejected, and its status documents carry files_sha256 and job.	3 failed, 50 passed	53 passed
0130888 Exchange again for the submit scope after a 403 without a code	Item 4; the fake refuses a token of another scope with 403 and no code.	3 failed, 53 passed	56 passed
62badd7 Report an already-staged bundle only with the commit that added it	Item 5: no fallback to the head when last_commit fails or reports nothing.	test_bundles.py: 2 failed, 28 passed	86 passed with tests/submissions
af1e35d Skip a collector submission whose folder is longer than 512 characters	Item 6.	test_delivery.py: 1 failed, 12 passed	tests/submissions: 57 passed
c3b6082 Refuse an upload instruction whose Host header names another host	Item 7; the fake lists Host. Sending Host as given already worked (that test passed before).	test_client.py: 1 failed, 39 passed	59 passed
28dcec2 Describe the collector answers and decision D10's disclosure rules	Item 8: only the Orrery outputs of licensed builds stay private until cleared; calibration data may be public; native evidence is never published.	Documentation only: no red step	—

Item 9 (the fake collector: scope, error codes, files_sha256 and job, Host, stored_object_mismatch as 200 rejected) came with items 1–4 and 7. Not done, as asked: polling the submission's status and fetching its receipt.

Re-run after the last commit: every test file this pull request adds or changes, 148 passed; with tests/test_boundaries.py, tests/huggingface and tests/telemetry/test_delivery.py, 167 passed. tools/check-artifacts.sh exit 0; ruff check, ruff format --check and tools/check-changelog.sh pass. The earlier four commits' hashes changed in the restack.

Review fixes (2026-10-20, first review of this pull request)

Each fix is a commit that adds a test failing because of the bug, then a commit with the code. Only the test files this pull request adds or changes were run, with loopback services only.

Commits (red → green)	Finding	Red (tests first)	Green
2ea1dd6 → 46012a2 Submit to the collector on a worker of its own, with time limits	tick ran the collector cycle after the Hugging Face cycle on the module's one worker thread. Uploads had no write timeout (httpx.Timeout(60.0, write=None)), and uncopied files were re-hashed on that thread before every attempt. So a slow or stalled submission delayed the Hugging Face commits a build waits for and stopped the commit at shutdown.	tests/submissions/test_delivery.py: 5 failed, 13 passed. With a collector upload held: run files never reached the staging repository; in a host runtime with a 2-second shutdown period, LocalServiceError: Service worker failed (TimeoutError); UPLOAD_TIMEOUT.write was None; the upload limit and the cycle budget did not exist.	18 passed; 153 across this pull request's changed test files
1701b3e → 19cac1b Keep collector errors out of the Hugging Face answers and startup	Review code-quality items 4, 8 and part of 1: an error queueing the collector row turned the bundle's answer into an error; an unusable collector setting (a non-loopback development URL) stopped the Hugging Face module from starting; an unexpected exception during a submission left the row leased for an hour.	test_delivery.py: 3 failed, 18 passed	21 passed

Design (finding 7). This uses the mechanism from #23's fix:

• Own worker. Collector submissions run on their own DeliveryWorker, so a slow or stalled submission never delays a Hugging Face commit, the commit at shutdown or a read-back. tick runs the collector cycle only when the module runs without workers (tests driven by hand). At shutdown a submission still running at the deadline is abandoned with the process, and the collector's HTTP client and the queue are released only once its worker has stopped.
• Write timeout. Uploads get httpx.Timeout(60.0): a stalled upload fails within a minute.
• Upload limit. Each upload also stops after upload_time_limit(size): five minutes, or longer for a file that needs it at 2 Mbit/s. The bytes are streamed in 1 MiB chunks, and the limit is checked before each one.
• Cycle budget. A cycle starts no new submission after MAX_CYCLE_SECONDS (ten minutes). With the limits on every request and upload, one cycle's length is bounded.
• Clock in the runtime tests. The two runtime tests use the real clock, as the host does: a submission waiting for its Hugging Face commit is retried seconds later.

Not done now: #29's collector_origin setting (review finding 2). #27 still chooses the collector origin itself in _collector_delivery (the development URL, else the production constant), so once #29 and this pull request are both merged, a configured collector_origin would be ignored for collector submissions. The fix needs #29 merged to main and carried up the stack first, since the resolver doesn't exist in this branch's history. It will be:

• call configured_collector_origin(self.configuration), as the telemetry and Orrery modules do, and allow loopback uploads only when the resolved origin is the development URL;
• an origin the resolver refuses switches collector submissions off with a warning, as an unusable setting already does after 19cac1b, instead of stopping the module.

Restack onto #23's and #24's fixes. 9a0dda7 (submit described bundles) changed while it was rebased: the module keeps its collector wiring next to the new delivery workers, and the test fixtures use handover_database_path and background_workers=False. As before, two of this pull request's intermediate commits don't pass on their own (9a0dda7 imports the old private credential lookup until bdf9410; a submission test is fixed by dc66627); the branch head passes.

Re-run after the last commit, every test file this pull request adds or changes: 156 passed; every test file the four stacked pull requests change, at this head: 480 passed. tools/check-artifacts.sh exit 0; ruff check, ruff format --check and tools/check-changelog.sh pass.

Not changed here: the collector row's hour-long lease is not yet renewed during an attempt (an emitter killed mid-upload still blocks that row for up to an hour; unexpected errors no longer do); Retry-After, the collector's error codes in receipts and the 1 MiB response cap (review items 3, 6, 7). Open decisions, unchanged: polling submission status and receipts, the interface role constants for H2c, and AGENTS.md's publishing rule.

Fix round 2 (2026-10-20)

Restack onto #23's and #24's round-2 fixes. Every commit has a new SHA; the titles are unchanged. The sections above use the earlier SHAs: 9a0dda7 is now c4c46c5, bdf9410 is 4c8c360, dc66627 is 51aec26, 2ea1dd6 is e081c07, 46012a2 is f2c94a7, 1701b3e is f901abd and 19cac1b is cdffe80. One change goes beyond replaying the commits: #23's DeliveryWorker now passes its stop check to each cycle, so f2c94a7 builds the collector worker's cycle as lambda _stop_requested: self.collector_delivery.run_cycle(). Without that, the worker would fail on every cycle. The collector cycle uses the stop check from f735a52 on. As before, two intermediate commits don't pass on their own (c4c46c5 imports the old private credential lookup until 4c8c360; a submission test is fixed by 51aec26).

Commits (red → green)	Finding	Red (tests first)	Green
b30ec7e → f735a52 Finish both shutdown commits before waiting briefly for the collector	close waited for every worker, the collector's included, before the second shutdown commit. A collector upload still running at shutdown used up the whole period, and the second commit, for the repository a bundle upload had held, was skipped even when that upload finished in time. On Modal those run files are lost.	tests/submissions/test_delivery.py, the review's case: 1 failed, 21 passed (AssertionError: the staging run files missed the second shutdown commit), in two runs	24 passed (two tests added with the fix: the bounded collector wait, and a cycle that claims nothing once stop is requested); 159 across this pull request's changed test files
d838a03 → e463794 Stop only stalled uploads, never ones that keep making progress	Every upload stopped after upload_time_limit(size), which assumes at least 2.1 Mbit/s. A large file on a slower link (399 MiB at 1.5 Mbit/s) was stopped on every attempt, started again from zero, and was never submitted.	tests/submissions/test_client.py: 1 failed, 41 passed (AssertionError: an upload that kept making progress was stopped; the outcome was ('pending', 'UPLOAD_FAILED'), at the second 1 MiB piece), in three runs. The stalled-upload test passes on the old code too: its write timeout already detected a stall.	42 passed; 160 across this pull request's changed test files

Design: shutdown (finding 6). close stops every worker, and the collector cycle now claims no further hand-over once stop is requested. close then:

1. makes the first shutdown commit;
2. waits until the deadline for the Hugging Face workers only (bundles and release changes);
3. makes the second shutdown commit when they stopped in time;
4. only then waits for a collector submission in progress, for at most COLLECTOR_SHUTDOWN_WAIT_SECONDS (2 s) and never past the deadline.

Collector work therefore never uses up the time the Hugging Face shutdown commits need, and a busy collector worker no longer holds the process for the whole shutdown period. A submission still running is abandoned with the process, as before, and its row is retried once its lease expires. The HTTP client and the queue stay open while any worker still runs.

Design: stall detection instead of a size-based limit (finding 7). A signed upload is one PUT that stores nothing unless it completes, so a limit on its total time only makes a slow upload start again from zero.

• upload_time_limit, chunks_until, UploadTimeLimitError, MIN_UPLOAD_SECONDS and MIN_UPLOAD_BYTES_PER_SECOND are gone.
• A stall is detected by UPLOAD_TIMEOUT's write timeout, which httpcore applies to each send to the connection: a send that has had no byte accepted for 60 seconds fails the attempt (pending, UPLOAD_FAILED). Connecting and waiting for the answer keep their 60-second limits. Files are still sent in 1 MiB pieces with an explicit Content-Length.
• The cycle's ten-minute budget only stops new submissions from starting, as before; its comment now says it never stops one in progress.
• Tests. Both ends of a loopback connection get small socket buffers (FakeCollector(receive_buffer_bytes=…) and httpx.HTTPTransport(socket_options=…)), so the client's sends wait for the fake to read. The tests patch UPLOAD_TIMEOUT to keep the stall threshold under a second. The slow-upload test also shortens any size-based limit with monkeypatch.setattr(..., raising=False), which has nothing to shorten after the fix.
• Known gap, unchanged. The collector row's one-hour lease is still not renewed during an attempt, so an attempt that outlasts it (a very large file on a slow link) can be claimed again by another emitter on the same machine. The lease's comment now says so.

Process note (finding 8; no history rewrite). In the first round of fixes, the red commit 2ea1dd6 (now e081c07) fails at a setup step rather than for its stated reason: its two runtime tests used the manual clock, and the green commit 46012a2 (now f2c94a7) also switched them to the real clock. With the manual clock, a collector row claimed before its bundle's Hugging Face commit is rescheduled 2 seconds ahead on a clock that never moves, so one of those tests fails with AssertionError: no upload started instead. The tests as they stand at 46012a2 do fail for the stated reasons against the unfixed code. Those commits are not rewritten. This round's red commits, b30ec7e and d838a03, were each run before their fix and fail for their stated reasons.

Re-run after the last commit, every test file this pull request adds or changes: 160 passed; every test file #23, #24 and this pull request change, at this head: 430 passed. tools/check-artifacts.sh exit 0; ruff check, ruff format --check and tools/check-changelog.sh pass.

Fix round 3 (2026-10-20)

Restack onto #24's round-3 fixes. Every commit has a new SHA, the same title and the same patch (git range-diff shows = for all 23). The sections above use the round-2 SHAs: c4c46c5 is now 8941474, 4c8c360 is 57c0136, 51aec26 is 4451d7f, e081c07 is c64d661, f2c94a7 is 394c9b5, f901abd is 3380529, cdffe80 is 4296daa, b30ec7e is 24062f1, f735a52 is 3f63781, d838a03 is f1e40f6 and e463794 is dc59b69. As before, 8941474 does not pass on its own until 57c0136.

Commits (red → green)	Finding	Red (tests first)	Green
c489ad7 → 8ecd153 Send uploads in 64 KiB pieces so HTTPS stalls are measured per piece	Over HTTPS the 60-second write timeout covered each whole 1 MiB piece, because CPython's ssl does no partial writes. A link slower than about 140 kbit/s failed partway and started again from zero.	tests/submissions/test_client.py: 1 failed, 42 passed (an HTTPS upload that kept making progress was stopped, outcome ('pending', 'UPLOAD_FAILED')), in three runs. With the threshold raised to 5 s the same test passes on that code, so the failure is the write timeout on the first piece.	43 passed, three runs
99fa0e3 → 6e8d912 Renew the collector row's lease while an attempt runs	The row's one-hour lease was never renewed, so a second emitter on the same machine could claim the row and upload the same file in parallel, and the first attempt's result was dropped.	tests/submissions/test_delivery.py: 1 failed, 24 passed (a second emitter claimed the collector row while its upload was running), in two runs. A guard (the row is free again within one lease after shutdown) passes before and after.	25 passed, three runs
a634cfb → 423139b Ask for the collector token again just before completing	Uploads longer than the token's life completed with an expired token and got 401, so the receipt said COLLECTOR_UNAVAILABLE until the next attempt.	test_client.py: 1 failed, 43 passed (the completion after a long upload used an expired token, outcome ('pending', 'COLLECTOR_UNAVAILABLE'))	44 passed, three runs

Before pushing, I amended the unpushed red commit 99fa0e3 to drop a final attempts == 1 check that raced: the row can first be attempted, and recorded as WAITING_FOR_HUGGING_FACE, before the bundle's Hugging Face commit. The amended commit still fails for the stated reason.

Design: 64 KiB pieces (replaces the round-2 wording "a send that has had no byte accepted for 60 seconds" and "files are still sent in 1 MiB pieces"). UPLOAD_CHUNK_BYTES is now 64 KiB, the size httpx itself reads files in. Each piece is one send under the 60-second write timeout. Over HTTPS, where a piece is written whole under one time limit, the slowest link that never fails averages 65,536 bytes per 60 seconds over each piece: 1,093 bytes a second, about 8.7 kbit/s. Over plain HTTP (a loopback development collector) a send still fails only after 60 seconds in which no byte was accepted. Nothing else changes on the wire. The code comment, the README and the changelog fragment state this rule and rate instead of "never stopped, however slow the link".

• Test. The fake collector gains tls_uploads: a second server for uploads that speaks TLS, as Cloud Storage does, while the collector stays plain HTTP. Both sides use only anonymous TLS 1.2 cipher suites, so no certificate or key is generated or committed. The fake reads uploads at about 1 MiB/s, the threshold is 0.5 s and the file is 1.5 MiB, so a 64 KiB piece takes about 75 ms and a 1 MiB piece more than a second.
• Follow-up commit 383a27f (Run the HTTPS slow-upload test over a Unix socket). The first CI run at 423139b failed this test on both macOS jobs (Ubuntu passed). Measured locally: a loopback TCP connection on macOS grows its receive buffer to about 280 KiB whatever SO_RCVBUF it is given, so it absorbs several pieces at once and the next piece waits up to about 0.3 s, close to the 0.5 s threshold. The upload server now listens on a Unix socket, which keeps the buffer it is given; the test client sends https:// requests over it (an httpx mount with uds=), and the upload URLs still name https://127.0.0.1. Each piece then takes a steady 70 to 80 ms. With the test as it stands after this commit, setting UPLOAD_CHUNK_BYTES back to 1 MiB makes it fail three times out of three for the stated reason, and 64 KiB passes three times out of three, also under Homebrew's Python 3.14, which the macOS CI jobs use.

Design: a renewed two-minute lease (replaces the round-2 "Known gap, unchanged" bullet). The collector row's lease is now 120 s (SUBMISSION_LEASE_SECONDS), renewed every 30 s (SUBMISSION_LEASE_RENEWAL_SECONDS) while an attempt runs, through the same HeldLeases the Hugging Face jobs use.

• HeldLeases gains for_job, which holds a job without a repository lease, and an optional stop check: renewal ends at stop, or once shutdown has begun. The Hugging Face leases are built without the stop check, so they behave as before.
• _attempt now returns an AttemptResult; run_cycle stops the renewal, then records the result with the renewed lease value, so the attempt's result is recorded.
• Another emitter on the machine never claims the row while an upload runs, however long it takes. An attempt abandoned at shutdown, or an emitter that was killed, frees the row within two minutes instead of an hour.
• Tests. The lease is shortened to 1 s and the renewal to 0.2 s; a second collector delivery on the same hand-over database, with a client that records each submission and sends nothing, runs its cycle every 0.1 s for two and a half leases while the first emitter's upload is held.

Design: a fresh token before completion. submit asks the session for the token once more after the uploads. The session returns the same token while it is valid and exchanges for a new one only when it is about to expire, so uploads that outlast the token (an hour) complete in the same attempt. The fake collector now issues tokens with a lifetime the test chooses and answers an expired one with 401.

Re-run after the last commit, every test file this pull request adds or changes: 164 passed; every test file #23, #24 and this pull request change, at this head: 446 passed (five runs at 423139b, once more at 383a27f). tools/check-artifacts.sh exit 0; ruff check, ruff format --check and tools/check-changelog.sh pass. No workflow, Dockerfile, Terraform, packaging or release script changed, and no new code reads an environment variable.

🤖 Generated with Claude Code