Give each non-blocking write a distinct task id - #2861
Open
HarperZ9 wants to merge 1 commit into
Open
Conversation
…monw#2859 execute_write_fn(fn, block=False) is documented to return "a UUID representing the queued task". Two things stopped that being true. _send_to_write_thread() derived the id from uuid.uuid5(NAMESPACE_DNS, "datasette.io"), which is deterministic, so every non-blocking write in every database in every process returned 3f143baa-4e3d-5842-a36f-4fa2f683b72f. A constant cannot identify a particular task. Now uuid4(). Refs simonw#2860. With num_sql_threads=0 there is no write thread, so execute_write_fn took the synchronous branch and `result` was the write function's return value, normally None. The block=False path then unpacked it unconditionally and raised TypeError: cannot unpack non-iterable NoneType object. The non-threaded branch now returns the same (task_id, reply_future) shape, with the future already resolved because the write has finished, so both modes share one code path. Refs simonw#2859. test_execute_write_fn_block_false only asserted isinstance(task_id, uuid.UUID), which a constant satisfies. The new test is parametrized over threaded and non-threaded and asserts two calls return different ids, so either regression fails it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes two problems with
execute_write_fn(fn, block=False), both in the same method. Refs #2860 and #2859.The docs describe the return value as:
#2860: the id was a constant
_send_to_write_thread()derived it fromuuid.uuid5(uuid.NAMESPACE_DNS, "datasette.io").uuid5is deterministic, so every non-blocking write in every database in every process returned3f143baa-4e3d-5842-a36f-4fa2f683b72f. A constant cannot identify a particular task, so a caller could not tell two queued writes apart.Introduced in 4284c74 (#2220, 2023-12-19), shipped since
1.0a10.#2859: TypeError with
num_sql_threads=0With no write thread,
execute_write_fntakes the synchronous branch, soresultis the write function's return value, normallyNone. Theblock=Falsepath then unpacked it unconditionally:which raised
TypeError: cannot unpack non-iterable NoneType object. Only_send_to_write_threadreturns that pair.The non-threaded branch now hands back the same
(task_id, reply_future)shape with the future already resolved, since the write has already finished. Both modes share one code path from there, so the event-dispatch logic below is unchanged and does not need a mode check.Why the existing test did not catch either
test_execute_write_fn_block_falseasserted onlyisinstance(task_id, uuid.UUID). A constant satisfies that, and the test never ran in non-threaded mode.The new test is parametrized over threaded and non-threaded, and asserts two calls return different ids. It fails on
mainin both modes:AssertionErrorfor the constant,TypeErrorfor the unpack.Scope
WriteTask.task_idis not used for dispatch, so nothing was misrouted and no data was lost.fnstill propagates before the return path is reached, which differs from the threaded "silently swallowed" wording in the docs. That felt like a separate question, so I left it alone rather than widen this PR.Verification
pytest tests/test_internals_database.py tests/test_write_wrapper.py tests/test_api_write.py→ 244 passed, 2 xpassedblack --checkandruff checkclean on both changed filesPermissionError: [WinError 32]tempdir teardown error appears intest_api_write.pyon my Windows machine. It reproduces on an unmodified checkout, so it is unrelated to this change.Happy to split this into two PRs if you would rather review them separately, or to change the fix shape for #2859. I picked the option that keeps one code path rather than branching on
executor is Nonetwice.Disclosure: I used an AI coding assistant on this work. The reproductions, the test, and the verification runs above were executed and confirmed by me.
🤖 Generated with Claude Code