Skip to content

Give each non-blocking write a distinct task id - #2861

Open
HarperZ9 wants to merge 1 commit into
simonw:mainfrom
HarperZ9:fix/nonthreaded-nonblocking-write-2859
Open

Give each non-blocking write a distinct task id#2861
HarperZ9 wants to merge 1 commit into
simonw:mainfrom
HarperZ9:fix/nonthreaded-nonblocking-write-2859

Conversation

@HarperZ9

Copy link
Copy Markdown
Contributor

Fixes two problems with execute_write_fn(fn, block=False), both in the same method. Refs #2860 and #2859.

The docs describe the return value as:

A UUID representing the queued task will be returned.

#2860: the id was a constant

_send_to_write_thread() derived it from uuid.uuid5(uuid.NAMESPACE_DNS, "datasette.io"). uuid5 is deterministic, so every non-blocking write in every database in every process returned 3f143baa-4e3d-5842-a36f-4fa2f683b72f. A constant cannot identify a particular task, so a caller could not tell two queued writes apart.

Introduced in 4284c74 (#2220, 2023-12-19), shipped since 1.0a10.

#2859: TypeError with num_sql_threads=0

With no write thread, execute_write_fn takes the synchronous branch, so result is the write function's return value, normally None. The block=False path then unpacked it unconditionally:

task_id, reply_future = result

which raised TypeError: cannot unpack non-iterable NoneType object. Only _send_to_write_thread returns that pair.

The non-threaded branch now hands back the same (task_id, reply_future) shape with the future already resolved, since the write has already finished. Both modes share one code path from there, so the event-dispatch logic below is unchanged and does not need a mode check.

Why the existing test did not catch either

test_execute_write_fn_block_false asserted only isinstance(task_id, uuid.UUID). A constant satisfies that, and the test never ran in non-threaded mode.

The new test is parametrized over threaded and non-threaded, and asserts two calls return different ids. It fails on main in both modes: AssertionError for the constant, TypeError for the unpack.

Scope

  • Writes themselves were always correct. WriteTask.task_id is not used for dispatch, so nothing was misrouted and no data was lost.
  • I found no in-core consumer of the returned id, so I am not claiming a user-facing break in shipped Datasette. The impact is on the documented plugin API.
  • Exception behaviour is unchanged. In non-threaded mode a raising fn still propagates before the return path is reached, which differs from the threaded "silently swallowed" wording in the docs. That felt like a separate question, so I left it alone rather than widen this PR.

Verification

  • pytest tests/test_internals_database.py tests/test_write_wrapper.py tests/test_api_write.py244 passed, 2 xpassed
  • black --check and ruff check clean on both changed files
  • One PermissionError: [WinError 32] tempdir teardown error appears in test_api_write.py on my Windows machine. It reproduces on an unmodified checkout, so it is unrelated to this change.

Happy to split this into two PRs if you would rather review them separately, or to change the fix shape for #2859. I picked the option that keeps one code path rather than branching on executor is None twice.


Disclosure: I used an AI coding assistant on this work. The reproductions, the test, and the verification runs above were executed and confirmed by me.

🤖 Generated with Claude Code

…monw#2859

execute_write_fn(fn, block=False) is documented to return "a UUID representing
the queued task". Two things stopped that being true.

_send_to_write_thread() derived the id from uuid.uuid5(NAMESPACE_DNS,
"datasette.io"), which is deterministic, so every non-blocking write in every
database in every process returned 3f143baa-4e3d-5842-a36f-4fa2f683b72f. A
constant cannot identify a particular task. Now uuid4(). Refs simonw#2860.

With num_sql_threads=0 there is no write thread, so execute_write_fn took the
synchronous branch and `result` was the write function's return value, normally
None. The block=False path then unpacked it unconditionally and raised
TypeError: cannot unpack non-iterable NoneType object. The non-threaded branch
now returns the same (task_id, reply_future) shape, with the future already
resolved because the write has finished, so both modes share one code path.
Refs simonw#2859.

test_execute_write_fn_block_false only asserted isinstance(task_id, uuid.UUID),
which a constant satisfies. The new test is parametrized over threaded and
non-threaded and asserts two calls return different ids, so either regression
fails it.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant