Skip to content

Replace TrinoResult generator with class-based iterator for error recovery#612

Closed
srchilukoori wants to merge 1 commit into
trinodb:masterfrom
srchilukoori:fix/trino-result-iterator-recovery
Closed

Replace TrinoResult generator with class-based iterator for error recovery#612
srchilukoori wants to merge 1 commit into
trinodb:masterfrom
srchilukoori:fix/trino-result-iterator-recovery

Conversation

@srchilukoori

Copy link
Copy Markdown
Contributor

Problem

Fixes #598.

TrinoResult.__iter__ uses a generator (yield). Python generators are permanently finalized after an unhandled exception propagates through the yield point — all subsequent next() calls raise StopIteration. Since #597 introduced lazy segment downloads during iteration, a transient I/O error (e.g. S3 timeout) mid-iteration kills the generator. fetchone() then returns None as if the query completed, silently dropping remaining rows.

Root cause

Generator frame finalization is a Python language constraint, not a bug in the iteration logic itself. Once an exception propagates through yield, the generator's frame is deallocated and cannot be resumed.

Fix

Replace the generator with a class-based iterator (__iter__ returning self, __next__ holding state in instance fields). The iteration logic is unchanged — batch prefetch via fetch() before exposing rows, same _rownumber tracking. The difference is that instance fields (_row_iter, _next_rows) survive exceptions, so callers can resume iteration after catching a transient error.

Testing

Added test_trino_result_iterator_survives_transient_error — injects a FailOnceIterator that raises IOError on the 3rd element, verifies the iterator resumes and returns remaining rows after the caller catches the exception.

Full unit test suite passes (123 tests, 0 failures).

Python generators are permanently finalized after an unhandled exception
propagates through yield. With lazy spooled segment iteration (PR trinodb#597),
a transient I/O error (e.g. S3 timeout) during iteration kills the
generator. Subsequent fetchone() calls return None as if the query
completed normally, silently dropping remaining rows.

Replace the generator-based __iter__ with __iter__/__next__ on the class
itself. Instance fields (_row_iter, _next_rows) survive exceptions, so
the iterator can resume on the next next() call.

Fixes trinodb#598
@cla-bot cla-bot Bot added the cla-signed label Jun 27, 2026
@wendigo wendigo requested a review from azawlocki-sbdt June 29, 2026 17:37
Comment thread trino/client.py
@srchilukoori

Copy link
Copy Markdown
Contributor Author

@hashhar , @azawlocki-sbdt I'm closing this PR as #615 is addressing all issues raised in the reviews above.

@srchilukoori srchilukoori deleted the fix/trino-result-iterator-recovery branch July 8, 2026 16:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Development

Successfully merging this pull request may close these issues.

TrinoResult generator cannot recover from errors during lazy segment iteration

3 participants