Replace TrinoResult generator with class-based iterator for error recovery#612
Closed
srchilukoori wants to merge 1 commit into
Closed
Replace TrinoResult generator with class-based iterator for error recovery#612srchilukoori wants to merge 1 commit into
srchilukoori wants to merge 1 commit into
Conversation
Python generators are permanently finalized after an unhandled exception propagates through yield. With lazy spooled segment iteration (PR trinodb#597), a transient I/O error (e.g. S3 timeout) during iteration kills the generator. Subsequent fetchone() calls return None as if the query completed normally, silently dropping remaining rows. Replace the generator-based __iter__ with __iter__/__next__ on the class itself. Instance fields (_row_iter, _next_rows) survive exceptions, so the iterator can resume on the next next() call. Fixes trinodb#598
Contributor
Author
|
@hashhar , @azawlocki-sbdt I'm closing this PR as #615 is addressing all issues raised in the reviews above. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Fixes #598.
TrinoResult.__iter__uses a generator (yield). Python generators are permanently finalized after an unhandled exception propagates through the yield point — all subsequentnext()calls raiseStopIteration. Since #597 introduced lazy segment downloads during iteration, a transient I/O error (e.g. S3 timeout) mid-iteration kills the generator.fetchone()then returnsNoneas if the query completed, silently dropping remaining rows.Root cause
Generator frame finalization is a Python language constraint, not a bug in the iteration logic itself. Once an exception propagates through
yield, the generator's frame is deallocated and cannot be resumed.Fix
Replace the generator with a class-based iterator (
__iter__returningself,__next__holding state in instance fields). The iteration logic is unchanged — batch prefetch viafetch()before exposing rows, same_rownumbertracking. The difference is that instance fields (_row_iter,_next_rows) survive exceptions, so callers can resume iteration after catching a transient error.Testing
Added
test_trino_result_iterator_survives_transient_error— injects aFailOnceIteratorthat raisesIOErroron the 3rd element, verifies the iterator resumes and returns remaining rows after the caller catches the exception.Full unit test suite passes (123 tests, 0 failures).