Symptom
A legacy (non-declarative) Stream that declares a cursor_field behaves incrementally even when the connection configures the stream as full_refresh:
- it EMITS real cursor state during a full-refresh sync, and
- it CONSUMES persisted cursor state during a full-refresh sync - the read comes back filtered.
Together these make a "full refresh" silently partial: a mid-job retry attempt resumes from the cursor emitted by the first attempt, and a connection switched from incremental back to full refresh keeps filtering by the stale cursor.
Root cause (refs at v7.17.4)
airbyte_cdk/sources/abstract_source.py L259-263: stream_instance.state = <persisted state> is assigned whenever the platform provides state - the configured sync mode is never consulted.
airbyte_cdk/sources/streams/core.py L526-533 (_checkpoint_mode): returns CheckpointMode.INCREMENTAL whenever is_resumable and cursor_field are set - again independent of the configured sync mode.
airbyte_cdk/sources/streams/core.py L344-345: is_resumable short-circuits to True when supports_incremental is True (i.e. whenever cursor_field is non-empty).
Consequence: the FullRefreshCheckpointReader (whose __ab_no_cursor_state_message marker is deliberately discarded on read-back, abstract_source.py L258-259) is bypassed for ANY cursor-bearing stream, even on a full-refresh-configured connection. The state a full-refresh sync emits is a normal cursor blob that nothing discards later.
Minimal reproduction
Take any legacy Stream subclass with a cursor_field, a state property and a state-driven request filter. Configure the stream as full_refresh in the catalog and run read twice, feeding the first run's emitted state into the second: run 1 emits a real cursor; run 2 issues filtered requests and returns a subset. Repeated end to end in airbytehq/airbyte#83704 with source-facebook-marketing (ad_creatives_from_ads): the pre-fix regression run shows the stream emitting cursor state where 6.0.2 emits only __ab_no_cursor_state_message, and a captured request_params confirms the cursor filter is applied under full refresh once state exists.
Impact
Every legacy connector that adds incremental support to an existing stream must hand-gate BOTH directions:
- the state-write path (do not set
self.state unless sync_mode == SyncMode.incremental), and
- the state-read path (drop the slice's persisted state for non-incremental runs).
Miss either one and full-refresh connections are exposed to silent partial reads after a retry or a sync-mode switch. Neither requirement is documented.
Suggested direction
Either consult the configured sync mode in _checkpoint_mode / the state assignment in AbstractSource._read_stream, or document the required connector-side double gate for legacy streams. The declarative/concurrent paths are unaffected; this is specifically the legacy Stream plumbing.
Precedent
Found during review of airbytehq/airbyte#83704 (source-facebook-marketing 6.1.0, making ad_creatives_from_ads incremental). The connector-level fixes and evidence: airbytehq/airbyte#83704 (comment) (write-side gate, regression evidence) and the follow-up commit gating the read side (80b78957).
Symptom
A legacy (non-declarative)
Streamthat declares acursor_fieldbehaves incrementally even when the connection configures the stream asfull_refresh:Together these make a "full refresh" silently partial: a mid-job retry attempt resumes from the cursor emitted by the first attempt, and a connection switched from incremental back to full refresh keeps filtering by the stale cursor.
Root cause (refs at v7.17.4)
airbyte_cdk/sources/abstract_source.pyL259-263:stream_instance.state = <persisted state>is assigned whenever the platform provides state - the configured sync mode is never consulted.airbyte_cdk/sources/streams/core.pyL526-533 (_checkpoint_mode): returnsCheckpointMode.INCREMENTALwheneveris_resumableandcursor_fieldare set - again independent of the configured sync mode.airbyte_cdk/sources/streams/core.pyL344-345:is_resumableshort-circuits toTruewhensupports_incrementalisTrue(i.e. whenevercursor_fieldis non-empty).Consequence: the
FullRefreshCheckpointReader(whose__ab_no_cursor_state_messagemarker is deliberately discarded on read-back,abstract_source.pyL258-259) is bypassed for ANY cursor-bearing stream, even on a full-refresh-configured connection. The state a full-refresh sync emits is a normal cursor blob that nothing discards later.Minimal reproduction
Take any legacy
Streamsubclass with acursor_field, astateproperty and a state-driven request filter. Configure the stream asfull_refreshin the catalog and runreadtwice, feeding the first run's emitted state into the second: run 1 emits a real cursor; run 2 issues filtered requests and returns a subset. Repeated end to end in airbytehq/airbyte#83704 withsource-facebook-marketing(ad_creatives_from_ads): the pre-fix regression run shows the stream emitting cursor state where6.0.2emits only__ab_no_cursor_state_message, and a capturedrequest_paramsconfirms the cursor filter is applied under full refresh once state exists.Impact
Every legacy connector that adds incremental support to an existing stream must hand-gate BOTH directions:
self.stateunlesssync_mode == SyncMode.incremental), andMiss either one and full-refresh connections are exposed to silent partial reads after a retry or a sync-mode switch. Neither requirement is documented.
Suggested direction
Either consult the configured sync mode in
_checkpoint_mode/ the state assignment inAbstractSource._read_stream, or document the required connector-side double gate for legacy streams. The declarative/concurrent paths are unaffected; this is specifically the legacyStreamplumbing.Precedent
Found during review of airbytehq/airbyte#83704 (source-facebook-marketing 6.1.0, making
ad_creatives_from_adsincremental). The connector-level fixes and evidence: airbytehq/airbyte#83704 (comment) (write-side gate, regression evidence) and the follow-up commit gating the read side (80b78957).