Skip to content

Legacy Stream path: checkpoint reader and state assignment ignore the configured sync mode - cursor-bearing streams leak and consume cursor state under full refresh #1112

Description

@bazarnov

Symptom

A legacy (non-declarative) Stream that declares a cursor_field behaves incrementally even when the connection configures the stream as full_refresh:

  1. it EMITS real cursor state during a full-refresh sync, and
  2. it CONSUMES persisted cursor state during a full-refresh sync - the read comes back filtered.

Together these make a "full refresh" silently partial: a mid-job retry attempt resumes from the cursor emitted by the first attempt, and a connection switched from incremental back to full refresh keeps filtering by the stale cursor.

Root cause (refs at v7.17.4)

  • airbyte_cdk/sources/abstract_source.py L259-263: stream_instance.state = <persisted state> is assigned whenever the platform provides state - the configured sync mode is never consulted.
  • airbyte_cdk/sources/streams/core.py L526-533 (_checkpoint_mode): returns CheckpointMode.INCREMENTAL whenever is_resumable and cursor_field are set - again independent of the configured sync mode.
  • airbyte_cdk/sources/streams/core.py L344-345: is_resumable short-circuits to True when supports_incremental is True (i.e. whenever cursor_field is non-empty).

Consequence: the FullRefreshCheckpointReader (whose __ab_no_cursor_state_message marker is deliberately discarded on read-back, abstract_source.py L258-259) is bypassed for ANY cursor-bearing stream, even on a full-refresh-configured connection. The state a full-refresh sync emits is a normal cursor blob that nothing discards later.

Minimal reproduction

Take any legacy Stream subclass with a cursor_field, a state property and a state-driven request filter. Configure the stream as full_refresh in the catalog and run read twice, feeding the first run's emitted state into the second: run 1 emits a real cursor; run 2 issues filtered requests and returns a subset. Repeated end to end in airbytehq/airbyte#83704 with source-facebook-marketing (ad_creatives_from_ads): the pre-fix regression run shows the stream emitting cursor state where 6.0.2 emits only __ab_no_cursor_state_message, and a captured request_params confirms the cursor filter is applied under full refresh once state exists.

Impact

Every legacy connector that adds incremental support to an existing stream must hand-gate BOTH directions:

  • the state-write path (do not set self.state unless sync_mode == SyncMode.incremental), and
  • the state-read path (drop the slice's persisted state for non-incremental runs).

Miss either one and full-refresh connections are exposed to silent partial reads after a retry or a sync-mode switch. Neither requirement is documented.

Suggested direction

Either consult the configured sync mode in _checkpoint_mode / the state assignment in AbstractSource._read_stream, or document the required connector-side double gate for legacy streams. The declarative/concurrent paths are unaffected; this is specifically the legacy Stream plumbing.

Precedent

Found during review of airbytehq/airbyte#83704 (source-facebook-marketing 6.1.0, making ad_creatives_from_ads incremental). The connector-level fixes and evidence: airbytehq/airbyte#83704 (comment) (write-side gate, regression evidence) and the follow-up commit gating the read side (80b78957).

Metadata

Metadata

Assignees

No one assigned

    Labels

    communityPRs and issues from community contributors

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions