fix(low-code): preserve check config_overrides during parameter propagation - #1125
fix(low-code): preserve check config_overrides during parameter propagation#1125devin-ai-integration[bot] wants to merge 6 commits into
Conversation
A check and a sync legitimately want different behaviour from the same manifest. A check is interactive and should fail fast with an actionable message; a sync can afford to wait out a rate limit window. Today the only way to express that difference is a Python `check_connection` override that builds a second component tree from a modified config, which a manifest-only connector cannot do. `CheckStream` and `CheckDynamicStream` gain an optional `config_overrides` mapping. `check()` overlays it onto the config for the duration of the check, which is enough to reach every component the checker builds because `streams()` interpolates from `self._config` and ignores its own `config` argument. Values are applied verbatim and are not interpolated. `config_validations` continue to run against the config the user supplied, so an override cannot fail a validation the user has no way to satisfy. Inert by default: with no `config_overrides` key, `check()` behaves exactly as before. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Follow-up on the same branch, addressing a local review of the feature. Documents three semantics the field description and docstring left implicit: overrides are merged one level deep so a nested object is replaced rather than deep-merged; they are applied after config migrations and transformations, so an override is not normalised and derived fields are not recomputed; and the overlay mutates shared state, which is safe only because check is one command per process. Renames `_user_provided_config` to `_config_for_validation`. It holds the config after migrations and transformations have run, not what the user typed, and the old name plus its comment invited the wrong reading. Adds tests for two contracts that were stated but unpinned: `CheckDynamicStream` is covered by the overlay, and override values are verbatim rather than interpolated. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Refuses `config_overrides` on a manifest that declares a `refresh_token_updater`. That turns an
`OAuthAuthenticator` into a `DeclarativeSingleUseRefreshTokenOauth2Authenticator`, which is handed the
config the component tree was built with - during a check, the overlay - and on refresh emits that
entire dict as a CONNECTOR_CONFIG control message for the platform to persist. A check-only override
would therefore become the connection's saved config and apply to every later sync, and restoring
`self._config` afterwards cannot recall a message already written to stdout. Threading the config
through more carefully would not help: the hazard is inherent to handing an overridden config to
something whose job is to write the config back. Until the emitter is fixed to emit the config it was
given plus only the token fields it owns, refusing the combination is the honest answer.
Refuses override keys prefixed with `__airbyte`. Those are the platform's channel into the config
rather than connector config - `CheckStream` reads `__airbyte_check_stream_names` out of the very
config this overlay writes to - and a manifest that wants to choose which streams a check tests
already has `stream_names`.
Logs the overridden keys at INFO, keys only, since an override may name a secret field. Warns when an
override key is absent from the spec's `connection_specification.properties`: the overlay is the one
part of the config nothing validates, so a typo is otherwise a silent no-op.
Corrects the shallow-copy paragraph in the docstring, which had it backwards. Because the copy is
shallow, every nested object is shared with the config the source was constructed with, so a write
into a nested path such as `("credentials", "access_token")` writes through to the user's config and
the restore does not undo it. Only a write to a top-level key is discarded.
Notes in `create_check_stream` and `create_check_dynamic_stream` that `model.config_overrides` is
deliberately unread there because the source applies it around the whole check operation, so nobody
wires it in twice or deletes it as dead.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`_manifest_writes_back_config` tested `refresh_token_updater` for truthiness. Every field of
`RefreshTokenUpdater` has a default and none are required, so `refresh_token_updater: {}` is a valid
way to take all of them - and it builds the same `DeclarativeSingleUseRefreshTokenOauth2Authenticator`
a populated one does, because the factory's `if model.refresh_token_updater:` tests a model instance,
which is always truthy. The transformer injects no `type` into it either, so the empty mapping stayed
empty and slipped past the guard, leaving the config-persistence hazard open on exactly the shape the
review reproduced.
Testing `is not None` matches the factory on all three shapes: `{}`, populated, and absent. The
rejection test is parametrized over the first two, and a new test covers the same OAuth authenticator
without the updater, so the scan is pinned as not rejecting every OAuth manifest.
Also document both rejections in the `config_overrides` description on `CheckStream` and
`CheckDynamicStream`, since neither the `__airbyte` prefix restriction nor the `refresh_token_updater`
one was discoverable before running a check.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…values The scan for `refresh_token_updater` walks the whole raw manifest rather than only recognised authenticators, and that coarseness is load-bearing: it runs before references are resolved, so an authenticator reached through a `$ref` is found only because the walk also visits `definitions`. The net was too wide in two places that hold config values rather than components. A connector whose spec declares a property named `refresh_token_updater` lost `config_overrides` entirely - every use refused, naming an authenticator feature the manifest never declared - and an override of a config field by that name was refused for the same reason. Neither key can contain an authenticator: `spec` exists only at the top level of a manifest, and `config_overrides` only on `CheckStream` and `CheckDynamicStream`, so skipping both subtrees removes the false positives without narrowing detection. Adds a regression test per false positive, plus one for a `refresh_token_updater` behind a `$ref`, so the property the coarse walk exists to provide is pinned against a future narrowing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…tion Co-Authored-By: bot_apk <apk@cognition.ai>
🤖 Devin AI EngineerI'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
|
👋 Greetings, Airbyte Team Member!Here are some helpful tips and reminders for your convenience. 💡 Show Tips and TricksTesting This CDK VersionYou can test this version of the CDK using the following: # Run the CLI from this branch:
uvx 'git+https://github.com/airbytehq/airbyte-python-cdk.git@devin/1787224085-fix-check-config-overrides-parameters#egg=airbyte-python-cdk[dev]' --help
# Update a connector to use the CDK from this branch ref:
cd airbyte-integrations/connectors/source-example
poe use-cdk-branch devin/1787224085-fix-check-config-overrides-parametersPR Slash CommandsAirbyte Maintainers can execute the following slash commands on your PR:
|
There was a problem hiding this comment.
Pull request overview
This PR fixes a low-code check() bug where config_overrides values (which are connector config objects, not declarative components) could be mutated during manifest parameter propagation when an override object contains a type field (e.g., OAuth credentials). The fix captures and deep-copies the check component’s config_overrides after $ref resolution but before ManifestComponentTransformer.propagate_types_and_parameters, and then uses that pristine copy when applying the check-time config overlay.
Changes:
- Extract and deep-copy
check.config_overridesbefore parameter propagation mutates the manifest. - Apply the check-time overlay from the extracted copy (instead of reading
config_overridesfrom the propagated manifest). - Add a regression unit test ensuring object-valued overrides containing
typeare not polluted with injected parameters.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| airbyte_cdk/sources/declarative/concurrent_declarative_source.py | Captures pristine config_overrides pre-propagation and uses it during check() config overlay to prevent parameter injection into override objects. |
| unit_tests/sources/declarative/checks/test_check_stream.py | Adds regression test that fails if parameter propagation injects keys into object-valued config_overrides entries with a type field. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
This PR targets PR #1122:
config_overridesonly exists on that branch, so this fix cannot be based onmain. It can be merged into #1122 or land immediately after it. If #1122 merges first, this will be retargeted tomain.Resolves https://github.com/airbytehq/airbyte-internal-issues/issues/16995:
Summary
check()readsconfig_overridesout ofself._source_config, which is the post-transform manifest.ManifestComponentTransformer.propagate_types_and_parameterstreats any nested mapping with a truthytypekey as a component, so when a manifest puts$parameterson the check component and an override value is an object carrying atypefield, the transformer injects the parameters plus a$parameterskey into that object — and the mangled object is what gets overlaid onto the connector config during check.Before, the overlay applied at check time was:
{"credentials": {"type": "oauth", "client_id": "x", "p": "v", "$parameters": {"p": "v"}}}The fix keeps the overrides out of the transformer's reach instead of teaching the shared transformer about field names (a field-name allowlist in
ManifestComponentTransformerwas explicitly rejected in the #1122 review thread — it would widen shared parser behaviour for a check-scoped feature)._pre_process_manifestnow deep-copies the check'sconfig_overridesbetween the two existing passes:References are already resolved at that point, so a
checkbehind a$refis still found; parameters have not been propagated yet, so the values are still as authored.check()then usesself._check_config_overridesrather thancheck.get("config_overrides").CheckDynamicStreamis covered too, since the overlay is read from the check definition and is checker-agnostic.Test Coverage
New test in
unit_tests/sources/declarative/checks/test_check_stream.py:test_given_object_valued_config_override_when_check_then_parameters_are_not_injected_into_it. It puts$parameterson the check component and an object-valued override containingtype, and has the stream echo the override object's keys into a query param, so the mocked request matches only if the object reached the stream exactly as authored. Verified it fails on this branch without the source change (assert_number_of_callsmismatch) and passes with it.Declarative-First Evaluation
No custom Python component is involved, and no declarative feature can address this: the corruption happens in the shared manifest transformer before any component is built, so
RecordFilter, transformations, or$refoverrides cannot prevent it. The fix stays in the check feature's own integration layer (ConcurrentDeclarativeSource), where the data is first read.Breaking Change Evaluation
Not breaking. No schema, spec, stream, primary key, cursor, or state format changes; the change only preserves values already authored in
config_overrides. This repo releases via semantic-pr-release-drafter, so no manual version bump or changelog edit is needed (fix:title drives a patch release).Requested by Devin Bot via the
/ai-fixworkflow.Link to Devin session: https://app.devin.ai/sessions/a9f68a5d7a5840dfa882111074154529