PRDCT-552: Generic Extractor under Components › Data Source Connectors (re-cut off main) - #1054
Merged
Merged
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Iamfle4ka
pushed a commit
that referenced
this pull request
Jul 29, 2026
…ified facts
Deep-review pass on the Generic Writer page set (Jordan, 2026-07-29): every
claim checked against a source of truth before re-requesting review.
Source of truth: component id kds-team.wr-generic, repo
bitbucket.org/kds_consulting_team/kds-team.wr-generic (public, live) —
src/configuration.py (schema), src/component.py (placeholders/iteration),
src/json_converter.py, src/user_functions.py, src/http_generic/auth.py, and
the recorded functional examples under docs/examples/001-037.
Rendering blocker (introduced by the phase-1 migration):
- The inlined writer-config-map include produced "```json {" / "} ```", which
is not a valid CommonMark fence pair. The result was one 168-line code block
swallowing lines 66-233, so "## Api", "### Base URL" and "### Retry Config"
rendered as literal text and the #api / #base-url / #retry-config anchors
did not exist. Headings 34 -> 37; longest code block 168 -> 84 lines.
- Removed the 46-line jQuery <script> + <style> inlined from
writer-config-events.js. There is no jQuery in this repo, its
span:contains() selectors targeted Jekyll/Rouge markup (this site uses
Expressive Code), and 6 of its hardcoded anchors were wrong.
- Reworded the config-map lede, which promised a clickable map that the
removed script used to provide.
Claims refuted by the component source:
- autodetect default is false, not true (ColumnDataTypes.autodetect = False).
- Four datatypes, not "three" (DataType enum); dropped the "(three levels)"
nesting limit, which no code enforces.
- Dropped UPDATE from the supported methods: it is not an HTTP verb and the
list was copied from python-http-client's never-enforced ALLOWED_METHODS.
Noted that the method is passed through unvalidated.
- base_url / endpoint_path placeholders are filled from iteration columns
only, not from user_parameters; documented that a base_url placeholder
resolves once and is reused for later rows.
- EMPTY_REQUEST sends one request without iterate_by_columns, not one per row.
- Iteration mode does not override chunk_size to 1; chunk_size still controls
payload shape (proven by docs/examples/005's recorded array request).
- Dynamic functions also evaluate in headers and query_parameters.
- json_mapping's nesting_delimiter / chunk_size / column_data_types are all
required when the section is present.
- Stale key names: request_options -> request_parameters (x2),
column_types -> column_data_types.datatype_override.
- [[id]] is canonical; {{id}} documented as a legacy alias.
- bool conversion also accepts 0 / "0" as False.
- retry_config codes: dropped leaked Python tuple notation.
- Fixed address__streed -> address__street and its invalid JSON.
- chunk_size sample 1 -> 2 to match the prose and the shown output.
Examples:
- Removed Authorization headers referencing an undefined token_encoded from
the Iterations and Customer.io examples: an {"attr": ...} with no matching
user parameter is a hard validation failure, so both configs could not run.
- Added the required "user_parameters": {} to the Customer.io example, which
otherwise raises KeyError: 'user_parameters'. All 6 JSON blocks now parse.
- Content-type application/csv -> application/json where the body is JSON.
- Slack example debug: true -> false (it prints payloads into the job log).
- Stripped the trailing "?" from all 4 endpoint_path values (a v1 converter
artifact).
- Replaced leaked @keboola.com addresses, including a real employee address,
with the @test.com values from the input table.
Links, anchors and typos:
- #enpoint-path -> #endpoint-path; empty [dynamic functions]() href.
- Rotted third-party links: Exponea -> Bloomreach Engagement (note that
.../reference/batch-commands is a soft 404, so .../batch-commands-2 is
used), Customer.io -> docs.customer.io/integrations/api/track/,
strftime.org (now a 403 vendor page) -> docs.python.org.
- index.md: relative overview/encryption/ link, which has no help target yet,
now points at the live dev URL; trailing slash on the examples link;
"encripted" and 7 other inherited typos; 3 sections that described "default
query parameters" regardless of their actual subject.
- Cross-linked Generic Writer from the Writers hub and Other hub pages.
Verified: build clean (259 pages); audit-phase2 unclosed code fences 1 -> 0,
0 missing images, 0 malformed tables, broken links unchanged at 6 (the 3 from
these pages are the known forward-refs to the Generic Extractor batch, #1054);
all 29 self-anchors resolve; the rendered config map parses as valid JSON;
all 3 /extend/generic-writer/* redirects resolve.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This was referenced Aug 3, 2026
Iamfle4ka
pushed a commit
that referenced
this pull request
Aug 3, 2026
The map still routed the 42 dev /extend/** pages to /components/development/**, a section that was never built: Jordan approved /extend/ as its own top-level section instead (#1046, merged 2026-07-29, commit 8504c9d). Every one of those 42 targets 404'd, so deleting the dev pages would have orphaned their URLs. #1046 preserved the dev path structure 1:1, so these become identity redirects across domains (developers.keboola.com/extend/X/ -> help.keboola.com/extend/X/). Verified: all 42 targets now resolve in a production build; conservation failures drop from 162 to 117, and the remaining 117 are units whose help pages are not on main yet (#1023, #1051-#1054, artifacts, jobs, overview/api, encryption, CLI). The NEW-sections table entry is marked superseded rather than deleted so the provenance of the change stays readable. Left untouched: the 3 rows still targeting /components/development/ (artifacts x2, overview/encryption). They are flagged `unsure` and are genuine open owner calls. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Iamfle4ka
pushed a commit
that referenced
this pull request
Aug 4, 2026
main moved 118 commits since the baseline was captured, so the gate went red on findings that already exist there (mostly /extend/* links waiting on #1053 / #1054). Re-baselined on the merged tree: 1155 → 1353 known findings, gate back to exit 0, so only new regressions fail. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Iamfle4ka
force-pushed
the
PRDCT-552-generic-extractor-recut
branch
from
August 4, 2026 23:40
cbdf987 to
cdd6a64
Compare
Iamfle4ka
pushed a commit
that referenced
this pull request
Aug 4, 2026
…ified facts
Deep-review pass on the Generic Writer page set (Jordan, 2026-07-29): every
claim checked against a source of truth before re-requesting review.
Source of truth: component id kds-team.wr-generic, repo
bitbucket.org/kds_consulting_team/kds-team.wr-generic (public, live) —
src/configuration.py (schema), src/component.py (placeholders/iteration),
src/json_converter.py, src/user_functions.py, src/http_generic/auth.py, and
the recorded functional examples under docs/examples/001-037.
Rendering blocker (introduced by the phase-1 migration):
- The inlined writer-config-map include produced "```json {" / "} ```", which
is not a valid CommonMark fence pair. The result was one 168-line code block
swallowing lines 66-233, so "## Api", "### Base URL" and "### Retry Config"
rendered as literal text and the #api / #base-url / #retry-config anchors
did not exist. Headings 34 -> 37; longest code block 168 -> 84 lines.
- Removed the 46-line jQuery <script> + <style> inlined from
writer-config-events.js. There is no jQuery in this repo, its
span:contains() selectors targeted Jekyll/Rouge markup (this site uses
Expressive Code), and 6 of its hardcoded anchors were wrong.
- Reworded the config-map lede, which promised a clickable map that the
removed script used to provide.
Claims refuted by the component source:
- autodetect default is false, not true (ColumnDataTypes.autodetect = False).
- Four datatypes, not "three" (DataType enum); dropped the "(three levels)"
nesting limit, which no code enforces.
- Dropped UPDATE from the supported methods: it is not an HTTP verb and the
list was copied from python-http-client's never-enforced ALLOWED_METHODS.
Noted that the method is passed through unvalidated.
- base_url / endpoint_path placeholders are filled from iteration columns
only, not from user_parameters; documented that a base_url placeholder
resolves once and is reused for later rows.
- EMPTY_REQUEST sends one request without iterate_by_columns, not one per row.
- Iteration mode does not override chunk_size to 1; chunk_size still controls
payload shape (proven by docs/examples/005's recorded array request).
- Dynamic functions also evaluate in headers and query_parameters.
- json_mapping's nesting_delimiter / chunk_size / column_data_types are all
required when the section is present.
- Stale key names: request_options -> request_parameters (x2),
column_types -> column_data_types.datatype_override.
- [[id]] is canonical; {{id}} documented as a legacy alias.
- bool conversion also accepts 0 / "0" as False.
- retry_config codes: dropped leaked Python tuple notation.
- Fixed address__streed -> address__street and its invalid JSON.
- chunk_size sample 1 -> 2 to match the prose and the shown output.
Examples:
- Removed Authorization headers referencing an undefined token_encoded from
the Iterations and Customer.io examples: an {"attr": ...} with no matching
user parameter is a hard validation failure, so both configs could not run.
- Added the required "user_parameters": {} to the Customer.io example, which
otherwise raises KeyError: 'user_parameters'. All 6 JSON blocks now parse.
- Content-type application/csv -> application/json where the body is JSON.
- Slack example debug: true -> false (it prints payloads into the job log).
- Stripped the trailing "?" from all 4 endpoint_path values (a v1 converter
artifact).
- Replaced leaked @keboola.com addresses, including a real employee address,
with the @test.com values from the input table.
Links, anchors and typos:
- #enpoint-path -> #endpoint-path; empty [dynamic functions]() href.
- Rotted third-party links: Exponea -> Bloomreach Engagement (note that
.../reference/batch-commands is a soft 404, so .../batch-commands-2 is
used), Customer.io -> docs.customer.io/integrations/api/track/,
strftime.org (now a 403 vendor page) -> docs.python.org.
- index.md: relative overview/encryption/ link, which has no help target yet,
now points at the live dev URL; trailing slash on the examples link;
"encripted" and 7 other inherited typos; 3 sections that described "default
query parameters" regardless of their actual subject.
- Cross-linked Generic Writer from the Writers hub and Other hub pages.
Verified: build clean (259 pages); audit-phase2 unclosed code fences 1 -> 0,
0 missing images, 0 malformed tables, broken links unchanged at 6 (the 3 from
these pages are the known forward-refs to the Generic Extractor batch, #1054);
all 29 self-anchors resolve; the rendered config map parses as valid JSON;
all 3 /extend/generic-writer/* redirects resolve.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…s (re-cut off main) Re-cut of phase-2 batch 2 (was #1031, stacked on #1027) as a standalone per-unit PR off main, per the 2026-07-27 pivot. Moves the 39 Generic Extractor reference pages to /components/extractors/generic-extractor/** — it is a component, documented with the other data source connectors (nav-titled "Generic Extractor Reference", next to the native /components/extractors/other/generic/ entry; they get woven together later). Old /extend/generic-extractor/* URLs keep working via redirect_from. Build clean (295 pp); audit 0 missing images, 0 links broken INTO these pages (no move breakage). Broken links are forward-refs to not-yet-migrated units (/extend/**, /overview/encryption/) — heal as those land. Supersedes the stacked #1031 (closed with the #1027 stack once the queue drains). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Full fact-check of the 39-page config reference against keboola/generic-extractor + keboola/juicer. Migration was drift-free; fixes are pre-existing dev-source defects, plus removing dead jQuery. - CRITICAL: api_key JSON examples were structurally invalid (double-nested http.http, and the query example used http.params) → fixed to api.http.headers and api.http.defaultOptions.params; corrected the concat→attr description. - MAJOR: default retry block put maxRetries inside http → moved to the retryConfig top level (RetryHandler reads it there). - MAJOR: removed the legacy jQuery <script>/<style> parameter-map blocks on the map + configuration-root pages (they relied on jQuery + Rouge span markup; dead under Astro/Starlight). The config JSON still renders; interactive deep-links dropped. - MINOR: fixed invalid JSON in examples (login stray slash, oauth_cc + functions missing/trailing commas), responseFilter→recursionFilter prose, 2-user-id→2:user-id placeholder, offset/response-url query typos, a garbled example URL, and a GET-section example mislabeled method POST → GET. Build clean (295 pp); audit 0 missing images, 0 links broken into these pages. Remaining broken links are forward-refs to not-yet-migrated units (/extend/**, /overview/encryption/). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ub with the migrated tree The recut added the full Generic Extractor tree as a second "Generic Extractor Reference" item while the old thin components/extractors/other/generic stub (which just points users to dev docs) was left in place — two "Generic Extractor" entries, one bouncing users off-site. - Delete the old stub (page + its src/ and public/ images). - Fold its only unique bit — the Template Mode section (+ generic-1/2.png) — into the migrated index so the #template-mode anchor survives on the canonical page. - Add redirect_from: /components/extractors/other/generic/ to the migrated index. - Retitle "Generic Extractor Reference" -> "Generic Extractor" (page + nav); drop the duplicate nav entry; regen sidebar. - Repoint all 13 inbound links (incl. 11 "switch to the JSON editor" #template-mode links) to the new slug. Build clean (294 pp); audit 0 missing images; the 19 broken links are pre-existing forward-refs in the migrated tree (heal as sibling units merge). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…able The automatic-mapping example table in configuration/config/mappings had a separator row one column wider than its header and data rows, so it rendered as literal pipes instead of a table. Caught by scripts/audit-phase2.mjs (MALFORMED TABLES 1 → 0). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Iamfle4ka
force-pushed
the
PRDCT-552-generic-extractor-recut
branch
from
August 4, 2026 23:44
cdd6a64 to
abfb69c
Compare
Iamfle4ka
pushed a commit
that referenced
this pull request
Aug 4, 2026
…ified facts
Deep-review pass on the Generic Writer page set (Jordan, 2026-07-29): every
claim checked against a source of truth before re-requesting review.
Source of truth: component id kds-team.wr-generic, repo
bitbucket.org/kds_consulting_team/kds-team.wr-generic (public, live) —
src/configuration.py (schema), src/component.py (placeholders/iteration),
src/json_converter.py, src/user_functions.py, src/http_generic/auth.py, and
the recorded functional examples under docs/examples/001-037.
Rendering blocker (introduced by the phase-1 migration):
- The inlined writer-config-map include produced "```json {" / "} ```", which
is not a valid CommonMark fence pair. The result was one 168-line code block
swallowing lines 66-233, so "## Api", "### Base URL" and "### Retry Config"
rendered as literal text and the #api / #base-url / #retry-config anchors
did not exist. Headings 34 -> 37; longest code block 168 -> 84 lines.
- Removed the 46-line jQuery <script> + <style> inlined from
writer-config-events.js. There is no jQuery in this repo, its
span:contains() selectors targeted Jekyll/Rouge markup (this site uses
Expressive Code), and 6 of its hardcoded anchors were wrong.
- Reworded the config-map lede, which promised a clickable map that the
removed script used to provide.
Claims refuted by the component source:
- autodetect default is false, not true (ColumnDataTypes.autodetect = False).
- Four datatypes, not "three" (DataType enum); dropped the "(three levels)"
nesting limit, which no code enforces.
- Dropped UPDATE from the supported methods: it is not an HTTP verb and the
list was copied from python-http-client's never-enforced ALLOWED_METHODS.
Noted that the method is passed through unvalidated.
- base_url / endpoint_path placeholders are filled from iteration columns
only, not from user_parameters; documented that a base_url placeholder
resolves once and is reused for later rows.
- EMPTY_REQUEST sends one request without iterate_by_columns, not one per row.
- Iteration mode does not override chunk_size to 1; chunk_size still controls
payload shape (proven by docs/examples/005's recorded array request).
- Dynamic functions also evaluate in headers and query_parameters.
- json_mapping's nesting_delimiter / chunk_size / column_data_types are all
required when the section is present.
- Stale key names: request_options -> request_parameters (x2),
column_types -> column_data_types.datatype_override.
- [[id]] is canonical; {{id}} documented as a legacy alias.
- bool conversion also accepts 0 / "0" as False.
- retry_config codes: dropped leaked Python tuple notation.
- Fixed address__streed -> address__street and its invalid JSON.
- chunk_size sample 1 -> 2 to match the prose and the shown output.
Examples:
- Removed Authorization headers referencing an undefined token_encoded from
the Iterations and Customer.io examples: an {"attr": ...} with no matching
user parameter is a hard validation failure, so both configs could not run.
- Added the required "user_parameters": {} to the Customer.io example, which
otherwise raises KeyError: 'user_parameters'. All 6 JSON blocks now parse.
- Content-type application/csv -> application/json where the body is JSON.
- Slack example debug: true -> false (it prints payloads into the job log).
- Stripped the trailing "?" from all 4 endpoint_path values (a v1 converter
artifact).
- Replaced leaked @keboola.com addresses, including a real employee address,
with the @test.com values from the input table.
Links, anchors and typos:
- #enpoint-path -> #endpoint-path; empty [dynamic functions]() href.
- Rotted third-party links: Exponea -> Bloomreach Engagement (note that
.../reference/batch-commands is a soft 404, so .../batch-commands-2 is
used), Customer.io -> docs.customer.io/integrations/api/track/,
strftime.org (now a 403 vendor page) -> docs.python.org.
- index.md: relative overview/encryption/ link, which has no help target yet,
now points at the live dev URL; trailing slash on the examples link;
"encripted" and 7 other inherited typos; 3 sections that described "default
query parameters" regardless of their actual subject.
- Cross-linked Generic Writer from the Writers hub and Other hub pages.
Verified: build clean (259 pages); audit-phase2 unclosed code fences 1 -> 0,
0 missing images, 0 malformed tables, broken links unchanged at 6 (the 3 from
these pages are the known forward-refs to the Generic Extractor batch, #1054);
all 29 self-anchors resolve; the rendered config map parses as valid JSON;
all 3 /extend/generic-writer/* redirects resolve.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
added 2 commits
August 7, 2026 01:22
…xtractor-recut # Conflicts: # src/content/docs/components/extractors/other/index.md
…ic Extractor pages The pages this PR migrates predate #1070's corpus-wide sweep, so they would reintroduce the literal `---` the sweep just removed. Same transformation, same scope rule: prose dashes only, nothing inside code fences.
This was referenced Aug 19, 2026
This was referenced Aug 19, 2026
…'t lost at retire The dev-side audit consolidation (PRDCT-365, developers-docs#410) fixes six things on pages this PR ports; the port was cut before that PR, so the recut still carried the broken text. Applying them here keeps #410 free to merge on dev while the retire stubs win later. - pagination: nextFlag -> nextPageFlag; count/field are mutually required, not both; stray ']' after the EX139 link - aws-signature: 'signaturing requests by AWS' -> proper Signature V4 wording - config: un-garble the userData column-collision sentence - ssh-proxy: 'tu' -> 'to' Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
davidesner
requested changes
Aug 25, 2026
davidesner
left a comment
Contributor
There was a problem hiding this comment.
LGTM I just found some relicts from the original version, that can be fixed straight away
- nav: 'Generic Extractor Tutorial' -> 'Tutorial' (redundant under the Generic Extractor node, and it clipped in the sidebar) - configuration: drop the Public Beta caution on the UI section — the UI is not a beta anymore - running: drop the old-UI configuration screenshot (and its now-orphaned image file) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Collaborator
Author
|
Thanks @davidesner — all three fixed in 7fd8382: nav title shortened to "Tutorial", the Public Beta caution dropped (not a beta anymore), and the old-UI screenshot removed from Running (the orphaned image file deleted with it). Re-requested your review. 🤖 Generated with Claude Code |
Iamfle4ka
pushed a commit
that referenced
this pull request
Sep 2, 2026
…added The generic-extractor recut (#1054) landed without descriptions; the generator fills them from each page's lead prose. The retired extractors/other/generic page follows main's deletion. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
Re-cut of phase-2 batch 2 (was #1031, stacked on the big-bang #1027) as a standalone per-unit PR off
main, per the 2026-07-27 pivot — same content/placement, no #1027 dependency.Moves the 39 Generic Extractor reference pages to
/components/extractors/generic-extractor/**— it is a component, so it lives with the other data source connectors. Nav-titled "Generic Extractor Reference", next to the native/components/extractors/other/generic/user-facing entry (the two get woven together later). Old/extend/generic-extractor/*URLs keep working viaredirect_from.Verification
npm run buildclean — 295 pages.audit-phase2.mjs: 0 missing images, 0 links broken into these pages (no move breakage). Remaining broken links are forward-refs to not-yet-migrated units (/extend/**from PRDCT-543: Extending Keboola as its own top-level section #1046,/overview/encryption/) — heal as those land.Supersedes
Replaces the stacked #1031 (closed with the #1027 stack once the per-unit queue is drained).
🤖 Generated with Claude Code
Placement / dedup (2026-07-31)
The recut originally added this as a second "Generic Extractor Reference" item while the
pre-existing thin stub
components/extractors/other/generic(which just points users to devdocs) was left in place — two "Generic Extractor" entries, one bouncing off-site. Fixed:
src/andpublic/images).generic-1/2.png) — into themigrated index, so the
#template-modeanchor survives on the canonical page.redirect_from: /components/extractors/other/generic/.duplicate nav entry.
#template-modelinks) to the new slug.
Open decision for the owner: where should Generic Extractor live — first-class under Data
Source Connectors (current, my rec), inside "Other", or its own category?