Skip to content

PRDCT-552: Generic Extractor under Components › Data Source Connectors (re-cut off main) - #1054

Merged
Iamfle4ka merged 9 commits into
mainfrom
PRDCT-552-generic-extractor-recut
Aug 26, 2026
Merged

Iamfle4ka merged 9 commits into
mainfrom
PRDCT-552-generic-extractor-recut

Conversation

@Iamfle4ka

@Iamfle4ka Iamfle4ka commented Jul 28, 2026 •

Copy link
Copy Markdown
Collaborator

What & why

Re-cut of phase-2 batch 2 (was #1031, stacked on the big-bang #1027) as a standalone per-unit PR off main, per the 2026-07-27 pivot — same content/placement, no #1027 dependency.

Moves the 39 Generic Extractor reference pages to /components/extractors/generic-extractor/** — it is a component, so it lives with the other data source connectors. Nav-titled "Generic Extractor Reference", next to the native /components/extractors/other/generic/ user-facing entry (the two get woven together later). Old /extend/generic-extractor/* URLs keep working via redirect_from.

Verification

  • npm run build clean — 295 pages.
  • audit-phase2.mjs: 0 missing images, 0 links broken into these pages (no move breakage). Remaining broken links are forward-refs to not-yet-migrated units (/extend/** from PRDCT-543: Extending Keboola as its own top-level section #1046, /overview/encryption/) — heal as those land.

Supersedes

Replaces the stacked #1031 (closed with the #1027 stack once the per-unit queue is drained).

🤖 Generated with Claude Code

Placement / dedup (2026-07-31)

The recut originally added this as a second "Generic Extractor Reference" item while the
pre-existing thin stub components/extractors/other/generic (which just points users to dev
docs) was left in place — two "Generic Extractor" entries, one bouncing off-site. Fixed:

  • Deleted the old stub (page + its src/ and public/ images).
  • Folded its only unique bit — the Template Mode section (+ generic-1/2.png) — into the
    migrated index, so the #template-mode anchor survives on the canonical page.
  • Added redirect_from: /components/extractors/other/generic/.
  • Retitled "Generic Extractor Reference" → "Generic Extractor" (page + nav); dropped the
    duplicate nav entry.
  • Repointed all 13 inbound links (incl. 11 "switch to the JSON editor" #template-mode
    links) to the new slug.

Open decision for the owner: where should Generic Extractor live — first-class under Data
Source Connectors (current, my rec), inside "Other", or its own category?

@vercel

vercel Bot commented Jul 28, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
connection-docs Ready Ready Preview Aug 26, 2026 4:52pm

Request Review

@linear-code

linear-code Bot commented Jul 28, 2026

Copy link
Copy Markdown

PRDCT-552

@Iamfle4ka Iamfle4ka added the dev-docs-migration developers.keboola.com → help.keboola.com migration label Jul 28, 2026
Iamfle4ka pushed a commit that referenced this pull request Jul 29, 2026
…ified facts

Deep-review pass on the Generic Writer page set (Jordan, 2026-07-29): every
claim checked against a source of truth before re-requesting review.

Source of truth: component id kds-team.wr-generic, repo
bitbucket.org/kds_consulting_team/kds-team.wr-generic (public, live) —
src/configuration.py (schema), src/component.py (placeholders/iteration),
src/json_converter.py, src/user_functions.py, src/http_generic/auth.py, and
the recorded functional examples under docs/examples/001-037.

Rendering blocker (introduced by the phase-1 migration):
- The inlined writer-config-map include produced "```json {" / "} ```", which
  is not a valid CommonMark fence pair. The result was one 168-line code block
  swallowing lines 66-233, so "## Api", "### Base URL" and "### Retry Config"
  rendered as literal text and the #api / #base-url / #retry-config anchors
  did not exist. Headings 34 -> 37; longest code block 168 -> 84 lines.
- Removed the 46-line jQuery <script> + <style> inlined from
  writer-config-events.js. There is no jQuery in this repo, its
  span:contains() selectors targeted Jekyll/Rouge markup (this site uses
  Expressive Code), and 6 of its hardcoded anchors were wrong.
- Reworded the config-map lede, which promised a clickable map that the
  removed script used to provide.

Claims refuted by the component source:
- autodetect default is false, not true (ColumnDataTypes.autodetect = False).
- Four datatypes, not "three" (DataType enum); dropped the "(three levels)"
  nesting limit, which no code enforces.
- Dropped UPDATE from the supported methods: it is not an HTTP verb and the
  list was copied from python-http-client's never-enforced ALLOWED_METHODS.
  Noted that the method is passed through unvalidated.
- base_url / endpoint_path placeholders are filled from iteration columns
  only, not from user_parameters; documented that a base_url placeholder
  resolves once and is reused for later rows.
- EMPTY_REQUEST sends one request without iterate_by_columns, not one per row.
- Iteration mode does not override chunk_size to 1; chunk_size still controls
  payload shape (proven by docs/examples/005's recorded array request).
- Dynamic functions also evaluate in headers and query_parameters.
- json_mapping's nesting_delimiter / chunk_size / column_data_types are all
  required when the section is present.
- Stale key names: request_options -> request_parameters (x2),
  column_types -> column_data_types.datatype_override.
- [[id]] is canonical; {{id}} documented as a legacy alias.
- bool conversion also accepts 0 / "0" as False.
- retry_config codes: dropped leaked Python tuple notation.
- Fixed address__streed -> address__street and its invalid JSON.
- chunk_size sample 1 -> 2 to match the prose and the shown output.

Examples:
- Removed Authorization headers referencing an undefined token_encoded from
  the Iterations and Customer.io examples: an {"attr": ...} with no matching
  user parameter is a hard validation failure, so both configs could not run.
- Added the required "user_parameters": {} to the Customer.io example, which
  otherwise raises KeyError: 'user_parameters'. All 6 JSON blocks now parse.
- Content-type application/csv -> application/json where the body is JSON.
- Slack example debug: true -> false (it prints payloads into the job log).
- Stripped the trailing "?" from all 4 endpoint_path values (a v1 converter
  artifact).
- Replaced leaked @keboola.com addresses, including a real employee address,
  with the @test.com values from the input table.

Links, anchors and typos:
- #enpoint-path -> #endpoint-path; empty [dynamic functions]() href.
- Rotted third-party links: Exponea -> Bloomreach Engagement (note that
  .../reference/batch-commands is a soft 404, so .../batch-commands-2 is
  used), Customer.io -> docs.customer.io/integrations/api/track/,
  strftime.org (now a 403 vendor page) -> docs.python.org.
- index.md: relative overview/encryption/ link, which has no help target yet,
  now points at the live dev URL; trailing slash on the examples link;
  "encripted" and 7 other inherited typos; 3 sections that described "default
  query parameters" regardless of their actual subject.
- Cross-linked Generic Writer from the Writers hub and Other hub pages.

Verified: build clean (259 pages); audit-phase2 unclosed code fences 1 -> 0,
0 missing images, 0 malformed tables, broken links unchanged at 6 (the 3 from
these pages are the known forward-refs to the Generic Extractor batch, #1054);
all 29 self-anchors resolve; the rendered config map parses as valid JSON;
all 3 /extend/generic-writer/* redirects resolve.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Iamfle4ka pushed a commit that referenced this pull request Aug 3, 2026
The map still routed the 42 dev /extend/** pages to /components/development/**,
a section that was never built: Jordan approved /extend/ as its own top-level
section instead (#1046, merged 2026-07-29, commit 8504c9d). Every one of those
42 targets 404'd, so deleting the dev pages would have orphaned their URLs.

#1046 preserved the dev path structure 1:1, so these become identity redirects
across domains (developers.keboola.com/extend/X/ -> help.keboola.com/extend/X/).
Verified: all 42 targets now resolve in a production build; conservation failures
drop from 162 to 117, and the remaining 117 are units whose help pages are not on
main yet (#1023, #1051-#1054, artifacts, jobs, overview/api, encryption, CLI).

The NEW-sections table entry is marked superseded rather than deleted so the
provenance of the change stays readable.

Left untouched: the 3 rows still targeting /components/development/ (artifacts x2,
overview/encryption). They are flagged `unsure` and are genuine open owner calls.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Iamfle4ka pushed a commit that referenced this pull request Aug 4, 2026
main moved 118 commits since the baseline was captured, so the gate went
red on findings that already exist there (mostly /extend/* links waiting
on #1053 / #1054). Re-baselined on the merged tree: 1155 → 1353 known
findings, gate back to exit 0, so only new regressions fail.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@Iamfle4ka
Iamfle4ka force-pushed the PRDCT-552-generic-extractor-recut branch from cbdf987 to cdd6a64 Compare August 4, 2026 23:40
Iamfle4ka pushed a commit that referenced this pull request Aug 4, 2026
…ified facts

Deep-review pass on the Generic Writer page set (Jordan, 2026-07-29): every
claim checked against a source of truth before re-requesting review.

Source of truth: component id kds-team.wr-generic, repo
bitbucket.org/kds_consulting_team/kds-team.wr-generic (public, live) —
src/configuration.py (schema), src/component.py (placeholders/iteration),
src/json_converter.py, src/user_functions.py, src/http_generic/auth.py, and
the recorded functional examples under docs/examples/001-037.

Rendering blocker (introduced by the phase-1 migration):
- The inlined writer-config-map include produced "```json {" / "} ```", which
  is not a valid CommonMark fence pair. The result was one 168-line code block
  swallowing lines 66-233, so "## Api", "### Base URL" and "### Retry Config"
  rendered as literal text and the #api / #base-url / #retry-config anchors
  did not exist. Headings 34 -> 37; longest code block 168 -> 84 lines.
- Removed the 46-line jQuery <script> + <style> inlined from
  writer-config-events.js. There is no jQuery in this repo, its
  span:contains() selectors targeted Jekyll/Rouge markup (this site uses
  Expressive Code), and 6 of its hardcoded anchors were wrong.
- Reworded the config-map lede, which promised a clickable map that the
  removed script used to provide.

Claims refuted by the component source:
- autodetect default is false, not true (ColumnDataTypes.autodetect = False).
- Four datatypes, not "three" (DataType enum); dropped the "(three levels)"
  nesting limit, which no code enforces.
- Dropped UPDATE from the supported methods: it is not an HTTP verb and the
  list was copied from python-http-client's never-enforced ALLOWED_METHODS.
  Noted that the method is passed through unvalidated.
- base_url / endpoint_path placeholders are filled from iteration columns
  only, not from user_parameters; documented that a base_url placeholder
  resolves once and is reused for later rows.
- EMPTY_REQUEST sends one request without iterate_by_columns, not one per row.
- Iteration mode does not override chunk_size to 1; chunk_size still controls
  payload shape (proven by docs/examples/005's recorded array request).
- Dynamic functions also evaluate in headers and query_parameters.
- json_mapping's nesting_delimiter / chunk_size / column_data_types are all
  required when the section is present.
- Stale key names: request_options -> request_parameters (x2),
  column_types -> column_data_types.datatype_override.
- [[id]] is canonical; {{id}} documented as a legacy alias.
- bool conversion also accepts 0 / "0" as False.
- retry_config codes: dropped leaked Python tuple notation.
- Fixed address__streed -> address__street and its invalid JSON.
- chunk_size sample 1 -> 2 to match the prose and the shown output.

Examples:
- Removed Authorization headers referencing an undefined token_encoded from
  the Iterations and Customer.io examples: an {"attr": ...} with no matching
  user parameter is a hard validation failure, so both configs could not run.
- Added the required "user_parameters": {} to the Customer.io example, which
  otherwise raises KeyError: 'user_parameters'. All 6 JSON blocks now parse.
- Content-type application/csv -> application/json where the body is JSON.
- Slack example debug: true -> false (it prints payloads into the job log).
- Stripped the trailing "?" from all 4 endpoint_path values (a v1 converter
  artifact).
- Replaced leaked @keboola.com addresses, including a real employee address,
  with the @test.com values from the input table.

Links, anchors and typos:
- #enpoint-path -> #endpoint-path; empty [dynamic functions]() href.
- Rotted third-party links: Exponea -> Bloomreach Engagement (note that
  .../reference/batch-commands is a soft 404, so .../batch-commands-2 is
  used), Customer.io -> docs.customer.io/integrations/api/track/,
  strftime.org (now a 403 vendor page) -> docs.python.org.
- index.md: relative overview/encryption/ link, which has no help target yet,
  now points at the live dev URL; trailing slash on the examples link;
  "encripted" and 7 other inherited typos; 3 sections that described "default
  query parameters" regardless of their actual subject.
- Cross-linked Generic Writer from the Writers hub and Other hub pages.

Verified: build clean (259 pages); audit-phase2 unclosed code fences 1 -> 0,
0 missing images, 0 malformed tables, broken links unchanged at 6 (the 3 from
these pages are the known forward-refs to the Generic Extractor batch, #1054);
all 29 self-anchors resolve; the rendered config map parses as valid JSON;
all 3 /extend/generic-writer/* redirects resolve.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Nikita and others added 4 commits August 5, 2026 01:42
…s (re-cut off main)

Re-cut of phase-2 batch 2 (was #1031, stacked on #1027) as a standalone
per-unit PR off main, per the 2026-07-27 pivot.

Moves the 39 Generic Extractor reference pages to
/components/extractors/generic-extractor/** — it is a component, documented
with the other data source connectors (nav-titled "Generic Extractor
Reference", next to the native /components/extractors/other/generic/ entry;
they get woven together later). Old /extend/generic-extractor/* URLs keep
working via redirect_from.

Build clean (295 pp); audit 0 missing images, 0 links broken INTO these pages
(no move breakage). Broken links are forward-refs to not-yet-migrated units
(/extend/**, /overview/encryption/) — heal as those land.

Supersedes the stacked #1031 (closed with the #1027 stack once the queue drains).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Full fact-check of the 39-page config reference against keboola/generic-extractor
+ keboola/juicer. Migration was drift-free; fixes are pre-existing dev-source
defects, plus removing dead jQuery.

- CRITICAL: api_key JSON examples were structurally invalid (double-nested
  http.http, and the query example used http.params) → fixed to api.http.headers
  and api.http.defaultOptions.params; corrected the concat→attr description.
- MAJOR: default retry block put maxRetries inside http → moved to the retryConfig
  top level (RetryHandler reads it there).
- MAJOR: removed the legacy jQuery <script>/<style> parameter-map blocks on the
  map + configuration-root pages (they relied on jQuery + Rouge span markup; dead
  under Astro/Starlight). The config JSON still renders; interactive deep-links dropped.
- MINOR: fixed invalid JSON in examples (login stray slash, oauth_cc + functions
  missing/trailing commas), responseFilter→recursionFilter prose, 2-user-id→2:user-id
  placeholder, offset/response-url query typos, a garbled example URL, and a
  GET-section example mislabeled method POST → GET.

Build clean (295 pp); audit 0 missing images, 0 links broken into these pages.
Remaining broken links are forward-refs to not-yet-migrated units (/extend/**,
/overview/encryption/).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ub with the migrated tree

The recut added the full Generic Extractor tree as a second "Generic Extractor
Reference" item while the old thin components/extractors/other/generic stub (which
just points users to dev docs) was left in place — two "Generic Extractor" entries,
one bouncing users off-site.

- Delete the old stub (page + its src/ and public/ images).
- Fold its only unique bit — the Template Mode section (+ generic-1/2.png) — into the
  migrated index so the #template-mode anchor survives on the canonical page.
- Add redirect_from: /components/extractors/other/generic/ to the migrated index.
- Retitle "Generic Extractor Reference" -> "Generic Extractor" (page + nav); drop the
  duplicate nav entry; regen sidebar.
- Repoint all 13 inbound links (incl. 11 "switch to the JSON editor" #template-mode
  links) to the new slug.

Build clean (294 pp); audit 0 missing images; the 19 broken links are pre-existing
forward-refs in the migrated tree (heal as sibling units merge).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…able

The automatic-mapping example table in configuration/config/mappings had a
separator row one column wider than its header and data rows, so it rendered
as literal pipes instead of a table. Caught by scripts/audit-phase2.mjs
(MALFORMED TABLES 1 → 0).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@Iamfle4ka
Iamfle4ka force-pushed the PRDCT-552-generic-extractor-recut branch from cdd6a64 to abfb69c Compare August 4, 2026 23:44
Iamfle4ka pushed a commit that referenced this pull request Aug 4, 2026
…ified facts

Deep-review pass on the Generic Writer page set (Jordan, 2026-07-29): every
claim checked against a source of truth before re-requesting review.

Source of truth: component id kds-team.wr-generic, repo
bitbucket.org/kds_consulting_team/kds-team.wr-generic (public, live) —
src/configuration.py (schema), src/component.py (placeholders/iteration),
src/json_converter.py, src/user_functions.py, src/http_generic/auth.py, and
the recorded functional examples under docs/examples/001-037.

Rendering blocker (introduced by the phase-1 migration):
- The inlined writer-config-map include produced "```json {" / "} ```", which
  is not a valid CommonMark fence pair. The result was one 168-line code block
  swallowing lines 66-233, so "## Api", "### Base URL" and "### Retry Config"
  rendered as literal text and the #api / #base-url / #retry-config anchors
  did not exist. Headings 34 -> 37; longest code block 168 -> 84 lines.
- Removed the 46-line jQuery <script> + <style> inlined from
  writer-config-events.js. There is no jQuery in this repo, its
  span:contains() selectors targeted Jekyll/Rouge markup (this site uses
  Expressive Code), and 6 of its hardcoded anchors were wrong.
- Reworded the config-map lede, which promised a clickable map that the
  removed script used to provide.

Claims refuted by the component source:
- autodetect default is false, not true (ColumnDataTypes.autodetect = False).
- Four datatypes, not "three" (DataType enum); dropped the "(three levels)"
  nesting limit, which no code enforces.
- Dropped UPDATE from the supported methods: it is not an HTTP verb and the
  list was copied from python-http-client's never-enforced ALLOWED_METHODS.
  Noted that the method is passed through unvalidated.
- base_url / endpoint_path placeholders are filled from iteration columns
  only, not from user_parameters; documented that a base_url placeholder
  resolves once and is reused for later rows.
- EMPTY_REQUEST sends one request without iterate_by_columns, not one per row.
- Iteration mode does not override chunk_size to 1; chunk_size still controls
  payload shape (proven by docs/examples/005's recorded array request).
- Dynamic functions also evaluate in headers and query_parameters.
- json_mapping's nesting_delimiter / chunk_size / column_data_types are all
  required when the section is present.
- Stale key names: request_options -> request_parameters (x2),
  column_types -> column_data_types.datatype_override.
- [[id]] is canonical; {{id}} documented as a legacy alias.
- bool conversion also accepts 0 / "0" as False.
- retry_config codes: dropped leaked Python tuple notation.
- Fixed address__streed -> address__street and its invalid JSON.
- chunk_size sample 1 -> 2 to match the prose and the shown output.

Examples:
- Removed Authorization headers referencing an undefined token_encoded from
  the Iterations and Customer.io examples: an {"attr": ...} with no matching
  user parameter is a hard validation failure, so both configs could not run.
- Added the required "user_parameters": {} to the Customer.io example, which
  otherwise raises KeyError: 'user_parameters'. All 6 JSON blocks now parse.
- Content-type application/csv -> application/json where the body is JSON.
- Slack example debug: true -> false (it prints payloads into the job log).
- Stripped the trailing "?" from all 4 endpoint_path values (a v1 converter
  artifact).
- Replaced leaked @keboola.com addresses, including a real employee address,
  with the @test.com values from the input table.

Links, anchors and typos:
- #enpoint-path -> #endpoint-path; empty [dynamic functions]() href.
- Rotted third-party links: Exponea -> Bloomreach Engagement (note that
  .../reference/batch-commands is a soft 404, so .../batch-commands-2 is
  used), Customer.io -> docs.customer.io/integrations/api/track/,
  strftime.org (now a 403 vendor page) -> docs.python.org.
- index.md: relative overview/encryption/ link, which has no help target yet,
  now points at the live dev URL; trailing slash on the examples link;
  "encripted" and 7 other inherited typos; 3 sections that described "default
  query parameters" regardless of their actual subject.
- Cross-linked Generic Writer from the Writers hub and Other hub pages.

Verified: build clean (259 pages); audit-phase2 unclosed code fences 1 -> 0,
0 missing images, 0 malformed tables, broken links unchanged at 6 (the 3 from
these pages are the known forward-refs to the Generic Extractor batch, #1054);
all 29 self-anchors resolve; the rendered config map parses as valid JSON;
all 3 /extend/generic-writer/* redirects resolve.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Nikita added 2 commits August 7, 2026 01:22
…xtractor-recut

# Conflicts:
#	src/content/docs/components/extractors/other/index.md
…ic Extractor pages

The pages this PR migrates predate #1070's corpus-wide sweep, so they would
reintroduce the literal `---` the sweep just removed. Same transformation,
same scope rule: prose dashes only, nothing inside code fences.
…'t lost at retire

The dev-side audit consolidation (PRDCT-365, developers-docs#410) fixes six
things on pages this PR ports; the port was cut before that PR, so the recut
still carried the broken text. Applying them here keeps #410 free to merge on
dev while the retire stubs win later.

- pagination: nextFlag -> nextPageFlag; count/field are mutually required,
  not both; stray ']' after the EX139 link
- aws-signature: 'signaturing requests by AWS' -> proper Signature V4 wording
- config: un-garble the userData column-collision sentence
- ssh-proxy: 'tu' -> 'to'

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

@davidesner davidesner left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM I just found some relicts from the original version, that can be fixed straight away

Comment thread _data/navigation.yml Outdated
Comment thread src/content/docs/components/extractors/generic-extractor/configuration/index.md Outdated
Comment thread src/content/docs/components/extractors/generic-extractor/running/index.md Outdated
- nav: 'Generic Extractor Tutorial' -> 'Tutorial' (redundant under the
  Generic Extractor node, and it clipped in the sidebar)
- configuration: drop the Public Beta caution on the UI section — the UI
  is not a beta anymore
- running: drop the old-UI configuration screenshot (and its now-orphaned
  image file)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@Iamfle4ka
Iamfle4ka requested a review from davidesner August 25, 2026 15:33
@Iamfle4ka

Copy link
Copy Markdown
Collaborator Author

Thanks @davidesner — all three fixed in 7fd8382: nav title shortened to "Tutorial", the Public Beta caution dropped (not a beta anymore), and the old-UI screenshot removed from Running (the orphaned image file deleted with it). Re-requested your review.

🤖 Generated with Claude Code

@davidesner davidesner left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@Iamfle4ka
Iamfle4ka merged commit 6dc3896 into main Aug 26, 2026
3 checks passed
@Iamfle4ka
Iamfle4ka deleted the PRDCT-552-generic-extractor-recut branch August 26, 2026 16:53
Iamfle4ka pushed a commit that referenced this pull request Sep 2, 2026
…added

The generic-extractor recut (#1054) landed without descriptions; the
generator fills them from each page's lead prose. The retired
extractors/other/generic page follows main's deletion.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

This branch was successfully deployed

1 active deployment
Preview — d5d78898 Deployed Aug 26, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dev-docs-migration developers.keboola.com → help.keboola.com migration

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants