[travon-speller]

Description

Adds Java-executed Clean data (clean.process) and Transform data (data.transform) nodes to Automation. They operate on live upstream tables through existing SEMOSS BI reactors, rather than adding another execution runtime or modifying shared frame implementations.

Companion UI: SEMOSS/semoss-ui#3774. Deploy the paired changes together and rebuild/restart the backend so the new node catalog is available.

Changes Made

• Register both node types, catalog defaults, typed operation values, and authoring/execution validation.
• Clean operations: purge matching rows, trim, replace, remove duplicates, filter empty values, and impute nulls.
• Transform operations: change type; drop, rename, add, duplicate, and concatenate columns; pivot/unpivot; lower/proper/upper case; split/split-and-unpivot; timestamp; percent; transpose; and standard joins.
• Route both node types through the existing Java graph runner in root graphs and loop bodies.
• Use an isolated same-backend working frame, preserve duplicate rows and visible filters, invoke standard reactors through NounStore, synchronize Python bindings/headers, and publish through existing frame registration and retained table capture.
• Add catalog, validator, operation-dispatch, frame-copy/duplicate, schema/binding, Purge, and single-frame failure/limit unit coverage.

The current diff contains 10 Automation production/test files. It does not modify shared frame classes, pandas helpers, importers, wrappers, scheduler persistence, or database schemas.

Execution Flow

flowchart TD
    A[Existing authorized Automation run] --> B[Validate graph and node configuration]
    B --> C[Java graph runner: root or loop body]
    C --> D[Resolve live upstream frame from run scope]
    D --> E[Validate operation, columns, backend and row bounds]
    E --> F[Create isolated working frame; preserve duplicates and filters]
    F --> G{Operation}
    G -->|Clean or single-table transform| H[Existing SEMOSS BI reactor]
    G -->|Join| I[Existing Merge reactor with right-side frame query]
    H --> J[Validate output; synchronize Python bindings and headers]
    I --> J
    J --> K[Register live output and capture retained table]
    K --> L[Existing downstream scope, run events and history]

Loading

Permissions and Runtime Boundaries

• Existing Automation project/run authorization remains in force; these nodes consume already-authorized live frames, not arbitrary engine IDs or historical JSON summaries.
• Most operations require Python frames. R frames are rejected. Native database-backed frames have limited support through Purge and standard same-database joins; there is no silent backend conversion or Polars implementation.
• Queries produced by the current Automation database node materialize Python frames. Compatibility is determined by the actual frame type, not the upstream node label.
• Purge removes matching rows from the working dataset, not records from the source database. Input and output size remain bounded by existing saved-table limits.
• Existing BI semantics/restrictions apply. This PR does not fix shared null-import, empty-frame, or other platform behavior.
• Errors propagate to the existing contextual run logger; cleanup warnings retain exception detail without logging table payloads.

Validation Status — 2026-10-28

Reviewed head: e6c264043af13443f9e341229e00a1bae553264e.

• Current GitHub build_dev and Snyk checks: passing.
• Diff whitespace check: passing.
• Backend CI skips test compilation/execution (maven.test.skip=true); a successful build is not a Java test pass.
• Current backend JUnit and authenticated live regression remain unverified in this review. No local Maven command was run; backend build/test validation is to use Eclipse/JUnit.
• Companion UI: 109 existing tests, changed-file Biome, and an isolated production build passed. Its full type-check has unrelated/base and local-untracked-file failures.

Open Review Findings Before Merge

• Align runtime delimiter validation with the catalog/authoring contract: the default single-space delimiter is currently rejected by isBlank() in safeScriptLiteral, breaking default concatenate and space-split configurations.
• Correct Join failure cleanup when Merge returns the same pandas frame and later validation/synchronization/registration throws; add same-object and shared-binding regressions.
• Complete current Eclipse/JUnit and signed-in live tests, including actual BI operation execution, downstream data, root/loop paths, retained history, failure cleanup, and permission denial cases.

These findings are not fixed by the description update. This review supports the existing architecture but does not establish complete end-to-end correctness.

How to Test

1. Build/restart this backend and the companion UI. Use an authorized test database with duplicate rows, null/empty values, date columns, and two joinable tables.
2. Save/reopen Query -> Clean -> Transform -> downstream consumer; inspect both original and transformed outputs. Verify original data is unchanged and new columns, dtypes, row values, and duplicates remain consistent.
3. Exercise every operation, including Purge AND conditions, multiple Pivot maintained columns, space delimiters, split/unpivot, and inner/left/right joins. Check unsupported frame/type/literal combinations fail clearly.
4. Repeat representative paths in loops. Inspect retained output pages and reopened history; check empty results and row-limit failures without stale bindings.
5. Run AutomationCleanExecutorUnitTests, AutomationDefinitionValidatorUnitTests, AutomationNodeCatalogUnitTests, and affected execution/frame/capture tests through Eclipse/JUnit; add and run the regression tests above.

[snyk-io]

✅ Snyk checks have passed. No issues have been found so far.

Status	Scan Engine	Critical	High	Medium	Low	Total (0)
✅	Open Source Security	0	0	0	0	0 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.