Feat(Workflows): Auto-inject inclusive gateways for batch routing#26918
Feat(Workflows): Auto-inject inclusive gateways for batch routing#26918yan-3005 wants to merge 32 commits intoram/workflow-improvementsfrom
Conversation
Check nodes (CheckEntityAttributes, CheckChangeDescription, DataCompleteness) now emit boolean flag variables (hasTrueEntities, hasFalseEntities, has_<band>_entities) instead of a single RESULT_VARIABLE. MainWorkflow auto-detects split/join points at BPMN build time via NodeInterface.getOutputPorts() and injects InclusiveGateways so both true and false branches fire simultaneously when a batch has mixed pass/fail entities. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
yan-3005
added
safe to test
yan-3005
changed the title
![gitar-bot[bot]](https://avatars.githubusercontent.com/in/827041?s=60&v=4){kind=small}
There was a problem hiding this comment.
Pull request overview
This PR updates OpenMetadata governance workflows to support batch entity processing via a new entityList variable, adds inclusive gateway injection for batch “check” nodes (true/false and quality-band branching), and introduces a v1140 migration to update persisted workflow JSON accordingly.
Changes:
- Replace/augment legacy
relatedEntityusage withentityListacross workflow triggers, node schemas, and runtime variable handling. - Inject inclusive split/join gateways in Flowable BPMN generation for nodes that produce multiple output ports (e.g., check tasks, data completeness bands).
- Add v1140 migration + extensive unit/integration test updates for new variable semantics and batch entity fetching.
Reviewed changes
Copilot reviewed 14 out of 14 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| openmetadata-ui/src/main/resources/ui/src/generated/governance/workflows/elements/nodes/automatedTask/sinkTask.ts | Generated UI type updates: InputNamespaceMap now uses entityList and allows additional properties. |
| openmetadata-ui/src/main/resources/ui/src/generated/governance/workflows/elements/nodes/automatedTask/setEntityAttributeTask.ts | Generated UI type updates to entityList-based input namespace map. |
| openmetadata-ui/src/main/resources/ui/src/generated/governance/workflows/elements/nodes/automatedTask/rollbackEntityTask.ts | Generated UI type updates + updated rollback description. |
| openmetadata-ui/src/main/resources/ui/src/generated/governance/workflows/elements/nodes/automatedTask/dataCompletenessTask.ts | Generated UI type updates to entityList-based input namespace map. |
| openmetadata-ui/src/main/resources/ui/src/generated/governance/workflows/elements/nodes/automatedTask/checkEntityAttributesTask.ts | Generated UI type updates + adds output?: string[] on task interface and entityList in namespace map. |
| openmetadata-ui/src/main/resources/ui/src/generated/governance/workflows/elements/nodes/automatedTask/checkChangeDescriptionTask.ts | Generated UI type updates + adds output?: string[] on task interface and entityList in namespace map. |
| openmetadata-ui/src/main/resources/ui/src/generated/api/governance/createWorkflowDefinition.ts | API types updated to include output?: string[] and allow entityList in trigger namespace mapping. |
| openmetadata-spec/src/main/resources/json/schema/governance/workflows/elements/triggers/periodicBatchEntityTrigger.json | Trigger schema output default now includes entityList; relaxes single-item constraint. |
| openmetadata-spec/src/main/resources/json/schema/governance/workflows/elements/triggers/eventBasedEntityTrigger.json | Trigger schema output default now includes entityList; relaxes single-item constraint. |
| openmetadata-spec/src/main/resources/json/schema/governance/workflows/elements/nodes/automatedTask/sinkTask.json | Node schema migrated to require entityList and allow additional properties for conditional keys. |
| openmetadata-spec/src/main/resources/json/schema/governance/workflows/elements/nodes/automatedTask/setEntityAttributeTask.json | Node schema migrated to require entityList and allow additional properties; input defaults/minItems adjusted. |
| openmetadata-spec/src/main/resources/json/schema/governance/workflows/elements/nodes/automatedTask/rollbackEntityTask.json | Node schema migrated to require entityList; description updated to fallback chain. |
| openmetadata-spec/src/main/resources/json/schema/governance/workflows/elements/nodes/automatedTask/dataCompletenessTask.json | Node schema migrated to require entityList; loosens input constraints. |
| openmetadata-spec/src/main/resources/json/schema/governance/workflows/elements/nodes/automatedTask/checkEntityAttributesTask.json | Node schema switches to entityList + adds explicit output defaults for derived entity lists. |
| openmetadata-spec/src/main/resources/json/schema/governance/workflows/elements/nodes/automatedTask/checkChangeDescriptionTask.json | Node schema switches to entityList + adds explicit output defaults for derived entity lists. |
| openmetadata-service/src/test/java/org/openmetadata/service/migration/utils/v1140/MigrationUtilTest.java | New unit tests for v1140 workflow JSON migration behavior. |
| openmetadata-service/src/test/java/org/openmetadata/service/migration/utils/v1105/MigrationUtilTest.java | Updates test fixtures to entityList in workflow JSON. |
| openmetadata-service/src/test/java/org/openmetadata/service/governance/workflows/flowable/MainWorkflowInclusiveGatewayTest.java | New tests verifying inclusive split/join gateway injection logic. |
| openmetadata-service/src/test/java/org/openmetadata/service/governance/workflows/flowable/builders/InclusiveGatewayBuilderTest.java | New unit tests for InclusiveGatewayBuilder. |
| openmetadata-service/src/test/java/org/openmetadata/service/governance/workflows/elements/nodes/automatedTask/sink/SinkTaskTest.java | Updates sink workflow JSON fixtures to entityList. |
| openmetadata-service/src/test/java/org/openmetadata/service/governance/workflows/elements/nodes/automatedTask/sink/SinkTaskDelegateTest.java | Updates delegate tests to batch entity fetching and entityList semantics. |
| openmetadata-service/src/test/java/org/openmetadata/service/governance/workflows/elements/nodes/automatedTask/SetEntityAttributeTaskTest.java | New unit tests for BPMN construction of SetEntityAttribute task with entityList. |
| openmetadata-service/src/test/java/org/openmetadata/service/governance/workflows/elements/nodes/automatedTask/RollbackEntityTaskTest.java | New unit tests for BPMN construction of RollbackEntity task with entityList. |
| openmetadata-service/src/test/java/org/openmetadata/service/governance/workflows/elements/nodes/automatedTask/impl/SetEntityAttributeImplTest.java | New unit tests for batch SetEntityAttributeImpl execution. |
| openmetadata-service/src/test/java/org/openmetadata/service/governance/workflows/elements/nodes/automatedTask/impl/CheckEntityAttributesImplTest.java | New unit tests for batch CheckEntityAttributesImpl execution and flags. |
| openmetadata-service/src/test/java/org/openmetadata/service/governance/workflows/elements/nodes/automatedTask/impl/CheckChangeDescriptionTaskImplTest.java | New unit tests for batch CheckChangeDescriptionTaskImpl execution and flags. |
| openmetadata-service/src/test/java/org/openmetadata/service/governance/workflows/elements/nodes/automatedTask/DataCompletenessTaskTest.java | New unit tests for BPMN construction of DataCompleteness task with entityList. |
| openmetadata-service/src/test/java/org/openmetadata/service/governance/workflows/elements/nodes/automatedTask/CheckEntityAttributesTaskTest.java | New unit tests for BPMN construction of CheckEntityAttributes task with entityList. |
| openmetadata-service/src/test/java/org/openmetadata/service/governance/workflows/elements/nodes/automatedTask/CheckChangeDescriptionTaskTest.java | New unit tests for BPMN construction of CheckChangeDescription task with entityList. |
| openmetadata-service/src/main/resources/json/data/governance/workflows/GlossaryApprovalWorkflow.json | Updates default workflow to emit/use entityList and conditional *_entityList wiring. |
| openmetadata-service/src/main/java/org/openmetadata/service/migration/utils/v1140/MigrationUtil.java | New v1140 migration utility to rewrite stored workflow JSON to entityList + conditional wiring. |
| openmetadata-service/src/main/java/org/openmetadata/service/migration/utils/v1105/MigrationUtil.java | Adds migration logic to rewrite glossary status node namespace map to entityList. |
| openmetadata-service/src/main/java/org/openmetadata/service/migration/postgres/v1140/Migration.java | New Postgres migration entrypoint invoking v1140 workflow migration. |
| openmetadata-service/src/main/java/org/openmetadata/service/migration/mysql/v1140/Migration.java | New MySQL migration entrypoint invoking v1140 workflow migration. |
| openmetadata-service/src/main/java/org/openmetadata/service/governance/workflows/WorkflowVariableHandler.java | Adds getEntityList() helper resolving entity lists from namespace maps (plain and conditional). |
| openmetadata-service/src/main/java/org/openmetadata/service/governance/workflows/WorkflowEventConsumer.java | Ensures event-based trigger variables include both relatedEntity and entityList. |
| openmetadata-service/src/main/java/org/openmetadata/service/governance/workflows/Workflow.java | Adds shared workflow constants for true/false lists and retry configuration. |
| openmetadata-service/src/main/java/org/openmetadata/service/governance/workflows/flowable/MainWorkflow.java | Injects inclusive split/join gateways based on node output ports + conditional edges. |
| openmetadata-service/src/main/java/org/openmetadata/service/governance/workflows/flowable/builders/InclusiveGatewayBuilder.java | New builder for Flowable InclusiveGateway. |
| openmetadata-service/src/main/java/org/openmetadata/service/governance/workflows/elements/triggers/PeriodicBatchEntityTrigger.java | Adjusts periodic trigger call-activity wiring to pass entityList for both single and parallel modes. |
| openmetadata-service/src/main/java/org/openmetadata/service/governance/workflows/elements/triggers/impl/FilterEntityImpl.java | Sets global entityList for event-based flows after filtering. |
| openmetadata-service/src/main/java/org/openmetadata/service/governance/workflows/elements/triggers/impl/FetchEntitiesImpl.java | Sets entityList, numberOfEntities, and entityToListMap for periodic triggers. |
| openmetadata-service/src/main/java/org/openmetadata/service/governance/workflows/elements/triggers/EventBasedEntityTrigger.java | Always passes entityList into the main workflow; avoids duplicating trigger outputs. |
| openmetadata-service/src/main/java/org/openmetadata/service/governance/workflows/elements/TriggerFactory.java | Refines batch-mode detection documentation and logic for trigger execution mode. |
| openmetadata-service/src/main/java/org/openmetadata/service/governance/workflows/elements/nodes/automatedTask/SinkTask.java | Ensures sink task namespace map handling uses defined map and always includes entityList. |
| openmetadata-service/src/main/java/org/openmetadata/service/governance/workflows/elements/nodes/automatedTask/sink/SinkTaskDelegate.java | Switches sink execution to entityList and introduces batched entity fetch + retries. |
| openmetadata-service/src/main/java/org/openmetadata/service/governance/workflows/elements/nodes/automatedTask/SetEntityAttributeTask.java | Ensures BPMN wiring includes default entityList namespace. |
| openmetadata-service/src/main/java/org/openmetadata/service/governance/workflows/elements/nodes/automatedTask/RollbackEntityTask.java | Ensures BPMN wiring includes default entityList namespace. |
| openmetadata-service/src/main/java/org/openmetadata/service/governance/workflows/elements/nodes/automatedTask/impl/SetEntityAttributeImpl.java | Converts to batch processing over entityList with retry + partial failure recording. |
| openmetadata-service/src/main/java/org/openmetadata/service/governance/workflows/elements/nodes/automatedTask/impl/RollbackEntityImpl.java | Converts to batch rollback over entityList with retry + fallback target version selection. |
| openmetadata-service/src/main/java/org/openmetadata/service/governance/workflows/elements/nodes/automatedTask/impl/DataCompletenessImpl.java | Converts to batch completeness evaluation, per-band entity lists, and gateway-ready flags. |
| openmetadata-service/src/main/java/org/openmetadata/service/governance/workflows/elements/nodes/automatedTask/impl/CheckEntityAttributesImpl.java | Converts to batch rule evaluation producing true/false entity lists and flags. |
| openmetadata-service/src/main/java/org/openmetadata/service/governance/workflows/elements/nodes/automatedTask/impl/CheckChangeDescriptionTaskImpl.java | Converts to batch change-description evaluation producing true/false entity lists and flags. |
| openmetadata-service/src/main/java/org/openmetadata/service/governance/workflows/elements/nodes/automatedTask/DataCompletenessTask.java | Declares output ports (bands) and defaults entityList mapping for gateway injection. |
| openmetadata-service/src/main/java/org/openmetadata/service/governance/workflows/elements/nodes/automatedTask/CheckEntityAttributesTask.java | Declares output ports (true/false) and defaults entityList mapping for gateway injection. |
| openmetadata-service/src/main/java/org/openmetadata/service/governance/workflows/elements/nodes/automatedTask/CheckChangeDescriptionTask.java | Declares output ports (true/false) and defaults entityList mapping for gateway injection. |
| openmetadata-service/src/main/java/org/openmetadata/service/governance/workflows/elements/NodeInterface.java | Adds getOutputPorts() hook used for inclusive gateway detection. |
| openmetadata-service/src/main/java/org/openmetadata/service/Entity.java | Adds getEntitiesByLinks() helper for efficient batch entity fetch by entity link strings. |
| openmetadata-integration-tests/src/test/java/org/openmetadata/it/tests/WorkflowDefinitionResourceIT.java | Updates integration tests to build/validate workflows using entityList and new outputs. |
| // Workflow: start → check → cert(true) / notify(false) → independent ends | ||
| WorkflowDefinition workflow = | ||
| createWorkflow( | ||
| "SplitOnlyWorkflow", | ||
| List.of( | ||
| startEvent("start"), | ||
| checkAttributesNode("check"), | ||
| endEvent("certEnd"), | ||
| endEvent("notifyEnd")), | ||
| List.of( | ||
| edge("start", "check", null), | ||
| edge("check", "cert", "true"), | ||
| edge("check", "notify", "false"), | ||
| edge("cert", "certEnd", null), | ||
| edge("notify", "notifyEnd", null))); |
There was a problem hiding this comment.
This test workflow defines edges to node ids (e.g. cert, notify) that are not included in the nodes list. That can produce a BPMN model with SequenceFlow targets/sources that don't exist, making the test less representative and potentially hiding issues if workflow validation becomes stricter. Consider adding minimal node definitions for cert/notify (or update edges to point to existing nodes) so the constructed workflow is structurally valid.
…ferences - Check nodes (CheckEntityAttributes, CheckChangeDescription) now set hasFalseEntities=true when entity list is empty, ensuring the split inclusive gateway always has at least one active branch and never stalls. - DataCompletenessImpl activates the lowest-score band flag when no entities are assigned to any band (empty input or all entities failed processing). - MainWorkflowInclusiveGatewayTest: replaced dangling edge targets (cert, notify, goldAction etc.) with actual declared end-event nodes so the constructed BPMN model is structurally valid. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Jest test CoverageUI tests summary
|
🔴 Playwright Results — 4 failure(s), 10 flaky✅ 3665 passed · ❌ 4 failed · 🟡 10 flaky · ⏭️ 102 skipped
Genuine Failures (failed on all attempts)❌
|
- DataCompletenessTask: guard against null getQualityBands() with Optional - DataCompletenessImpl: extract bandFlagVariable() static helper used by both the impl and MainWorkflow to avoid naming convention drift - InclusiveGatewayBuilder: remove misleading exclusive() setter; async job exclusivity is hardcoded true (not a routing concern, matches ExclusiveGatewayBuilder) - MainWorkflow: add @slf4j + warn log when a conditional edge targets a join node (condition is intentionally ignored); use DataCompletenessImpl.bandFlagVariable() - DataCompletenessImplTest: add tests for empty entity list and all-entities-fail scenarios, both verifying lowest-band flag is activated as deadlock fallback Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ram/inclusive-gateway
…ata/OpenMetadata into ram/inclusive-gateway
|
✅ TypeScript Types Auto-UpdatedThe generated TypeScript types have been automatically updated based on JSON schema changes in this PR. |
Resolved conflict in MainWorkflow.java: took base branch's 3-arg NodeFactory.createNode() with workflowDefinition.getFullyQualifiedName(); kept nodeInstanceMap.put() from this branch. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ram/inclusive-gateway
…y PR
- InclusiveGatewayBuilder: remove setExclusive(true) — verified against
Flowable 7.2.0 source that this flag only affects async-job scheduler
exclusivity; InclusiveGatewayActivityBehavior never reads it; routing
semantics come from the InclusiveGateway class type, not this property
- MainWorkflow: wire defaultFlow on each split gateway pointing to the
corresponding join gateway (via new buildSplitToJoinMap helper).
Verified against TakeOutgoingSequenceFlowsOperation source: without a
defaultFlow, Flowable throws FlowableException when all conditions are
false. buildGatewayCondition now emits ${var == true} instead of ${var}
so UelExpressionCondition.evaluate never receives a bare null
- DataCompletenessImpl: remove anyBandActive fallback that forced the
lowest-score band flag to true with an empty entity list. All flags are
already written as !bandEntities.isEmpty() in the per-band loop, so
when all conditions are false the split gateway safely falls through to
its defaultFlow, eliminating spurious audit log entries from stage
listeners firing on zero-entity SetEntityAttribute executions
- Tests updated throughout to reflect new condition format (== true),
3-incoming-flows on join gateway (2 conditional + 1 defaultFlow), and
all-false flags for empty/all-failed data completeness runs
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Switch Include.ALL to Include.NON_DELETED in WorkflowEventConsumer and SetEntityAttributeImpl so deleted entities are never picked up for workflow processing. Add skipDeleted guard in bulkUpdateEntitiesForGovernanceWorkflow so even if a deleted entity reaches the bulk-update path it is skipped rather than restored. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Code Review 👍 Approved with suggestions 5 resolved / 6 findingsAuto-inject inclusive gateways feature adds batch routing logic but requires fixes: retry logic unnecessarily wraps in-memory computation in DataCompletenessImpl, split gateway detection ignores unconditional edges with null conditions (causing incorrect routing), and empty qualityBands fallback is non-functional. Five related issues were addressed but three remain open. 💡 Performance: Retry wraps pure in-memory computation in DataCompletenessImpl📄 openmetadata-service/src/main/java/org/openmetadata/service/governance/workflows/elements/nodes/automatedTask/impl/DataCompletenessImpl.java:63 📄 openmetadata-service/src/main/java/org/openmetadata/service/governance/workflows/elements/nodes/automatedTask/impl/DataCompletenessImpl.java:74-80 In DataCompletenessImpl (lines 74-80), Other task impls (CheckEntityAttributes, SetEntityAttribute) correctly apply retry only around I/O operations like RuleEngine or EntityFieldUtils. ✅ 5 resolved✅ Edge Case: Inclusive gateway deadlocks when entity list is empty
✅ Edge Case: RollbackEntityImpl silently continues on total failure
✅ Edge Case: Null guard on getQualityBands() but not on getConfig()
✅ Edge Case: Split gateway wires unconditional edges with null condition (always-true)
✅ Edge Case: Empty qualityBands bypasses all band flags → gateway deadlock
🤖 Prompt for agentsOptionsDisplay: compact → Showing less information. Comment with these commands to change:
Was this helpful? React with 👍 / 👎 | Gitar |
|
|
Describe your changes:
Phase 2 of the batch workflow improvements. Builds on Phase 1 (PR #26715) which made all task nodes batch-aware with entity lists.
Problem: Check nodes previously emitted a single
RESULT_VARIABLEboolean, which drove exclusive gateway routing — only one branch (true OR false) could fire. In batch mode, a single check node can produce both passing and failing entities simultaneously, so both branches need to fire.Solution: Auto-inject Flowable inclusive gateways at BPMN build time based on graph structure — no changes needed to workflow JSON definitions.
Changes:
InclusiveGatewayBuilder(new) — mirrorsExclusiveGatewayBuilderforInclusiveGatewayelementsNodeInterface.getOutputPorts()(new default method) — nodes declare which output ports they produce. Only nodes overriding this with ≥2 ports get inclusive gateways injected.UserApprovalTaskand other single-result nodes return the default empty set and are unaffected.CheckEntityAttributesTask,CheckChangeDescriptionTask— overridegetOutputPorts()returning{"true", "false"}DataCompletenessTask— overridesgetOutputPorts()returning the configured quality band names (e.g.{"gold", "silver"})RESULT_VARIABLE; now emithasTrueEntities+hasFalseEntitiesboolean flags (check nodes), orhas_<band>_entitiesper band (DataCompleteness)MainWorkflow— detects split/join points at build time and injects inclusive gateways automatically:getOutputPorts().size() ≥ 2and ≥2 conditional outgoing edges; conditions map to${nodeName_hasTrueEntities},${nodeName_hasFalseEntities},${nodeName_has_<band>_entities}Before (exclusive — only one branch fires):
After (inclusive — both branches fire when batch has mixed results):
WorkflowEventConsumer.java,SetEntityAttributeImpl.java,EntityRepository.java— soft-deleted entity guard: entity lookups switched toInclude.NON_DELETED;EntityRepository.bulkUpdateEntitiesForGovernanceWorkflowgained askipDeletedparameter set totruefor all governance callers; prevents governance workflows from inadvertently resurrecting glossary terms (or any entity) deleted while a workflow batch is in flightType of change:
Checklist:
Summary by Gitar
EntityRepositoryandSetEntityAttributeImplto skip soft-deleted entities, preventing unauthorized restoration during workflow execution.WorkflowEventConsumerto exclude deleted entities when triggering workflows, ensuring governance actions only apply to active data.This will update automatically on new commits.