open-metadata
diff --git a/‎.claude/skills/connector-audit‎
Lines changed: 0 additions & 1 deletion b/‎.claude/skills/connector-audit‎
Lines changed: 0 additions & 1 deletion
diff --git a/‎CLAUDE.md‎
Lines changed: 16 additions & 0 deletions b/‎CLAUDE.md‎
Lines changed: 16 additions & 0 deletions
diff --git a/‎adr-incident-manager-governance-workflows.md‎
Lines changed: 305 additions & 0 deletions b/‎adr-incident-manager-governance-workflows.md‎
Lines changed: 305 additions & 0 deletions
@@ -139,6 +139,22 @@ make unit_ingestion            # Python unit tests with coverage
 yarn test:coverage             # Frontend test coverage
 ```
 
+### Backend Integration Tests
+All backend API integration tests MUST be placed in `openmetadata-integration-tests/src/test/java/org/openmetadata/it/tests/` directory. Tests should:
+- Use naming convention `*IT.java` (Integration Test)
+- Extend `BaseEntityIT<T, K>` for entity CRUD tests
+- Be designed to run concurrently (use `@Execution(ExecutionMode.CONCURRENT)`)
+- Use `TestNamespace` for test isolation
+- Use `SdkClients` for API calls (e.g., `SdkClients.adminClient().tables().create(...)`)
+
+```bash
+# Run a specific integration test
+mvn test -pl openmetadata-integration-tests -Dtest=TaskResourceIT
+
+# Run all integration tests
+mvn test -pl openmetadata-integration-tests
+```
+
 ## Code Generation and Schemas
 
 OpenMetadata uses a schema-first approach with JSON Schema definitions driving code generation:
 
@@ -0,0 +1,305 @@
+# Integrate Incident Manager in the Governance Workflows Framework
+
+ADR-#: 1
+Authors: Pablo Takara
+Reviewers: Teddy Crépineau, Ram Narayan Balaji
+Date: February 27, 2026
+Status: Proposed
+
+> Migrate incident lifecycle into a governance workflow using a new Task Lifecycle Node. The node uses OpenMetadata tasks as the source of truth (not Flowable UserTask), receives a template with configurable statuses, and exposes each status transition to the main workflow graph via process variables. Users wire hooks on any transition using standard edges. Non-terminal statuses loop back; terminal statuses auto-close the task.
+
+---
+
+## Context
+
+The Incident Manager handles the lifecycle of data quality incidents in OpenMetadata. When a test case fails, an incident is created; it progresses through `New → Ack → Assigned → Resolved` as humans triage it.
+
+Today, this lifecycle is a **switch statement** in `TestCaseResolutionStatusRepository.storeInternal()`. It handles state transitions, task creation, assignment, and resolution. The state machine is simple, correct, and performant, but it has **no extension points**. Adding a behavior like "on Assigned, notify via Slack" or "on New, auto-assign to table owner" requires modifying repository code, testing, and redeploying.
+
+Meanwhile, OpenMetadata ships a **governance workflows framework** built on Flowable BPM. It is fully configurable via REST API and UI. Users configure workflows as abstract **trigger → nodes → edges** graphs (they never see BPMN XML). The backend compiles these to Flowable process definitions automatically via `NodeFactory` and `MainWorkflow`.
+
+The two systems live side by side but do not interact.
+
+Additionally, the **task refactor** promotes tasks to first-class entities with standard `ChangeEvents`. This enables Flowable to be notified of every status transition — not just resolution — unlocking configurable hooks on any transition from day one.
+
+### Specific Gaps
+
+1. **No auto-close when tests pass.** `TestCaseResultRepository.setTestCaseResultIncidentId()` sets `incidentId = null` when a test succeeds but **never resolves the incident or closes its task**.
+2. **No auto-assign on incident creation.** Every incident starts in `New` and requires manual acknowledgement.
+3. **No extensibility.** Organizations cannot define configurable rules like "on any status change, execute action X" without code changes.
+4. **Fixed lifecycle.** The `New → Ack → Assigned → Resolved` states are hardcoded. Organizations with different triage processes have no way to customize.
+5. **No incident TTL.** No mechanism to auto-close stale incidents.
+
+### Enterprise scale context
+
+- 5M assets, 10-30% with data quality tests = 500K-1.5M test cases
+- At 2-5% failure rate = **10K-75K concurrent open incidents** (typical)
+- `getOrCreateIncident()` enforces one unresolved incident per test case
+
+---
+
+## Use Cases
+
+**UC-1 — Auto-close incident when test passes**
+The system automatically resolves the open incident (reason: AutoResolved) and closes its task. No human intervention required.
+
+**UC-2 — Auto-assign incident on creation**
+When a new incident is created, the system automatically assigns it to a configured user or team.
+
+**UC-3 — Auto-close stale incidents (TTL)**
+An incident open longer than a configurable deadline is automatically resolved (reason: Expired).
+
+**UC-4 — User-defined hooks on any status transition**
+Users wire follow-up steps (notifications, Jira tickets, etc.) on any status change via workflow edges — no code changes.
+
+---
+
+## Decision
+
+### Task Lifecycle Node
+
+A new governance workflow node that does NOT use Flowable's BPMN UserTask. It creates an OpenMetadata task, waits for status changes via `IntermediateCatchEvent`, and exposes each status to the parent workflow for routing.
+
+**Internal BPMN structure:**
+```
+┌─ SubProcess ──────────────────────────────────────────────────────┐
+│                                                                    │
+│  [Start] → [Setup] → [Gateway: created?]                          │
+│               │          no → [End: skip]                          │
+│               │          yes ↓                                     │
+│               │        [IntermediateCatchEvent: wait]               │
+│               │          ↓ message with {status}                   │
+│               │        [Gateway: terminal?]                        │
+│               │          yes → [CloseTask] → [SetResult] → [End]  │
+│               │          no  → [SetResult] → [End]                 │
+│               │                                                    │
+│               │  Setup (idempotent):                               │
+│               │    • Check for existing open incident              │
+│               │      → if exists with active process: skip         │
+│               │      → if orphaned process: terminate it           │
+│               │    • Create incident record (New)                  │
+│               │    • Create OM task                                │
+│               │    • Auto-assign (from template config)            │
+│               │    • Set process variable omTaskId = task UUID     │
+│                                                                    │
+│  + [TTL Boundary Timer: configurable, interrupting]                │
+│      → [AutoResolve via repository] → [End]                       │
+└────────────────────────────────────────────────────────────────────┘
+```
+
+**Node config:**
+```json
+{
+  "type": "taskLifecycleNode",
+  "config": {
+    "template": "incident",
+    "statuses": ["New", "Ack", "Assigned", "Resolved"],
+    "terminal": ["Resolved"],
+    "responsibles": { "source": "tableOwner" },
+    "ttl": "P30D"
+  }
+}
+```
+
+The node:
+1. **Setup** — Creates the OM task (idempotent on re-entry). Sets `omTaskId` process variable.
+2. **Wait** — `IntermediateCatchEvent` with `messageExpression="${omTaskId}"`. Subscribes to a message named after the task UUID (~2 Flowable DB rows).
+3. **On message** — Evaluates whether the received status is terminal.
+4. **Terminal** — Closes the OM task (idempotent), sets `{nodeName}_result` at parent scope, subprocess exits.
+5. **Non-terminal** — Sets `{nodeName}_result` at parent scope, subprocess exits. Parent-level edges route back to the node.
+
+### Status exposed via graph edges (with cycles)
+
+Status is set as a Flowable process variable when the subprocess exits. Parent-level edges condition on this variable. Non-terminal edges loop back to the node.
+
+```
+         ┌────── "ack" ───────────────────────────┐
+         │  ┌─── "assigned" → [NotifySlack] ──────┤
+         ▼  ▼                                      │
+[Start] → [ManageIncident] ── "resolved" → [End]
+```
+
+**Workflow definition example:**
+```json
+{
+  "name": "incident-lifecycle",
+  "trigger": {
+    "type": "eventBasedEntity",
+    "config": {
+      "entityTypes": ["TestCase"],
+      "events": ["Updated"],
+      "filter": { "TestCase": { "==": [{"var": "testCaseStatus"}, "Failed"] } }
+    }
+  },
+  "nodes": [
+    { "type": "startEvent", "name": "start" },
+    { "type": "taskLifecycleNode", "name": "incident", "config": {
+        "template": "incident",
+        "statuses": ["New", "Ack", "Assigned", "Resolved"],
+        "terminal": ["Resolved"],
+        "responsibles": { "source": "tableOwner" },
+        "ttl": "P30D"
+    }},
+    { "type": "automatedTask", "subType": "sinkTask", "name": "notifySlack" },
+    { "type": "endEvent", "name": "end" }
+  ],
+  "edges": [
+    { "from": "start", "to": "incident" },
+    { "from": "incident", "to": "incident", "condition": { "status": "Ack" } },
+    { "from": "incident", "to": "notifySlack", "condition": { "status": "Assigned" } },
+    { "from": "notifySlack", "to": "incident" },
+    { "from": "incident", "to": "end", "condition": { "status": "Resolved" } }
+  ]
+}
+```
+
+### Message delivery via task ChangeEvents
+
+With the task refactor, tasks emit `ChangeEvents` on status changes. These drive message delivery to Flowable:
+
+1. Task status changes (via REST API / `storeInternal`)
+2. `ChangeEvent` emitted
+3. Listener correlates message to waiting `IntermediateCatchEvent`
+
+The OM task is already updated before the message fires. If correlation fails, the task state is correct — Flowable catches up on the next status change.
+
+**Mechanism TBD**: Listener on task `ChangeEvents` (clean separation) vs direct hook in task status update code (fewer hops).
+
+### What the workflow controls vs the repository
+
+| Action | Who handles it |
+| --- | --- |
+| Task creation | Node setup phase (idempotent) |
+| Status changes (Ack, Assigned, etc.) | Repository — synchronous, unchanged |
+| Resolution | Repository — synchronous, unchanged |
+| Task closure | Both — node closes on terminal, repository may also close. Idempotent. |
+| Flowable notification | Task ChangeEvent → message to IntermediateCatchEvent |
+| Follow-up hooks | Workflow edges — user-configurable |
+| TTL auto-resolve | Boundary timer on node |
+| Auto-close on test pass | Separate short-lived workflow |
+
+### Why this approach
+
+1. **Hooks on any transition.** Status exposed to parent graph → users wire follow-up steps via edges.
+2. **Configurable lifecycle.** Template defines statuses and terminal set. No hardcoded lifecycle.
+3. **OM task is source of truth.** No BPMN UserTask. ~2 DB rows per task vs ~5-10.
+4. **Repository stays in the critical path.** All transitions are synchronous. Flowable is notified after the fact. If Flowable is down, transitions still succeed.
+5. **Unified abstraction.** Same node type for incidents, approvals, certifications — different templates.
+
+---
+
+## Consequences
+
+### Positive
+
+- **Hooks on any status transition** without code changes.
+- **Configurable lifecycle from day one** via template config.
+- **Lightweight** — ~2 Flowable DB rows per task (IntermediateCatchEvent).
+- **Safe** — repository owns all transitions synchronously; Flowable is follow-up only.
+- **Default workflow replicates current behavior** and ships enabled.
+- **Unified abstraction** — incidents, approvals, certifications share one node type.
+
+### Negative
+
+- **MainWorkflow compiler must support cycles.** Today it assumes a DAG. Biggest technical risk.
+- **More Flowable interactions.** Every status change sends a message (vs resolution only). ~225K correlations over lifetime of 75K incidents with ~3 transitions each.
+- **Task refactor dependency.** Fallback: direct `reportOutcome()` from `storeInternal()` if not ready.
+
+### Neutral
+
+- REST API surface unchanged.
+- `TestCaseResolutionStatus` schema changes minimally (add `AutoResolved`, `Expired` reasons).
+- Resolution business logic in the repository is unchanged.
+
+---
+
+## Alternatives Considered
+
+### Bookends only (no intermediate state hooks)
+
+Handle only creation + resolution in the workflow. Intermediate states stay entirely in `storeInternal()`.
+
+**Not chosen:** Users cannot wire hooks on Ack/Assigned. The task refactor makes full lifecycle hooks possible now — deferring them means two migrations.
+
+### Internal loop (cycle hidden inside SubProcess)
+
+The message loop lives inside the node. Status exposed only on terminal exit. Outer graph stays a DAG.
+
+**Not chosen:** Users cannot wire hooks on non-terminal transitions. The point is exposing every status change to the parent graph.
+
+### Resolution through Flowable (not fire-and-forget)
+
+Route resolution through the Flowable process.
+
+**Not chosen:** Puts Flowable in the critical path. If Flowable is slow/down, resolution is blocked.
+
+### Extend state machine with Java hooks
+
+**Rejected:** Parallel automation system, requires code changes for every new behavior.
+
+### CMMN (Case Management)
+
+**Rejected:** Zero existing infrastructure, overkill.
+
+---
+
+## Design Choices
+
+### IntermediateCatchEvent with messageExpression
+
+`messageExpression="${omTaskId}"` gives unique-per-instance subscriptions. `EventSubscriptionQuery.eventName(taskId)` is an indexed lookup. No MessageCorrelationBuilder (doesn't exist in Flowable 7.2.0).
+
+### Idempotent setup on loop re-entry
+
+When non-terminal edges loop back, Setup detects the existing task and reuses it. Safe for any number of loops.
+
+### Terminal auto-close — both sides
+
+`storeInternal(Resolved)` closes the task. The node's `CloseTask` also closes on terminal status. Both are idempotent. This handles TTL (node-initiated) and human resolution (repository-initiated) uniformly.
+
+### Business key = test case FQN
+
+Enables idempotent creation, fire-and-forget termination, auto-close correlation.
+
+### Governance-bot loop prevention
+
+`WorkflowEventConsumer` skips events from `governance-bot`. The workflow runs as `governance-bot`, so its own events don't re-trigger workflows.
+
+---
+
+## Open Questions
+
+- [ ] **Message delivery mechanism**: Listener on task ChangeEvents vs direct hook in task status update.
+- [ ] **TestCaseResult.incidentId linking**: If creation moves to async workflow, test result may store before incident exists. Recommendation: keep `getOrCreateIncident()` synchronous.
+- [ ] **Cycle validation**: Should the compiler enforce that every non-terminal edge path routes back to a task node?
+
+---
+
+## Risks
+
+| Risk | Impact | Mitigation |
+| --- | --- | --- |
+| Cycle support in MainWorkflow | Blocks the design | Spike early. Workaround: invisible gateway node. |
+| Task refactor not ready | No ChangeEvents for message delivery | Fall back to direct reportOutcome() from storeInternal() |
+| Race condition | Message lost during follow-up execution | EventSubscriptionQuery returns null → skipped. Java-side buffer later. |
+| ACT_RU growth | ~2 rows per open incident | 75K incidents = 150K rows. Measure in hardening phase. |
+| Process orphaning | Never-resolved incidents linger | TTL handles deadlines. Batch sweep for the rest. |
+
+---
+
+## Follow-up Work
+
+1. **Batch sweep** for orphaned processes.
+2. **Migrate UserApprovalTask** (glossary) to same node type with `template: "approval"`.
+3. **SLA timer escalation** — optional boundary timer using same infrastructure as TTL.
+
+---
+
+## References
+
+- `TestCaseResolutionStatusRepository.storeInternal()` — Current state machine
+- `WorkflowHandler.java` — Flowable ProcessEngine, message delivery
+- `MainWorkflow.java` — BPMN compiler (needs cycle support)
+- `UserApprovalTask.java` — Current UserTask pattern (being replaced)
+- `NodeFactory.java` — Node type registration
+- `WorkflowEventConsumer.java` — Event routing, governance-bot loop prevention