API: Add TaskPipeline for multi-stage orchestration

[gjkim42]

What type of PR is this?

/kind api

What this PR does / why we need it:

Adds the TaskPipeline v1alpha2 API and controller for declarative multi-stage Task orchestration.

• Models pipelines as named DAG nodes with explicit dependencies
• Supports static matrix fan-out with deterministic child Task names
• Makes dependency outputs and results available to downstream Go templates
• Aggregates child Task state and results into pipeline status
• Installs the CRD, generated clients, RBAC, console relationships, documentation, and an end-to-end example

The initial API deliberately stays focused on the core orchestration contract. Retry policy, suspend/resume, conditional edges, per-node timeouts, and TaskSpawner integration remain follow-up capabilities.

Which issue(s) this PR is related to:

Fixes #983

Special notes for your reviewer:

Validation performed:

• make update
• make verify
• env -u CODEX_HOME -u CODEX_API_KEY -u CODEX_AUTH_JSON make test
• make test-integration TEST_FLAGS='-run=^TestIntegration$' (155/155 passed)
• Install integration suite (12/12 passed)
• make build WHAT=cmd/kelos-controller

Does this PR introduce a user-facing change?

Add the v1alpha2 TaskPipeline API for declarative DAG-based Task orchestration with matrix fan-out and dependency result aggregation.

[cubic-dev-ai]

2 issues found across 31 files

Not reviewed (too large): internal/manifests/charts/kelos/charts/kelos-crds/templates/taskpipeline-crd.yaml (~7,430 lines), internal/manifests/install-crd.yaml (~7,428 lines) - if these are generated or fixture files, add them to ignored paths to exclude them from future reviews.

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="internal/controller/taskpipeline_controller.go">

<violation number="1" location="internal/controller/taskpipeline_controller.go:458">
P2: When a WorkerPool-backed dependency completes before its pod output is readable, this retry window expires and the dependent node fails permanently. Retry WorkerPool output capture or keep this dependency retryable until results are available.</violation>
</file>

<file name="examples/07-task-pipeline/pipeline.yaml">

<violation number="1" location="examples/07-task-pipeline/pipeline.yaml:41">
P2: The write-tests and open-pr prompts read `{{index .Tasks "..." 0 "Results" "branch"}}`, but Results is only populated from agent output lines of the form `key: value`, and the scaffold/write-tests prompts never tell the agent to emit a `branch:` result. Since missing template keys fail the node and pipeline, this example depends on the upstream agent incidentally producing that key. Because each node already hardcodes `branch: feature/auth`, either drop the results reference or instruct the upstream agent to emit the structured result.</violation>
</file>

<sub>Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.

Re-trigger cubic</sub>

[cubic-dev-ai]

P2: When a WorkerPool-backed dependency completes before its pod output is readable, this retry window expires and the dependent node fails permanently. Retry WorkerPool output capture or keep this dependency retryable until results are available.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At internal/controller/taskpipeline_controller.go, line 458:

<comment>When a WorkerPool-backed dependency completes before its pod output is readable, this retry window expires and the dependent node fails permanently. Retry WorkerPool output capture or keep this dependency retryable until results are available.</comment>

<file context>
@@ -0,0 +1,780 @@
+	var retryAfter time.Duration
+	for _, dependency := range node.DependsOn {
+		for _, task := range tasksByNode[dependency] {
+			if task.Status.Phase != kelos.TaskPhaseSucceeded || task.Status.CompletionTime == nil {
+				continue
+			}
</file context>

[cubic-dev-ai]

P2: The write-tests and open-pr prompts read {{index .Tasks "..." 0 "Results" "branch"}}, but Results is only populated from agent output lines of the form key: value, and the scaffold/write-tests prompts never tell the agent to emit a branch: result. Since missing template keys fail the node and pipeline, this example depends on the upstream agent incidentally producing that key. Because each node already hardcodes branch: feature/auth, either drop the results reference or instruct the upstream agent to emit the structured result.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At examples/07-task-pipeline/pipeline.yaml, line 41:

<comment>The write-tests and open-pr prompts read `{{index .Tasks "..." 0 "Results" "branch"}}`, but Results is only populated from agent output lines of the form `key: value`, and the scaffold/write-tests prompts never tell the agent to emit a `branch:` result. Since missing template keys fail the node and pipeline, this example depends on the upstream agent incidentally producing that key. Because each node already hardcodes `branch: feature/auth`, either drop the results reference or instruct the upstream agent to emit the structured result.</comment>

<file context>
@@ -1,74 +1,71 @@
+            name: my-workspace
+        branch: feature/auth
+        prompt: |
+          The scaffold task created an auth module on branch {{index .Tasks "scaffold" 0 "Results" "branch"}}.
 
-    Run the tests to make sure they pass. Commit and push.
</file context>

[bio1-aws]

Great work on this PR — the TaskPipeline API fills a real gap in the orchestration story, and the scope discipline (leaving retry, suspend, conditionals, and TaskSpawner for follow-ups) is exactly the right call for a v1alpha2. The DAG model with explicit dependencies, matrix fan-out, and template-accessible task results is a clean foundation that maps well to how teams actually structure multi-stage work.

I want to flag three design areas that I think deserve a closer look before this hardens into a more stable API version.

First, deterministic child Task naming and idempotency. I don't see an explicit naming scheme documented in the types or the PR description, and this is the kind of thing that bites you hard in production. If a matrix parameter value changes, or a node is renamed, do we orphan old Tasks or try to adopt them? What happens on a controller restart mid-reconcile — can we safely re-create a Task we think we never created, without risking a duplicate? I'd strongly recommend a content-hash based suffix (e.g., <pipeline>-<node>-<hash>) derived from the node spec + matrix combination, so that reconciliation is idempotent and spec changes produce predictable new Task names. This also makes garbage collection and debugging much easier than relying on owner references alone.

Second, aggregation policy extensibility. Right now the phase is a simple binary: all nodes succeed → Succeeded, any failure → Failed. That's correct for the common case, but I'd like us to think about whether the status structure leaves room for more sophisticated policies later (any-of, majority, weighted, custom selectors). Specifically, I'm wondering if TaskPipelinePhase should remain a flat enum, or if we should introduce an explicit aggregation policy field now — even if it only supports All in v1alpha2 — so that the status contract doesn't need a breaking change when we add Any or Majority. The PR body mentions conditional edges as follow-up work, which is fine, but the aggregation model is closer to the core status contract and is harder to retrofit.

Third, nested pipeline resource boundaries. The design notes mention bidirectional nesting as a future capability, and I want to make sure we're not painting ourselves into a corner. If a pipeline can be a node in another pipeline, do child pipelines inherit parent-level timeouts, retry budgets, and resource quotas, or are they fully independent? The answer has implications for how we structure the spec (top-level policy fields vs. per-node overrides) and how status propagates upward. Even if we defer the feature, I'd like to see a brief note in the API design docs about the intended inheritance model so we don't accidentally close off that path.

Overall this is a really solid foundation — the validation rules on PipelineMatrix and PipelineTaskTemplate are thorough, the template context design is intuitive, and the scope is appropriately bounded. Happy to see this land; the naming/idempotency piece is the one I'd prioritize addressing before merge.