Skip to content

[sdk] Locally executed components and pipelines finish with status success despite obvious errors #13332

@filip-komarzyniec

Description

@filip-komarzyniec

Environment

  • KFP version:
    executed locally

  • KFP SDK version:
    tested on both 2.15.2 and 2.16

  • All dependencies version:

    kfp                                       2.16.0
    kfp-components                            1.11.0      <my_location_of_editable_install>
    kfp-kubernetes                            2.16.0
    kfp-pipeline-spec                         2.16.0
    kfp-server-api                            2.16.0

Steps to reproduce

Minimal reproducible example

from kfp import local
from kfp.dsl import pipeline, component

local.init(runner=local.DockerRunner())

@component(base_image=<my_image_placeholder>, install_kfp_package=False)
def minimal_reproducible_component():
    raise ValueError("This component is failing..")

@pipeline
def minimal_reproducible_pipeline():

    minimal_reproducible_component()

minimal_reproducible_pipeline()

Running the above yields the following results:

18:31:17.314 - INFO - Running pipeline: 'minimal-reproducible-pipeline'
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
18:31:17.315 - INFO - Executing task 'minimal-reproducible-component'
18:31:17.315 - INFO - Streamed logs:

    Found image '<my_image_placeholder>'

    [KFP Executor 2026-04-29 16:31:17,613 INFO]: Looking for component `minimal_reproducible_component` in --component_module_path `/tmp/tmp.ww4yd0OgNg/ephemeral_component.py`
    [KFP Executor 2026-04-29 16:31:17,613 INFO]: Loading KFP component "minimal_reproducible_component" from /tmp/tmp.ww4yd0OgNg/ephemeral_component.py (directory "/tmp/tmp.ww4yd0OgNg" and module name "ephemeral_component")
    [KFP Executor 2026-04-29 16:31:17,613 INFO]: Got executor_input:
    {
        "inputs": {},
        "outputs": {
            "outputFile": "<my_local_path>/local_outputs/minimal-reproducible-pipeline-2026-04-29-18-31-17-314903/minimal-reproducible-component/executor_output.json"
        }
    }
    Traceback (most recent call last):
      File "<frozen runpy>", line 198, in _run_module_as_main
      File "<frozen runpy>", line 88, in _run_code
      File "/opt/app-root/lib64/python3.12/site-packages/kfp/dsl/executor_main.py", line 109, in <module>
        executor_main()
      File "/opt/app-root/lib64/python3.12/site-packages/kfp/dsl/executor_main.py", line 101, in executor_main
        output_file = executor.execute()
                      ^^^^^^^^^^^^^^^^^^
      File "/opt/app-root/lib64/python3.12/site-packages/kfp/dsl/executor.py", line 417, in execute
        result = self.func(**func_kwargs)
                 ^^^^^^^^^^^^^^^^^^^^^^^^
      File "/tmp/tmp.ww4yd0OgNg/ephemeral_component.py", line 8, in minimal_reproducible_component
        raise ValueError("This component is failing..")
    ValueError: This component is failing..
18:31:17.719 - INFO - Task 'minimal-reproducible-component' finished with status SUCCESS
18:31:17.720 - INFO - Task 'minimal-reproducible-component' has no outputs
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
18:31:17.720 - INFO - Pipeline 'minimal-reproducible-pipeline' finished with status SUCCESS

As one can see although the error was raised in the component's code both the component and pipeline have finished with status SUCCESS.

Expected result

Similarly to executing the pipeline on the remote kfp server:

  • the components' exit statuses should reflect the code failures,
  • the pipeline status should NOT be successful when any of the component fails,

Materials and Reference


Impacted by this bug? Give it a 👍.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions