Add standalone function-worker component (split the worker out of the broker)#698
Merged
Merged
Conversation
… broker) Run the Pulsar Functions worker as a separate component instead of embedded in the broker, enabled with components.function_worker. When enabled, the broker's embedded function worker is disabled completely. Running the function worker separately separates its workload from the brokers, which is useful for operational reasons (scale and manage it independently of the brokers) and security reasons (a reduced attack surface: functions run user code, so a compromise of the function worker -- for example a remote code execution -- does not directly impact the brokers). The worker is broker-attached (it connects to the broker and keeps function metadata in system topics). Function instances run with the Kubernetes runtime (one pod per instance), so the worker has its own RBAC. - New component templates modeled on the broker: function-worker statefulset, configmap (functions_worker.yml via the PF_ env mechanism), service + headless service, service-account, rbac, podmonitor, pdb, and the _function_worker.tpl helpers. New values: components.function_worker, images.function_worker, tls.function_worker, and the function_worker block. - broker-configmap forces functionsWorkerEnabled=false and broker-statefulset uses the Parallel podManagementPolicy when the standalone component is enabled. - proxy-configmap routes the Functions REST API to the function-worker service (functionWorkerWebServiceURL[TLS]); tls-certs-internal issues its cert. - Docs + example (values-function-worker.yaml); render-all patch1 exercises the new templates.
Add a CI install scenario that runs the Pulsar Functions worker as a separate component (components.function_worker) instead of embedded in the broker, and exercises it end to end with the existing function smoke test (components.functions=true gates ci::test_pulsar_function). The new .ci/clusters/values-function-worker.yaml sets toolset.useProxy=true: once the broker's embedded worker is disabled, the broker returns HTTP 409 for Functions admin REST calls and Pulsar has no broker-side redirect, so admin calls must reach the standalone worker through the proxy (which the chart already wires via functionWorkerWebServiceURL). The CI default is toolset.useProxy=false (toolset -> broker direct), hence the override here. Function packages are stored in BookKeeper via DistributedLog (default ZooKeeper+BookKeeper); the worker fetches the metadata/bookkeeper service URIs from the broker's internal-configuration endpoint, so no FileSystemPackagesStorage is required. Also: - disable the function-worker PodMonitor in .ci/values-common.yaml (CI omits the Prometheus Operator CRD, mirroring every other component), otherwise helm install --wait fails on the missing PodMonitor kind; - pin the spawned function-instance pods to small resource requests so they schedule on the single kind node (mirrors the embedded-worker scenarios); - fix the misleading function_worker.rbac.limit_to_namespace default comments (the default is namespace-scoped) and document in the example that function admin clients must reach the worker via the proxy. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RZUbdHbb856wdmKxBU4V48
…separation # Conflicts: # .github/workflows/pulsar-helm-chart-ci.yaml
…rker, not the broker
With the standalone function worker, the broker's embedded worker is disabled,
so the broker returns HTTP 409 ("Pulsar Function Worker is not enabled") for any
Functions admin REST call. The Kubernetes-runtime function instance pod runs
`pulsar-admin --admin-url <pulsarAdminUrl> functions download ...` before exec'ing
the instance, and pulsarAdminUrl was pointed at the broker — so every function
instance crashed in a loop on the download step (HTTP 409) and never reached
numRunning, as seen in the new Separate Function Worker CI scenario.
Point the instance's functionRuntimeFactoryConfigs/kubernetesContainerFactory
pulsarAdminUrl at the standalone worker's web service (which serves the Functions
REST API, including download). pulsarServiceUrl stays on the broker for the
instance's input/output/state topics. Adds a worker TLS variant.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RZUbdHbb856wdmKxBU4V48
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
By default the Pulsar Functions worker runs embedded in the broker. Running it as a separate
component separates the function worker workload from the brokers, which is useful for:
compromise of the function worker (for example a remote code execution) does not directly impact the
brokers.
Modifications
Add a standalone function-worker component, enabled with
components.function_worker. When enabled, thebroker's embedded function worker is disabled completely (
functionsWorkerEnabled=false) and a dedicatedfunction-workerStatefulSet runsbin/pulsar functions-worker.functions_worker.ymlvia thePF_env mechanism), service + headless service, service-account, RBAC,podmonitor, pdb, and the
_function_worker.tplhelpers.components.function_worker,images.function_worker,tls.function_worker, and thefunction_workerconfig block.pulsarServiceUrl/pulsarWebServiceUrland keeps function metadata in system topics). Function instances run with the Kubernetes runtime (one pod
per instance), so the worker has its own RBAC.
broker-configmapforcesfunctionsWorkerEnabled=falseandbroker-statefulsetuses theParallelpodManagementPolicywhen the standalone component is enabled.proxy-configmaproutes the Functions REST API to the function-worker service(
functionWorkerWebServiceURL[TLS]);tls-certs-internalissues its server certificate.examples/values-function-worker.yaml); the render-alltemplates-all-values-patch1overlay exercises the new templates (with TLS).
This change is independent of package storage. (For uploaded function packages on Oxia, see the separate
broker
FileSystemPackagesStoragechange, #697.)Verifying this change
helm lintandhelm templatepass locally for: defaults (output unchanged),components.function_worker=true(the component renders, the broker's embedded worker is disabled, and the proxy routes to the new service),
and the render-all
templates-all-values+patch1overlay (function-worker on, with TLS).