[Python Functions] Close the configuration parity gaps with the Java runtime (master issue) #26412

Search before reporting

• I searched in the issues and found nothing similar.

This is a master issue. The individual gaps are tracked as the sub-issues listed below; this one holds the comparison that produced them. It is the Python counterpart to #26404, which tracks the same audit for the Go runtime.

Motivation

The Python function runtime is much closer to the Java reference than the Go runtime is, but it still drops a set of FunctionDetails fields. As with Go, the failures are silent: the configuration is accepted by pulsar-admin, stored by the worker, reported back by functions get, carried into the instance in the protobuf, and then never read.

I audited the three runtimes field by field against Function.proto, reading the consuming code rather than searching for field names — Function_pb2.py is generated and contains every field name, so a naive grep reports full support for everything.

(An earlier revision of this issue listed cleanupSubscription as a Python gap that Go honoured. Both halves were wrong: it is implemented by the worker in FunctionActioner, runtime-agnostically, so no instance runtime implements it and none needs to. The row has been removed.)

Producer maxPendingMessages and maxPendingMessagesAcrossPartitions are also unapplied on master, but they are already fixed by the open batching PR and are not tracked separately here.

Solution

Tracked individually:

• batchingSpec ignored, so every function has a fixed 10ms publish-latency floor — [Functions] Python function runtime hardcodes producer batching config, ignoring producerSpec.batchingSpec #26390 (fixed by [fix][fn] Honour producerSpec batching configuration in the Python function runtime #26392)
• retryDetails ignored, so no dead letter policy is created — [Python Functions] Python instance runtime silently ignores deadLetterTopic / maxMessageRetries #26397 (PR open: [fix][fn] Support deadLetterTopic and maxMessageRetries for Python functions #26400)
• consumerProperties, poolMessages and messagePayloadProcessorSpec ignored — [Python Functions] consumerProperties, poolMessages and messagePayloadProcessorSpec are ignored #26410
• negativeAckRedeliveryDelayMs ignored, so the 60s client default always applies — [Python Functions] negativeAckRedeliveryDelayMs is ignored, so the client default always applies #26411 (fixed by [fix][fn] Honour negativeAckRedeliveryDelayMs in the Python function runtime #26413)

Two of the four are now fixed on master (#26392, #26413). A third, retryDetails, has PR #26400 open. The table above reflects the current state rather than the state at filing.

A note on how these should fail. The Go runtime refuses EFFECTIVELY_ONCE explicitly rather than ignoring it:

// pulsar-function-go/pf/instanceConf.go:137
panic("Go instance current not support EFFECTIVELY_ONCE processing guarantees.")

Every row marked "no" above fails the other way: accepted and dropped. Where a field is not going to be supported — messagePayloadProcessorSpec is the likely candidate here, being a Java-centric extension point — refusing it, or logging a warning at startup, is better than continuing to accept it silently. That is the same point raised in #26404 and is worth settling once for both runtimes.

Alternatives

Fixing these as one change was considered and rejected: they touch different parts of python_instance.py, two already have separate PRs in review, and cleanupSubscription may not belong in the instance at all — it may be the worker's delete path that needs it, which is a different discussion from the consumer-option fields.

Anything else?

The audit was done against origin/master. Java is treated as the reference implementation; I spot-checked the Java side for the rows that mattered (PulsarSource for retryDetails and the negative-ack delay, BatchingUtils for batchingSpec, JavaInstanceRunnable for the presence guards) rather than auditing it exhaustively.

Are you willing to submit a PR?

• I'm willing to submit a PR!