opentelemetry-instrumentation-celery: spans lost when worker_max_tasks_per_child recycles the pool child

[syron]

Describe your environment

opentelemetry-api==1.44.0
opentelemetry-sdk==1.44.0
opentelemetry-instrumentation==0.65b0
opentelemetry-instrumentation-celery==0.65b0
opentelemetry-distro==0.65b0
celery==5.6.3
billiard==4.2.4
kombu==5.6.2
Python 3.12.13, linux/arm64 (container), Redis-compatible broker (valkey 9)

What happened?

When a Celery worker recycles its pool children - worker_max_tasks_per_child / --max-tasks-per-child - spans recorded by those children are never exported. The child is terminated as soon as its task returns, while BatchSpanProcessor is still holding that task's spans waiting on its schedule_delay (5s by default). Nothing flushes on the way out, so the queue dies with the process.

There is no error, no warning, and no dropped-span counter. The worker looks healthy, the tasks succeed, and the traces simply never arrive.

At max_tasks_per_child = 1 - which is a real configuration, not a corner case - this loses effectively everything: 10 tasks in, 1 span out.

Steps to Reproduce

Self-contained reproduction (compose + Dockerfile + three small scripts): https://github.com/syron/paperless-ngx-otel/tree/main/repro/celery-prefork

podman compose build && TASKS=10 podman compose run --rm repro

It runs one Celery task that opens one span, ten times, under five configurations, and counts what the console exporter emitted.

Expected Result

Spans recorded in a pool child are exported before that child exits, the same as in every other configuration. Ten tasks should yield ten spans.

Actual Result

	Worker configuration	run/repro.emit	task-span
A	celery --pool prefork	10	10
B	celery --pool solo	10	10
D	prefork + sitecustomize shim	10	10
E	prefork + --max-tasks-per-child 1	1	1

Scenario C uses a bare os.fork() with no Celery at all, and exported all three of its spans: parent-before-fork, child-after-fork and parent-after-fork.

Only the recycling worker loses spans - 9 of 10. The one that survives is the final child, which exits with the worker rather than being recycled.

Additional context

The other scenarios exist to eliminate the explanations that look obvious first, and they all check out:

• The prefork pool is fine (A). Forked children export normally.
• --pool solo is avoidance, not a fix (B). It has no children to recycle.
• The SDK's fork handling works (C). os.register_at_fork restarts the batch processor's worker thread in the child, and a span created after a bare os.fork() exports.
• Attachment method is irrelevant (D). opentelemetry-instrument and a sitecustomize.py on PYTHONPATH behave identically.

What remains is the recycle itself.

Why this is worth more than the setting suggests. Recycling children after N tasks is standard practice wherever tasks leak memory, much of the image-processing and ML-adjacent Celery population. And because the failure is silent, it does not diagnose as "child recycling". From the outside it looks like "the prefork pool doesn't export", which sends people to a workaround that costs them concurrency for the wrong reason. That is the conclusion I reached first, and it was wrong.

One concrete instance: paperless-ngx hardcodes CELERY_WORKER_MAX_TASKS_PER_CHILD = 1 and does not expose it as a setting, so every paperless deployment is in scenario E and cannot configure its way out.

Suggested fix. Celery already emits worker_process_shutdown in the child before it exits. opentelemetry-instrumentation-celery connects to worker_process_init today; connecting the matching shutdown signal and calling force_flush() on the tracer (and meter) provider would close this on an existing, well-defined path.

An argument could be made that this belongs in core instead - that BatchSpanProcessor should flush when its process is going away regardless of who owns the process - but the Celery signal is the narrower, more targeted fix and does not change behaviour for anyone else.

The same loss applies to metrics recorded in a recycled child, via the same mechanism.

Would you like to implement a fix?

Yes

Tip

_{React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.}

[axiom-of-choice]

Reproduced on main (80d2306) with celery 5.3.6 / billiard 4.2.1, using a prefork worker over a filesystem broker so no container or Redis is needed. 10 tasks, each opening one span, counting spans that actually reached the exporter (written to a file, so the count survives the child dying):

max_tasks_per_child   spans exported   expected
1                                  0         20
2                                  0         20
5                                 10         20
10                                16         20
1000 (no recycling)               20         20

The last row is the control: with recycling disabled nothing is lost, which isolates the variable. Loss scales with recycling frequency, and at max_tasks_per_child=1 nothing is exported at all.

On where to hook the flush: billiard's Worker._do_exit runs inside the child, in the finally of its __call__, and calls on_exit -> celery's process_destructor -> the worker_process_shutdown signal, all before os._exit. I confirmed the signal fires once per recycled child with a different pid each time, so it is a usable flush point inside the dying process rather than in the parent.

That gives a fix matching what aws-lambda already does for the Lambda freeze: connect to worker_process_shutdown and force_flush the tracer and meter providers, skipping providers that have no force_flush and logging rather than raising if a flush fails, so it cannot stop the child from exiting. With that, every row above becomes 20/20.

Is that the direction you'd want? I have the change and tests ready if so.

[syron]

Thanks for digging into this properly. Your numbers line up with mine once you account for two spans per task, and your table is better evidence than my original: the no-recycling control isolates the variable, and the gradient shows the loss scales with recycle frequency rather than being all-or-nothing, which I had implied.

Confirming the billiard path is the part I couldn't. I proposed worker_process_shutdown from the docs; you've shown it actually runs inside the dying child before os._exit, once per recycle with a distinct pid. That settles it as the hook.

Direction looks right to me, with two suggestions. Bound the flush with a timeout the way aws-lambda does via OTEL_INSTRUMENTATION_AWS_LAMBDA_FLUSH_TIMEOUT - otherwise a stuck exporter delays every child recycle and this becomes a throughput problem. And it's worth confirming what happens when a child is killed hard rather than recycled gracefully, e.g. on a task time limit: I'd expect it to bypass that finally entirely, in which case it's a documented limitation rather than something the fix covers.

That said, I'm the reporter, not a maintainer, so I can't greenlight the approach. Could someone from the celery side confirm the direction? @axiom-of-choice has the change and tests ready.