Skip to content

docs(observability): add actor application telemetry best practices#522

Open
Da Huang (git286) wants to merge 1 commit into
agent-substrate:mainfrom
git286:docs/actor-telemetry-best-practices
Open

docs(observability): add actor application telemetry best practices#522
Da Huang (git286) wants to merge 1 commit into
agent-substrate:mainfrom
git286:docs/actor-telemetry-best-practices

Conversation

@git286

Copy link
Copy Markdown
Collaborator

The observability guide covers telemetry emitted by Substrate's own control plane, but says nothing to actor authors whose application code emits OpenTelemetry metrics and spans. Those signals sit in the OTel SDK's in-memory buffers (PeriodicReader accumulators, BatchSpanProcessor queues) and are subject to actor lifecycle events in ways a normal long-lived process never is.

Add a section documenting the actual behavior:

  • onPause: Full preserves the buffer; onPause: Data discards it, since process memory is excluded from the snapshot and the containers are deleted at the end of every checkpoint.
  • The workload receives no SIGTERM and no pre-suspend callback, so a flush can only happen where the actor itself initiates the suspend.
  • On resume, gVisor's CLOCK_MONOTONIC includes the suspended interval, so overdue export timers fire immediately.
  • Metric data points are stamped at collection time and so are attributed to the resume; span timestamps are recorded when the span ran and so go stale across a long suspension.
  • The OTLP connection does not survive a pod migration.

Along with the corresponding recommendations: flush before self-suspend, stay on onPause: Full, shorten the metric and span export timers separately, configure gRPC keepalives, and use a DurableDir volume under onPause: Data.

Known gaps are tracked in #450 (PreSuspend/PostResume hooks), #23 (graceful termination) and #503.

@google-cla

google-cla Bot commented Jul 24, 2026

Copy link
Copy Markdown

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

The observability guide covers telemetry emitted by Substrate's own
control plane, but says nothing to actor authors whose application code
emits OpenTelemetry metrics and spans. Those signals sit in the OTel
SDK's in-memory buffers (PeriodicReader accumulators, BatchSpanProcessor
queues) and are subject to actor lifecycle events in ways a normal
long-lived process never is.

Add a section documenting the actual behavior:

  * onPause: Full preserves the buffer; onPause: Data discards it, since
    process memory is excluded from the snapshot and the containers are
    deleted at the end of every checkpoint.
  * The workload receives no SIGTERM and no pre-suspend callback, so a
    flush can only happen where the actor itself initiates the suspend.
  * On resume, gVisor's CLOCK_MONOTONIC includes the suspended interval,
    so overdue export timers fire immediately.
  * Metric data points are stamped at collection time and so are
    attributed to the resume; span timestamps are recorded when the span
    ran and so go stale across a long suspension.
  * The OTLP connection does not survive a pod migration.

Along with the corresponding recommendations: flush before self-suspend,
stay on onPause: Full, shorten the metric and span export timers
separately, configure gRPC keepalives, and use a DurableDir volume under
onPause: Data.

Known gaps are tracked in agent-substrate#450 (PreSuspend/PostResume hooks), agent-substrate#23
(graceful termination) and agent-substrate#503.
@git286
Da Huang (git286) force-pushed the docs/actor-telemetry-best-practices branch from 388f5e5 to b59ca19 Compare July 24, 2026 20:50

@EItanya Eitan Yarmush (EItanya) left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think these notes will be important, but I think it's worth waiting until we have some specific direction on the self-suspend and graceful-suspend before putting this doc out there as it will be superseded by that

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants