docs(bqaa): document ADK Java preview-readiness fixes (pause/resume tracing, drop stats, redaction)

docs/integrations/bigquery-agent-analytics.md

@@ -29,21 +29,24 @@ transfers, state checkpoints, event compaction, and long-running tools. It adds
 four new event types — `AGENT_TRANSFER`, `AGENT_STATE_CHECKPOINT`,
 `EVENT_COMPACTION`, and `TOOL_PAUSED` — and stamps an `attributes.adk` envelope
 on every row so you can reconstruct the agent execution graph and join a paused
-tool to the row that resumes it. See [Agent workflow and pause/resume events
+tool to the row that resumes it. In **Java**, this support currently covers the
+`TOOL_PAUSED` event and its pause/resume pairing keys only (without the
+`attributes.adk` envelope). See [Agent workflow and pause/resume events
 (ADK 2.0)](#adk-2-events) for details.
 
-The plugin includes three reliability and observability fixes:
+The plugin includes three reliability and observability fixes (Java: v1.7.0 or
+later):
 
 - **Cross-region Storage Write API routing.** Writes to BigQuery datasets
   outside the `US` multi-region (for example `EU` or `northamerica-northeast1`)
   now route to the region that owns the write stream. Previously they could
   fail with a "session not found" / stream-not-found error and silently drop
   every row.
 - **Dropped-event observability.** Dropped rows are tracked per drop reason
-  (`queue_full`, `arrow_prep_failed`, `retry_exhausted`, `non_retryable`,
-  `unexpected_error`) and exposed via
-  `BigQueryAgentAnalyticsPlugin.get_drop_stats()` so a host can poll and export
-  the counts to its own monitoring.
+  and exposed via `BigQueryAgentAnalyticsPlugin.get_drop_stats()` (Python) or
+  `getDropStats()` (Java) so a host can poll and export the counts to its own
+  monitoring. The reason keys differ per language — see [Dropped-event
+  observability](#dropped-event-observability).
 - **No duplicate spans in Cloud Trace.** When Agent Engine telemetry
   (`GOOGLE_CLOUD_AGENT_ENGINE_ENABLE_TELEMETRY=true`) or any other Cloud Trace
   exporter is wired to the global tracer provider, the plugin no longer
@@ -723,7 +726,7 @@ provides a comprehensive reference with example values.
 | **session_id** | `STRING` | `NULLABLE` | A persistent identifier for the entire conversation thread. Stays constant across multiple turns and sub-agent calls. | `04275a01-1649-4a30-b6a7-5b443c69a7bc` |
 | **invocation_id** | `STRING` | `NULLABLE` | The unique identifier for a single execution turn or request cycle. Corresponds to `trace_id` in many contexts. | `e-b55b2000-68c6-4e8b-b3b3-ffb454a92e40` |
 | **user_id** | `STRING` | `NULLABLE` | The identifier of the user (human or system) initiating the session. Extracted from the `User` object or metadata. | `test_user` |
-| **trace_id** | `STRING` | `NULLABLE` | 32-character hex Trace ID. Inherited from the ambient OpenTelemetry span when one is active (e.g. Agent Engine's invocation span or the ADK Runner span) so BigQuery rows join cleanly to your existing Cloud Trace traces; otherwise generated by the plugin per invocation. Links all operations within a single distributed request lifecycle. | `a2c7f13d3a3f0bbb8793692f76a6012a` |
+| **trace_id** | `STRING` | `NULLABLE` | Trace identifier. It is a 32-character hex OpenTelemetry trace ID when inherited from an ambient span (e.g. Agent Engine's invocation span or the ADK Runner span), so BigQuery rows join cleanly to your existing Cloud Trace traces. Without an ambient span, Python also generates that format as its per-invocation fallback, while Java falls back to the ADK invocation ID. Links all operations within a single distributed request lifecycle. | `a2c7f13d3a3f0bbb8793692f76a6012a` |
 | **span_id** | `STRING` | `NULLABLE` | 16-character hex Span ID identifying this specific atomic operation. **Tracked on the plugin's internal stack, not exported as an OTel span** — the plugin does not call `tracer.start_span` against your configured OpenTelemetry provider. The root invocation span reuses the ambient OTel span's id when one is active; child spans are generated internally (see [Tracing and observability](#tracing-and-observability)). | `3916f5762bcd4d42` |
 | **parent_span_id** | `STRING` | `NULLABLE` | 16-character hex Span ID of the immediate caller. Used to reconstruct the parent-child execution tree (DAG). | `4c4a42bfdeb84934` |
 | **content** | `JSON` | `NULLABLE` | The primary event payload. Structure is polymorphic based on `event_type`. | `{"system_prompt": "You are...", "prompt": [{"role": "user", "content": "hello"}], "response": "Hi", "usage": {"total": 15}}` |
@@ -848,7 +851,11 @@ columns:
 The four workflow views (`v_agent_transfer`, `v_agent_state_checkpoint`,
 `v_event_compaction`, `v_tool_paused`) and the `pause_kind` / `function_call_id`
 columns on `v_tool_completed` come with the [ADK 2.0 workflow event
-support](#adk-2-events).
+support](#adk-2-events). In **Java** (v1.7.0+), only
+`v_tool_paused` and the `pause_kind` / `function_call_id` columns on
+`v_tool_completed` are created; `v_agent_transfer`, `v_agent_state_checkpoint`,
+and `v_event_compaction` are Python-only (the Java plugin does not emit those
+events).
 
 ## Event types and payloads {#event-types}
 
@@ -1037,8 +1044,8 @@ updated by tools).
 !!! note "Built-in redaction"
 
     State keys prefixed with `temp:` or `secret:` are automatically redacted to
-    `[REDACTED]` in the logged `state_delta`. See [Built-in
-    redaction](#built-in-redaction) for details.
+    `[REDACTED]` in the logged `state_delta` (Java: `temp:` only). See
+    [Built-in redaction](#built-in-redaction) for details.
 
 ```json
 {
@@ -1141,9 +1148,18 @@ Logged when an A2A remote agent call completes.
 ### Agent workflow and pause/resume events (ADK 2.0) {#adk-2-events}
 
 <div class="language-support-tag">
-  <span class="lst-supported">Supported in ADK</span><span class="lst-python">Python v2.3.0</span>
+  <span class="lst-supported">Supported in ADK</span><span class="lst-python">Python v2.3.0</span><span class="lst-java">Java v1.7.0</span>
 </div>
 
+!!! note "Java support"
+
+    The **Java** plugin supports a subset of this section: it emits
+    `TOOL_PAUSED` and the pause/resume pairing described below,
+    but does **not** emit `AGENT_TRANSFER`, `AGENT_STATE_CHECKPOINT`, or
+    `EVENT_COMPACTION`, and does not write the `attributes.adk` envelope — the
+    Java plugin stores `pause_kind` and `function_call_id` at the **top level**
+    of `attributes` instead (see the query note below).
+
 ADK 2.0 introduced multi-agent workflows (agents that transfer control,
 checkpoint their state, and compact long histories) and long-running tools that
 pause and resume across turns. The plugin makes these flows observable with four
@@ -1152,7 +1168,8 @@ rows back to the ADK event that produced them.
 
 #### The `attributes.adk` envelope
 
-Every row now carries an `attributes.adk` object. `schema_version` and
+The envelope is written by the **Python** plugin only. Every row now carries an
+`attributes.adk` object. `schema_version` and
 `app_name` are always present; the remaining fields are added only for rows that
 originate from an ADK event (lifecycle and workflow events), so on a callback-only
 row they are simply absent (and resolve to SQL `NULL` when queried).
@@ -1254,6 +1271,20 @@ see the note below.)
 }
 ```
 
+!!! note "Java attribute location"
+
+    The Java plugin writes the pair keys at the top level of `attributes` —
+    `"attributes": {"pause_kind": "tool", "function_call_id": "call-789"}` —
+    with no `adk` wrapper. In the base-table query below, replace
+    `'$.adk.pause_kind'` / `'$.adk.function_call_id'` with `'$.pause_kind'` /
+    `'$.function_call_id'`. The view-based query works unchanged for both
+    languages, because the views expose the keys as flat columns.
+
+    Java also stamps the same top-level pair keys on `HITL_*_REQUEST_COMPLETED`
+    rows, so a HITL `TOOL_PAUSED` row can be joined directly to its completion
+    on `function_call_id` in the base table (there is no dedicated view for
+    HITL completions).
+
 !!! note "Relationship to HITL events"
 
     A HITL request (`adk_request_confirmation`, etc.) still emits its dedicated
@@ -1924,6 +1955,21 @@ by credential services) is never persisted in BigQuery.
     on raw content, so use it to add masking for secrets that may appear in
     free-form payloads.
 
+!!! note "Built-in redaction in Java"
+
+    The Java plugin includes built-in redaction in v1.7.0 and later.
+    It redacts the same six key names (case-insensitive) recursively across the
+    assembled `attributes` tree — including session state and state deltas —
+    and any key prefixed with **`temp:`**. Unlike Python, the Java plugin does
+    **not** redact keys prefixed `secret:`; keep secrets out of non-`temp:`
+    state scopes or mask them with a custom `contentFormatter`.
+
+    A custom Java `contentFormatter` must be **thread-safe** (it is called
+    concurrently across invocations) and **fast/non-blocking** (it runs on the
+    event-processing path), and it must return a **new object** rather than
+    mutating the content it receives. If it throws, the Java plugin drops the
+    row's content (fail-closed) rather than logging the unformatted payload.
+
 ### Use `content_formatter` to redact additional secrets
 
 Provide a custom `content_formatter` function in `BigQueryLoggerConfig` to strip
@@ -2119,7 +2165,10 @@ call) reconstructs cleanly from BigQuery.
   `TracerProvider`, so its instrumentation never reaches your configured
   exporter — this is what prevents duplicate spans in Cloud Trace when Agent
   Engine telemetry is enabled (`GOOGLE_CLOUD_AGENT_ENGINE_ENABLE_TELEMETRY=true`)
-  or when you wire any other Cloud Trace exporter into the host process.
+  or when you wire any other Cloud Trace exporter into the host process. The
+  same internal, ID-only span tracking applies to the Java plugin in v1.7.0
+  and later; earlier Java builds created plugin-owned OpenTelemetry
+  spans that could surface as duplicates next to framework spans.
 - **`trace_id` inherited from the ambient OTel span when present.** If the
   surrounding runtime has already started an OTel span — Agent Engine's
   invocation span, the ADK `Runner` invocation span, or any span you opened
@@ -2128,7 +2177,8 @@ call) reconstructs cleanly from BigQuery.
   Cloud Trace traces via a shared `trace_id`.
 - **Fallback when no ambient span is present.** If no ambient OTel span is
   active (e.g. a non-Agent-Engine deployment with no host-side tracer
-  configured), the plugin generates a per-invocation 32-hex `trace_id` so the
+  configured), the plugin generates a per-invocation 32-hex `trace_id` (the
+  Java plugin falls back to the ADK invocation ID as the `trace_id`) so the
   parent-child hierarchy is always preserved in BigQuery, even without any
   external tracer setup.
 - **No `TracerProvider` is required.** Configuring an OpenTelemetry
@@ -2181,6 +2231,14 @@ call) reconstructs cleanly from BigQuery.
 
     - **`plugin.close()`**: Gracefully shuts down the plugin, flushing pending events and releasing resources (including the BigQuery write client and executors).
     - **Automatic Closure**: If you are using `InMemoryRunner`, calling `runner.close()` will automatically close all registered plugins, including the BigQuery Agent Analytics plugin.
+    - **`plugin.getDropStats()`** (v1.7.0+): Returns an
+      `ImmutableMap<String, Long>` of dropped-event counts per drop reason. See
+      [Dropped-event observability](#dropped-event-observability).
+    - **JVM shutdown hook** (v1.7.0+): The plugin registers
+      a shutdown hook at construction, so pending events are drained
+      (best-effort, bounded by `shutdownTimeout`) at JVM exit even if `close()`
+      is never called. An explicit `close()` deregisters the hook. Still prefer
+      calling `close()` for a deterministic flush.
 
     ```java
     // Manual shutdown
@@ -2194,7 +2252,7 @@ queue overflows or when a write ultimately fails. The plugin tracks dropped
 rows per `drop_reason` and exposes a polling API so a host can detect, alert
 on, and ship the counts to its own monitoring.
 
-**Drop reasons:**
+**Drop reasons (Python):**
 
 | Reason | Cause |
 |---|---|
@@ -2204,15 +2262,42 @@ on, and ship the counts to its own monitoring.
 | `non_retryable` | Storage Write API returned a non-retryable error (permissions, quota, schema rejection). Usually requires operator intervention. |
 | `unexpected_error` | Any other exception caught while preparing or writing the batch. |
 
+**Drop reasons (Java, v1.7.0+):**
+
+| Reason | Cause |
+|---|---|
+| `queue_full` | The in-memory batch queue overflowed. Increase `queueMaxSize` on `BigQueryLoggerConfig`, raise `batchSize`, or scale the consumer side. |
+| `append_error` | Batch preparation or append failed for a cause other than `AppendSerializationError` — including timeouts, exhausted or non-retryable writes, and unexpected conversion failures. |
+| `serialization_error` | A row could not be serialized for the write stream (typically a schema/type mismatch). Inspect logs for the offending field. |
+| `after_close` | A row reached an already-closed per-invocation processor. |
+| `shutdown_timeout` | Queued rows remained when the bounded final drain expired. |
+| `writer_permit_exhausted` | The live-writer safety cap was exhausted — normally during a Storage Write outage or delayed cleanup. |
+| `writer_create_error` | `StreamWriter` construction or processor startup failed. |
+| `late_after_finalize` | Async work completed after its invocation was finalized or while the plugin was closing. |
+
 **Reading the counts:**
 
-```python
-# Snapshot of {drop_reason: count} since plugin start.
-stats = plugin.get_drop_stats()
-# Example: {"queue_full": 12, "retry_exhausted": 0, ...}
+=== "Python"
 
-total_dropped = sum(stats.values())
-```
+    ```python
+    # Snapshot of {drop_reason: count} since plugin start.
+    stats = plugin.get_drop_stats()
+    # Example: {"queue_full": 12, "retry_exhausted": 0, ...}
+
+    total_dropped = sum(stats.values())
+    ```
+
+=== "Java"
+
+    ```java
+    // Snapshot of {drop_reason: count} since plugin start.
+    ImmutableMap<String, Long> stats = plugin.getDropStats();
+    // Example: {queue_full=12, append_error=0, serialization_error=0,
+    //           after_close=0, shutdown_timeout=0, writer_permit_exhausted=0,
+    //           writer_create_error=0, late_after_finalize=0}
+
+    long totalDropped = stats.values().stream().mapToLong(Long::longValue).sum();
+    ```
 
 **Exporting to your monitoring system** — poll periodically and ship the deltas:
 
@@ -2237,9 +2322,10 @@ async def export_loop(plugin):
         await asyncio.sleep(60)
 ```
 
-A non-zero `queue_full` or `retry_exhausted` count on a sustained basis is the
-clearest signal that BQAA is at risk of data loss — surface it on a dashboard or
-alert.
+Any non-zero count means analytics rows were dropped before reaching BigQuery.
+Alert on every reason; sustained `queue_full` or write-error counts
+(`retry_exhausted` / `non_retryable` in Python, `append_error` in Java)
+generally indicate throughput or Storage Write health problems.
 
 ### Multiprocessing and fork safety