Skip to content

fix(reporting): stream BigQuery rows and isolate reporting errors in EvalService - #729

Open
graceqi-g wants to merge 2 commits into
GoogleCloudPlatform:mainfrom
graceqi-g:fix/bq-streaming-reporting-isolation
Open

graceqi-g wants to merge 2 commits into
GoogleCloudPlatform:mainfrom
graceqi-g:fix/bq-streaming-reporting-isolation

Conversation

@graceqi-g

Copy link
Copy Markdown
Collaborator

Summary

BigQuery enforces a hard quota of 1,500 load jobs per table per day (load_job_per_table.long). Previously, BigQueryReporter.store() created a separate load_table_from_dataframe() load job for each of the four EvalBench tables (configs, results, scores, summary) on every evaluation, exhausting the per-table token bucket under high-concurrency CI suites.

In addition, EvalService._process_results() called reporter.store() without isolating per-reporter exceptions, so a BigQuery failure aborted CSV/artifact reporters, skipped /tmp temp-file cleanup, and failed the entire completed Eval() RPC.

Changes

  • evalbench/reporting/bqstore.py: Switch BigQueryReporter.store() from load_table_from_dataframe() to streaming inserts via client.insert_rows_json() (0 load jobs consumed):
    • Creates missing tables on NotFound with inferred BigQuery schemas.
    • Patches newly added columns via update_table() (handling concurrent updates on Conflict/PreconditionFailed) to preserve schema-addition behavior.
    • Coerces DataFrame values (including DATETIME/TIMESTAMP/DATE, REPEATED fields, INT64 summary/score columns, NaN/Inf/pd.NA) to match target BigQuery SchemaField types.
    • Splits streaming batches by both row count and serialized byte size (< 8 MiB, safely below BigQuery's 10 MB insertAll limit).
    • Retries transient errors (TooManyRequests, ServiceUnavailable, InternalServerError, NotFound, and transient/schema-propagation row errors) with exponential backoff without retrying permanent failures like Forbidden.
  • evalbench/eval_service.py:
    • Extract _store_reports() (matching evalbench.py) so per-reporter failures are logged and recorded under summary["reporting_errors"] without blocking other reporters or discarding completed evaluation scores.
    • Wrap temp-file loading in try/finally with unlink(missing_ok=True) so /tmp JSON temp files are always cleaned up even on EmptyEvalResultError or unexpected exceptions.
  • Unit tests:
    • Add evalbench/test/bqstore_test.py covering streaming inserts, zero load jobs, missing table creation, schema column additions, repeated/NaN coercion, and retry vs. fail-fast error handling.
    • Add evalbench/test/eval_service_reporting_test.py covering reporter failure isolation and temp-file cleanup.

Verification

  • pytest evalbench/test/bqstore_test.py evalbench/test/eval_service_reporting_test.py evalbench/test/evalbench_test.py evalbench/test/eval_executor_test.py evalbench/test/gcs_artifact_test.py evalbench/test/agent_version_reporting_test.py evalbench/test/analyzer_empty_fix_test.py -vv (57 passed)
  • pycodestyle . --config=.pycodestyle (0 errors)

…EvalService

BigQuery enforces a hard quota of 1,500 load jobs per table per day
(load_job_per_table.long). BigQueryReporter.store() previously created a
separate load_table_from_dataframe() load job for each of the four EvalBench
tables (configs, results, scores, summary) on every evaluation, exhausting the
per-table token bucket under high-concurrency CI suites. In addition,
EvalService._process_results() called reporter.store() without isolating
per-reporter exceptions, so a BigQuery failure aborted CSV/artifact reporters,
skipped temp-file cleanup, and failed the entire completed Eval() RPC.

- Switch BigQueryReporter.store() from load_table_from_dataframe() to
  streaming inserts via client.insert_rows_json() (0 load jobs consumed),
  creating missing tables on NotFound, patching new columns via update_table()
  to preserve schema-addition behavior, coercing DataFrame values to target
  BigQuery SchemaField types, splitting requests below BigQuery's 10 MB
  insertAll limit, and retrying transient/schema-propagation errors.
- Extract _store_reports() in eval_service.py (matching evalbench.py) so a
  reporter failure logs the error and records summary["reporting_errors"]
  without discarding completed evaluation scores or leaking /tmp files.
- Add unit tests in bqstore_test.py and eval_service_reporting_test.py.
…coercion, and reporter init isolation

Address self-review findings on PR GoogleCloudPlatform#729:
- Allow sibling 'stopped' row errors when checking retryable BigQuery insertErrors
  while still failing fast on permanent row errors.
- Pass stable per-batch row_ids to insert_rows_json() and retry only failed indices
  on partial retryable errors to prevent duplicate rows.
- Infer schema across the full DataFrame (all non-null values/elements, unwrapping
  NumPy scalars) before chunking, and retry update_table() up to 5 times on
  concurrent Conflict/PreconditionFailed.
- Only refresh cached table schemas on NotFound or 'no such field' errors, and
  re-coerce pending rows when the refreshed schema changed.
- Fix _coerce_scalar() / _dataframe_to_json_rows() edge cases: exact INT64 precision
  above 2^53, empty/offset DATETIME/TIMESTAMP/DATE strings (Python 3.10-safe),
  NaN/Inf sanitization in JSON, and recursive RECORD/STRUCT coercion.
- Move reporter construction inside _process_results() via _build_reporters() so
  reporter init failures are isolated and /tmp files are always unlinked; raise in
  Eval() when summary_in_response is falsy and every configured reporter failed.
- Expand unit tests in bqstore_test.py and eval_service_reporting_test.py.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant