Repository navigation
Conversation
…EvalService BigQuery enforces a hard quota of 1,500 load jobs per table per day (load_job_per_table.long). BigQueryReporter.store() previously created a separate load_table_from_dataframe() load job for each of the four EvalBench tables (configs, results, scores, summary) on every evaluation, exhausting the per-table token bucket under high-concurrency CI suites. In addition, EvalService._process_results() called reporter.store() without isolating per-reporter exceptions, so a BigQuery failure aborted CSV/artifact reporters, skipped temp-file cleanup, and failed the entire completed Eval() RPC. - Switch BigQueryReporter.store() from load_table_from_dataframe() to streaming inserts via client.insert_rows_json() (0 load jobs consumed), creating missing tables on NotFound, patching new columns via update_table() to preserve schema-addition behavior, coercing DataFrame values to target BigQuery SchemaField types, splitting requests below BigQuery's 10 MB insertAll limit, and retrying transient/schema-propagation errors. - Extract _store_reports() in eval_service.py (matching evalbench.py) so a reporter failure logs the error and records summary["reporting_errors"] without discarding completed evaluation scores or leaking /tmp files. - Add unit tests in bqstore_test.py and eval_service_reporting_test.py.
graceqi-g
requested review from
IsmailMehdi,
helloeve and
prernakakkar-google
as code owners
November 20, 2026 23:27
…coercion, and reporter init isolation Address self-review findings on PR GoogleCloudPlatform#729: - Allow sibling 'stopped' row errors when checking retryable BigQuery insertErrors while still failing fast on permanent row errors. - Pass stable per-batch row_ids to insert_rows_json() and retry only failed indices on partial retryable errors to prevent duplicate rows. - Infer schema across the full DataFrame (all non-null values/elements, unwrapping NumPy scalars) before chunking, and retry update_table() up to 5 times on concurrent Conflict/PreconditionFailed. - Only refresh cached table schemas on NotFound or 'no such field' errors, and re-coerce pending rows when the refreshed schema changed. - Fix _coerce_scalar() / _dataframe_to_json_rows() edge cases: exact INT64 precision above 2^53, empty/offset DATETIME/TIMESTAMP/DATE strings (Python 3.10-safe), NaN/Inf sanitization in JSON, and recursive RECORD/STRUCT coercion. - Move reporter construction inside _process_results() via _build_reporters() so reporter init failures are isolated and /tmp files are always unlinked; raise in Eval() when summary_in_response is falsy and every configured reporter failed. - Expand unit tests in bqstore_test.py and eval_service_reporting_test.py.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
BigQuery enforces a hard quota of 1,500 load jobs per table per day (
load_job_per_table.long). Previously,BigQueryReporter.store()created a separateload_table_from_dataframe()load job for each of the four EvalBench tables (configs,results,scores,summary) on every evaluation, exhausting the per-table token bucket under high-concurrency CI suites.In addition,
EvalService._process_results()calledreporter.store()without isolating per-reporter exceptions, so a BigQuery failure aborted CSV/artifact reporters, skipped/tmptemp-file cleanup, and failed the entire completedEval()RPC.Changes
evalbench/reporting/bqstore.py: SwitchBigQueryReporter.store()fromload_table_from_dataframe()to streaming inserts viaclient.insert_rows_json()(0 load jobs consumed):NotFoundwith inferred BigQuery schemas.update_table()(handling concurrent updates onConflict/PreconditionFailed) to preserve schema-addition behavior.DATETIME/TIMESTAMP/DATE,REPEATEDfields,INT64summary/score columns,NaN/Inf/pd.NA) to match target BigQuerySchemaFieldtypes.insertAlllimit).TooManyRequests,ServiceUnavailable,InternalServerError,NotFound, and transient/schema-propagation row errors) with exponential backoff without retrying permanent failures likeForbidden.evalbench/eval_service.py:_store_reports()(matchingevalbench.py) so per-reporter failures are logged and recorded undersummary["reporting_errors"]without blocking other reporters or discarding completed evaluation scores.try/finallywithunlink(missing_ok=True)so/tmpJSON temp files are always cleaned up even onEmptyEvalResultErroror unexpected exceptions.evalbench/test/bqstore_test.pycovering streaming inserts, zero load jobs, missing table creation, schema column additions, repeated/NaN coercion, and retry vs. fail-fast error handling.evalbench/test/eval_service_reporting_test.pycovering reporter failure isolation and temp-file cleanup.Verification
pytest evalbench/test/bqstore_test.py evalbench/test/eval_service_reporting_test.py evalbench/test/evalbench_test.py evalbench/test/eval_executor_test.py evalbench/test/gcs_artifact_test.py evalbench/test/agent_version_reporting_test.py evalbench/test/analyzer_empty_fix_test.py -vv(57 passed)pycodestyle . --config=.pycodestyle(0 errors)