Skip to content

fix(datasets): limit the variables that a dataset can read on the eval server - #733

Draft
omkargaikwad23 wants to merge 1 commit into
mainfrom
fix/server-dataset-env-allowlist
Draft

omkargaikwad23 wants to merge 1 commit into
mainfrom
fix/server-dataset-env-allowlist

Conversation

@omkargaikwad23

Copy link
Copy Markdown
Collaborator

What this fixes

A dataset can contain ${VAR} placeholders, and EvalBench replaces them with values from the environment. On the eval server, the dataset comes from the client, and ListEvalInputs sends the expanded dataset to the client. Thus a client could read each environment variable of the server.

How

  • On the eval server (eval_server.py), a dataset can read only the variables in EVALBENCH_DATASET_ENV_VARS (comma-separated). The default is EVAL_GCP_PROJECT_ID,EVAL_GCP_PROJECT_REGION,GOOGLE_CLOUD_PROJECT.
  • A placeholder for a different variable fails the load. The error names the variable and the setting. It does not contain the value.
  • On the eval server, the form without braces ($VAR) is not expanded.
  • A local run does not change.

Behavior change

On the eval server, a dataset that reads a variable outside the list now fails. To permit a variable, add it to EVALBENCH_DATASET_ENV_VARS on the server.

Testing

  • New tests in dataset_env_placeholders_test.py.
  • A before/after comparison is in a comment on this PR.

@omkargaikwad23

omkargaikwad23 commented Nov 21, 2026 •

Copy link
Copy Markdown
Collaborator Author

Verification

I loaded the same dataset with load_gemini_cli_json on main (6ca77dc) and on this branch. The environment had EVAL_GCP_PROJECT_ID=my-project and EVAL_DB_PASSWORD=FAKE-DB-PASSWORD. The table shows the starting_prompt after the load.

# Process starting_prompt in the dataset main This PR
1 eval_server.py pw=${EVAL_DB_PASSWORD} pw=FAKE-DB-PASSWORD ValueError, the load fails
2 eval_server.py pw=$EVAL_DB_PASSWORD pw=FAKE-DB-PASSWORD pw=$EVAL_DB_PASSWORD
3 eval_server.py p=${EVAL_GCP_PROJECT_ID} p=my-project p=my-project
4 evalbench.py pw=${EVAL_DB_PASSWORD} pw=FAKE-DB-PASSWORD pw=FAKE-DB-PASSWORD
5 evalbench.py p=${EVAL_GCP_PROJECT_ID} p=my-project p=my-project

The error text in row 1:

Unresolved ${...} placeholder(s) in dataset evalset.json: ['EVAL_DB_PASSWORD'].
On the eval server, a dataset can read only these variables:
['EVAL_GCP_PROJECT_ID', 'EVAL_GCP_PROJECT_REGION', 'GOOGLE_CLOUD_PROJECT'].
The EVALBENCH_DATASET_ENV_VARS variable of the server sets this list.

Local tests:

  • dataset_env_placeholders_test.py and the existing tests that load a dataset: 61 passed.
  • pytest evalbench/test with SKIP_CLOUD_TESTS=true: 1303 passed, 11 skipped.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant