Skip to content

fix(reporting): remove secret values from the reported config - #731

Draft
omkargaikwad23 wants to merge 1 commit into
mainfrom
fix/report-redact-secret-config-values
Draft

omkargaikwad23 wants to merge 1 commit into
mainfrom
fix/report-redact-secret-config-values

Conversation

@omkargaikwad23

Copy link
Copy Markdown
Collaborator

What this fixes

config_to_df writes the run config, the model config, and the database configs to configs.csv and to BigQuery. It wrote each value as it was. Thus a password or an API key in a config was in the report. This includes a value that !ENV reads from the environment.

How

redact_secrets makes a copy of the config for the report:

  • It replaces a value with [REDACTED] if the last words of the key name a secret: password, passwd, secret, token, credential, credentials, authorization, apikey, or key after api, private, access, or secret.
  • It replaces the password in a URL of the form scheme://user:password@host.
  • It keeps a Secret Manager resource name (projects/.../secrets/...). That value is a reference, not a secret.

The config that the run uses does not change.

Limits

  • The check uses the key name. A secret under a different key name stays in the report.
  • Rows that are already in BigQuery, and old configs.csv files, do not change.

Testing

  • New tests in config_redaction_test.py.
  • A before/after comparison of the report is in a comment on this PR.

@omkargaikwad23

omkargaikwad23 commented Nov 21, 2026 •

Copy link
Copy Markdown
Collaborator Author

Verification

I called config_to_df on main (6ca77dc) and on this branch with the same config. All secret values are fake.

Config key main This PR
password in the db_configs list FAKE-DB-PASSWORD [REDACTED]
connection_string in the db_configs list mongodb://u:FAKE-URL-PASSWORD@host/db mongodb://u:[REDACTED]@host/db
model_config.env.GOOGLE_API_KEY FAKE-API-KEY [REDACTED]
model_config.setup.mcp_servers.s.headers.Authorization Bearer FAKE-TOKEN [REDACTED]
model_config.env.GOOGLE_CLOUD_PROJECT p p
model_config.max_tokens 1024 1024
model_config.openai_api_key_secret projects/p/secrets/s/versions/1 projects/p/secrets/s/versions/1
experiment_config.orchestrator agent agent

Fake secret values in the report: 4 on main, 0 on this PR.

Local tests:

  • config_redaction_test.py, agent_version_reporting_test.py, and evalbench_test.py: 19 passed.
  • pytest evalbench/test with SKIP_CLOUD_TESTS=true: 1302 passed, 11 skipped.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant