Skip to content

fix(server): limit the custom classes that the eval server loads - #734

Draft
omkargaikwad23 wants to merge 1 commit into
mainfrom
fix/server-custom-class-allowlist
Draft

omkargaikwad23 wants to merge 1 commit into
mainfrom
fix/server-custom-class-allowlist

Conversation

@omkargaikwad23

Copy link
Copy Markdown
Collaborator

What this fixes

A model config can set generator_class, and a database config can set connector_class. EvalBench imports the class and calls it with the config. On the eval server, the config comes from the client. Thus a client could make the server call any importable class, for example subprocess:Popen.

How

  • On the eval server (eval_server.py), load_custom_class loads a class only from the modules in EVALBENCH_CUSTOM_CLASS_MODULES (comma-separated). A module in the list also permits its submodules. The default list is empty.
  • For a different module, load_custom_class raises PermissionError before the import.
  • A local run does not change.

Behavior change

On the eval server, generator_class and connector_class fail unless the server permits the module with EVALBENCH_CUSTOM_CLASS_MODULES.

Limits

  • The check uses the module name. A permitted module must not expose an unsafe class.
  • The scorers that run files from a workspace or a script path are not changed.

Testing

  • New tests in class_loader_server_test.py.
  • A before/after comparison is in a comment on this PR.

@omkargaikwad23

omkargaikwad23 commented Nov 21, 2026 •

Copy link
Copy Markdown
Collaborator Author

Verification

I called get_generator with the same model config on main (6ca77dc) and on this branch. For subprocess:Popen, the config makes the command touch <marker file> in a temporary directory.

# Process generator_class EVALBENCH_CUSTOM_CLASS_MODULES main This PR
1 eval_server.py subprocess:Popen not set Loaded, the command ran PermissionError, no command
2 eval_server.py collections:OrderedDict not set Loaded PermissionError
3 eval_server.py collections:OrderedDict collections Loaded Loaded
4 eval_server.py subprocess:Popen collections Loaded, the command ran PermissionError, no command
5 evalbench.py collections:OrderedDict not set Loaded Loaded

Local tests:

  • class_loader_server_test.py and spi_custom_class_loader_test.py: 26 passed.
  • pytest evalbench/test with SKIP_CLOUD_TESTS=true: 1302 passed, 11 skipped.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant