(cli-configuration)=

Configuration

BenchBox supports configuration through multiple sources with the following precedence:

Command-line arguments (highest priority)
Environment variables
Configuration files
Default values (lowest priority)

Configuration File Format

BenchBox uses YAML configuration files. Default location: ~/.benchbox/config.yaml

Example configuration:

# Output settings
output:
  compression:
    enabled: true
    type: zstd
    level: 3
  formats:
    - json
    - csv

# Platform settings
platforms:
  databricks:
    enabled: true
    warehouse_id: "abc123"

  bigquery:
    enabled: true
    project_id: "my-project"

# Tuning settings
tuning:
  default_mode: notuning
  enable_constraints: false

Platform Configuration

Each platform can have specific configuration options:

Databricks:

platforms:
  databricks:
    warehouse_id: "warehouse-id"
    catalog: "main"
    schema: "benchbox"

BigQuery:

platforms:
  bigquery:
    project_id: "my-project"
    dataset: "benchbox"
    location: "US"

Snowflake:

platforms:
  snowflake:
    account: "account-name"
    warehouse: "COMPUTE_WH"
    database: "BENCHBOX"
    schema: "PUBLIC"

(cli-env-vars)=

Environment Variables

BenchBox recognizes these environment variables:

Core Settings

These variables override the corresponding settings in the configuration file (equivalent to editing ~/.benchbox/config.yaml):

Variable	Config Path	Type	Default	Description
`BENCHBOX_DATABASE_PREFERRED`	`database.preferred`	string	`duckdb`	Preferred database platform
`BENCHBOX_SCALE_FACTOR`	`benchmarks.default_scale`	float	`0.01`	Default scale factor
`BENCHBOX_VERBOSE`	`execution.verbose`	boolean	`true`	Enable verbose output (`true`/`1`/`yes`/`on`)
`BENCHBOX_MAX_WORKERS`	`execution.max_workers`	integer	`4`	Maximum parallel worker threads
`BENCHBOX_TUNING_ENABLED`	`tuning.enabled`	boolean	`false`	Enable table tuning by default
`BENCHBOX_TUNING_CONFIG`	`tuning.default_config_file`	string	-	Path to tuning configuration file
`BENCHBOX_OUTPUT_DIR`	`output.directory`	string	`./benchmark_runs/results`	Output directory for result files
`BENCHBOX_MEMORY_LIMIT_GB`	`execution.memory_limit_gb`	integer	`0` (auto)	Memory limit in GB; `0` means no limit

Boolean parsing: true, 1, yes, on (case-insensitive) are treated as true.

General Settings

BENCHBOX_NON_INTERACTIVE=true: Enable non-interactive mode
BENCHBOX_NO_COMPRESSION=true: Disable data compression
BENCHBOX_CONFIG_PATH=/path/to/config.yaml: Custom config file location

Advanced Settings

These variables control lower-level behaviors, useful for CI, offline environments, or advanced workflows:

Variable	Description
`BENCHBOX_DATA_DIR`	Override the default local data directory for generated benchmark files
`BENCHBOX_CACHE_DIR`	Override the cache directory for DataFrame benchmark data
`BENCHBOX_TUNING_PATH`	Override the tuning file search root; BenchBox looks for `{platform}/{benchmark}_tuned.yaml` under this directory
`BENCHBOX_QUERY_VALIDATION_MODE`	TPC-DS query validation mode: `exact`, `loose`, `range`, or `skip` (default: `skip`)
`BENCHBOX_NO_DOWNLOAD`	Set to `1` or `true` to disable automatic download of expected-answer files; useful in offline/airgapped environments
`BENCHBOX_ANSWERS_URL`	Override the base URL for downloading expected-answer archives (default: GitHub releases)
`BENCHBOX_ENABLE_EXPERIMENTAL`	Set to `1` or `true` to show experimental platforms in the platform registry
`BENCHBOX_DATA_ORG_MAX_IN_MEMORY_BYTES`	Maximum bytes held in memory when sorting generated data files (integer; default: 512 MB)

Platform Authentication

Databricks:

DATABRICKS_TOKEN: Authentication token
DATABRICKS_HOST: Workspace URL

BigQuery:

GOOGLE_APPLICATION_CREDENTIALS: Service account key file path

Snowflake:

SNOWFLAKE_USER: Username
SNOWFLAKE_PASSWORD: Password
SNOWFLAKE_ACCOUNT: Account identifier

Redshift:

AWS_ACCESS_KEY_ID: AWS access key
AWS_SECRET_ACCESS_KEY: AWS secret key

ClickHouse:

CLICKHOUSE_HOST: Server hostname
CLICKHOUSE_USER: Username
CLICKHOUSE_PASSWORD: Password

(cli-platform-options)=

Platform-Specific Options

Each platform supports specific options via --platform-option KEY=VALUE:

Universal Keys (All Platforms)

These keys are available for every platform:

Key	Description
`driver_version`	Pin the Python driver package to a specific version (e.g. `1.2.0`). Useful for reproducing results or testing a specific connector release.
`driver_auto_install`	When `true`, automatically installs the requested `driver_version` via uv if the package is not already present.

`uv run benchbox run` syncs the environment to `uv.lock` before Python starts, which
can silently revert a version you installed manually. Use `driver_version` +
`driver_auto_install=true` or `uv run --with "pkg==X"` to reliably test a specific
version. See {ref}`driver-version-management` for the full guide.

Example: pin DuckDB driver and auto-install it:

benchbox run --platform duckdb --benchmark tpch \
  --platform-option driver_version=1.2.0 \
  --platform-option driver_auto_install=true

Example: pin Snowflake connector version:

benchbox run --platform snowflake --benchmark tpch \
  --platform-option driver_version=3.12.0 \
  --platform-option driver_auto_install=true \
  --platform-option account=xy12345.us-east-1 \
  --platform-option warehouse=COMPUTE_WH

Athena Spark Engine Version

For Athena Spark only, the Spark engine version can be explicitly selected:

Key	Description
`engine_version`	Spark engine version string (e.g. `PySpark engine version 3`). Defaults to the workgroup's configured version. Athena Spark only - on other cloud platforms the engine version is auto-detected.

benchbox run --platform athena-spark --benchmark tpch \
  --platform-option workgroup=my-spark-workgroup \
  --platform-option s3_staging_dir=s3://my-bucket/benchbox \
  --platform-option "engine_version=PySpark engine version 3"

ClickHouse Options

mode=local: Use local ClickHouse instance
secure=true: Enable TLS encryption
port=9000: Custom port number
database=default: Target database name

Example:

benchbox run --platform clickhouse --benchmark tpch \
  --platform-option mode=local \
  --platform-option secure=true \
  --platform-option port=9440

View Platform Details

Use benchbox platforms status to see platform information and capabilities:

benchbox platforms status clickhouse
benchbox platforms status databricks

(cli-config-sections)=

Configuration Sections Reference

Complete reference for all settings in ~/.benchbox/config.yaml (or ./benchbox.yaml). Unset values fall back to the defaults shown here.

`system`

System profiling and detection settings.

Setting	Type	Default	Description
`auto_profile`	boolean	`true`	Automatically detect system capabilities on startup
`save_profile`	boolean	`true`	Persist the detected system profile to disk
`profile_cache_hours`	integer	`24`	Hours to cache the system profile before re-detecting

`database`

Database connection settings.

Setting	Type	Default	Description
`preferred`	string	`duckdb`	Default platform when none is specified on the CLI
`connection_timeout`	integer	`30`	Connection timeout in seconds
`auto_detect`	boolean	`true`	Automatically detect available database platforms

`benchmarks`

Default benchmark execution parameters.

Setting	Type	Default	Description
`default_scale`	float	`0.01`	Default scale factor
`timeout_minutes`	integer	`60`	Maximum benchmark execution time
`max_memory_gb`	integer	`8`	Maximum memory allocation hint in GB
`continue_on_error`	boolean	`false`	Continue running remaining queries when one fails

`output`

Result output and export settings.

Setting	Type	Default	Description
`formats`	list	`[json, console]`	Output format list
`directory`	string	`./benchmark_runs/results`	Results directory
`timestamp_format`	string	`%Y%m%d_%H%M%S`	Timestamp format for result filenames
`submit_to_service`	boolean	`false`	Legacy placeholder; use `benchbox submit --service` for hosted upload
`service_url`	string	`https://api.benchbox.dev/v1`	Default hosted results service URL placeholder
`compression.enabled`	boolean	`true`	Enable result file compression
`compression.type`	string	`zstd`	Compression algorithm (`zstd`, `gzip`)
`compression.level`	integer	`null`	Compression level; `null` uses the algorithm default

`execution`

Performance and execution control settings.

Setting	Type	Default	Description
`parallel_queries`	boolean	`false`	Execute queries in parallel
`max_workers`	integer	`4`	Maximum worker threads for parallel execution
`memory_limit_gb`	integer	`0`	Memory cap in GB; `0` means no limit (auto)
`verbose`	boolean	`true`	Enable verbose progress output

execution.power_run - settings for multi-iteration power runs:

Setting	Type	Default	Description
`iterations`	integer	`3`	Number of measurement iterations per query
`warm_up_iterations`	integer	`1`	Warm-up iterations before measurement (not included in stats)
`timeout_per_iteration_minutes`	integer	`60`	Timeout for each iteration
`fail_fast`	boolean	`false`	Stop immediately when any iteration fails
`collect_metrics`	boolean	`true`	Collect resource metrics during execution

execution.concurrent_queries - settings for throughput / concurrent-stream runs:

Setting	Type	Default	Description
`enabled`	boolean	`false`	Enable concurrent query streams
`max_concurrent`	integer	`2`	Number of concurrent query streams
`query_timeout_seconds`	integer	`300`	Per-query timeout in concurrent mode
`stream_timeout_seconds`	integer	`3600`	Timeout for an entire concurrent stream
`retry_failed_queries`	boolean	`true`	Retry failed queries in concurrent streams
`max_retries`	integer	`3`	Maximum retry attempts per query

`tuning`

Table tuning and optimization settings.

Setting	Type	Default	Description
`enabled`	boolean	`false`	Apply tuning configurations by default
`default_config_file`	string	`null`	Path to a default tuning YAML file
`validate_on_load`	boolean	`true`	Validate tuning configurations when loaded
`allow_platform_incompatible`	boolean	`false`	Allow tuning configs that contain platform-incompatible directives

Example full configuration file:

system:
  profile_cache_hours: 48

database:
  preferred: duckdb

benchmarks:
  default_scale: 1.0
  timeout_minutes: 120
  continue_on_error: true

output:
  directory: ./results
  compression:
    enabled: true
    type: zstd

execution:
  max_workers: 8
  memory_limit_gb: 16
  power_run:
    iterations: 5
    warm_up_iterations: 2

tuning:
  enabled: true
  default_config_file: ./tuning/my_tuning.yaml

Run Command - Platform and tuning options during execution
Platforms Command - Platform management and setup
Platform Documentation - Detailed platform guides

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Configuration

Configuration File Format

Platform Configuration

Environment Variables

Core Settings

General Settings

Advanced Settings

Platform Authentication

Platform-Specific Options

Universal Keys (All Platforms)

Athena Spark Engine Version

ClickHouse Options

View Platform Details

Configuration Sections Reference

`system`

`database`

`benchmarks`

`output`

`execution`

`tuning`

Related

FilesExpand file tree

configuration.md

Latest commit

History

configuration.md

File metadata and controls

Configuration

Configuration File Format

Platform Configuration

Environment Variables

Core Settings

General Settings

Advanced Settings

Platform Authentication

Platform-Specific Options

Universal Keys (All Platforms)

Athena Spark Engine Version

ClickHouse Options

View Platform Details

Configuration Sections Reference

system

database

benchmarks

output

execution

tuning

Related

`system`

`database`

`benchmarks`

`output`

`execution`

`tuning`