Skip to content

feat: allow configuring datafusion-cli history file location#23797

Open
dariocurr wants to merge 1 commit into
apache:mainfrom
dariocurr:cli/configurable-history-file-location
Open

feat: allow configuring datafusion-cli history file location#23797
dariocurr wants to merge 1 commit into
apache:mainfrom
dariocurr:cli/configurable-history-file-location

Conversation

@dariocurr

@dariocurr dariocurr commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

datafusion-cli's interactive REPL hardcodes its rustyline history file as the relative path .history. Because the path is relative, the history file is created (and looked up) in whatever directory the CLI happens to be launched from, so every working directory silently accumulates its own hidden .history file with no way to point it somewhere else.

What changes are included in this PR?

  • Add a --history-file <PATH> CLI flag to datafusion-cli, validated at parse time (rejects a path that is an existing directory, or whose parent directory doesn't exist).
  • Add a DATAFUSION_HISTORY_FILE environment variable as an alternative way to set it (e.g. for scripted/containerized setups). The --history-file flag takes precedence over the environment variable.
  • The existing default behavior (.history, relative to the current directory) is unchanged when neither is set, so this is non-breaking.
  • exec_from_repl now takes history_file: Option<&Path> instead of hardcoding .history twice; the .history default now lives in one place inside exec_from_repl itself.
  • Updated datafusion-cli/examples/cli-session-context.rs (the only other caller of exec_from_repl) and docs/source/user-guide/cli/usage.md accordingly.

Are these changes tested?

This is a small, mostly mechanical CLI argument-plumbing change with no new branching logic worth unit-testing in isolation (the added validator mirrors the existing parse_valid_file/parse_valid_data_dir helpers, which aren't separately unit-tested either). I manually verified end-to-end via the built binary:

  • Default (.history in cwd, unchanged behavior)
  • --history-file <path> override
  • DATAFUSION_HISTORY_FILE=<path> override
  • --history-file taking precedence over DATAFUSION_HISTORY_FILE when both are set
  • --history-file pointing at a non-existent parent directory fails fast with a clear error instead of a late rustyline error

Existing datafusion-cli unit tests pass (cargo test -p datafusion-cli --lib --bins).

Are there any user-facing changes?

Yes: a new --history-file CLI flag and DATAFUSION_HISTORY_FILE environment variable, documented in docs/source/user-guide/cli/usage.md. No breaking changes; default behavior is unchanged.

datafusion-cli hardcoded the interactive shell's rustyline history as
the relative path `.history`, so every launch directory silently got
its own history file with no way to point it elsewhere.

Add a `--history-file <PATH>` flag and a `DATAFUSION_HISTORY_FILE`
environment variable (flag takes precedence), keeping `.history` as
the default so existing behavior is unchanged.

Closes apache#23796.
@github-actions github-actions Bot added the documentation Improvements or additions to documentation label Jul 22, 2026
@github-actions

Copy link
Copy Markdown

Thank you for opening this pull request!

Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).

Details
     Cloning apache/main
    Building datafusion-cli v54.1.0 (current)
       Built [ 198.130s] (current)
     Parsing datafusion-cli v54.1.0 (current)
      Parsed [   0.035s] (current)
    Building datafusion-cli v54.1.0 (baseline)
       Built [ 190.643s] (baseline)
     Parsing datafusion-cli v54.1.0 (baseline)
      Parsed [   0.033s] (baseline)
    Checking datafusion-cli v54.1.0 -> v54.1.0 (no change; assume patch)
     Checked [   0.120s] 223 checks: 222 pass, 1 fail, 0 warn, 30 skip

--- failure function_parameter_count_changed: pub fn parameter count changed ---

Description:
A publicly-visible function now takes a different number of parameters.
        ref: https://doc.rust-lang.org/cargo/reference/semver.html#fn-change-arity
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.49.0/src/lints/function_parameter_count_changed.ron

Failed in:
  datafusion_cli::exec::exec_from_repl now takes 3 parameters instead of 2, in /home/runner/work/datafusion/datafusion/datafusion-cli/src/exec.rs:130

     Summary semver requires new major version: 1 major and 0 minor checks failed
    Finished [ 392.162s] datafusion-cli

@github-actions github-actions Bot added the auto detected api change Auto detected API change label Jul 22, 2026
@dariocurr

Copy link
Copy Markdown
Contributor Author

Re the cargo-semver-checks note above: this is an expected, intentional minor breaking change. exec_from_repl gained a required history_file: Option<&Path> parameter so the interactive shell's history location can be configured (this PR's whole purpose); the only in-tree caller besides main.rs is datafusion-cli/examples/cli-session-context.rs, which has been updated accordingly.

Per the API health policy, could a committer add the api change label to this PR? I don't have push access to add it myself.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

auto detected api change Auto detected API change documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

datafusion-cli: allow configuring the location of the interactive shell's history file

1 participant