Add an extension point to allow SQL based registration of external catalogs

[pepijnve]

Which issue does this PR close?

• Closes Add support for SQL based registration of external catalogs #25111.

Rationale for this change

Adding a CatalogProviderFactory trait, similar to TableProviderFactory, makes it possible for extensions to register support for external catalogs that can then be used by consumers without requiring additional code. This is quite convenient when working with open table formats via catalogs.

What changes are included in this PR?

• Adds a CatalogProviderFactory trait similar to TableProviderFactory
• Adds CREATE EXTERNAL CATALOG and DROP CATALOG support

What is the testing strategy for this PR?

• Covered by unit and integration tests

Are there any user-facing changes?

Yes, the supported DDL has been extended. Documentation has been updated to reflect this.

[github-actions]

Thank you for opening this pull request!

Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).

Details

     Cloning apache/main
    Building datafusion v55.0.0 (current)
       Built [  64.307s] (current)
     Parsing datafusion v55.0.0 (current)
      Parsed [   0.036s] (current)
    Building datafusion v55.0.0 (baseline)
       Built [  65.551s] (baseline)
     Parsing datafusion v55.0.0 (baseline)
      Parsed [   0.037s] (baseline)
    Checking datafusion v55.0.0 -> v55.0.0 (no change; assume patch)
     Checked [   0.625s] 223 checks: 223 pass, 31 skip
     Summary no semver update required
    Finished [ 132.137s] datafusion
    Building datafusion-catalog v55.0.0 (current)
       Built [  42.739s] (current)
     Parsing datafusion-catalog v55.0.0 (current)
      Parsed [   0.026s] (current)
    Building datafusion-catalog v55.0.0 (baseline)
       Built [  41.137s] (baseline)
     Parsing datafusion-catalog v55.0.0 (baseline)
      Parsed [   0.024s] (baseline)
    Checking datafusion-catalog v55.0.0 -> v55.0.0 (no change; assume patch)
     Checked [   0.116s] 223 checks: 223 pass, 31 skip
     Summary no semver update required
    Finished [  85.032s] datafusion-catalog
    Building datafusion-cli v55.0.0 (current)
       Built [  96.765s] (current)
     Parsing datafusion-cli v55.0.0 (current)
      Parsed [   0.033s] (current)
    Building datafusion-cli v55.0.0 (baseline)
       Built [  96.756s] (baseline)
     Parsing datafusion-cli v55.0.0 (baseline)
      Parsed [   0.035s] (baseline)
    Checking datafusion-cli v55.0.0 -> v55.0.0 (no change; assume patch)
     Checked [   0.123s] 223 checks: 223 pass, 31 skip
     Summary no semver update required
    Finished [ 196.330s] datafusion-cli
    Building datafusion-expr v55.0.0 (current)
       Built [  29.764s] (current)
     Parsing datafusion-expr v55.0.0 (current)
      Parsed [   0.078s] (current)
    Building datafusion-expr v55.0.0 (baseline)
       Built [  29.548s] (baseline)
     Parsing datafusion-expr v55.0.0 (baseline)
      Parsed [   0.078s] (baseline)
    Checking datafusion-expr v55.0.0 -> v55.0.0 (no change; assume patch)
     Checked [   1.242s] 223 checks: 221 pass, 1 fail, 1 warn, 31 skip

--- failure enum_variant_added: enum variant added on exhaustive enum ---

Description:
A publicly-visible enum without #[non_exhaustive] has a new variant.
        ref: https://doc.rust-lang.org/cargo/reference/semver.html#enum-variant-new
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.50.0/src/lints/enum_variant_added.ron

Failed in:
  variant DdlStatement:CreateExternalCatalog in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/ddl.rs:54
  variant DdlStatement:DropCatalog in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/ddl.rs:64
  variant DdlStatement:CreateExternalCatalog in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/ddl.rs:54
  variant DdlStatement:DropCatalog in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/ddl.rs:64

--- warning partial_ord_enum_variants_reordered: enum variants reordered in #[derive(PartialOrd)] enum ---

Description:
A public enum that derives PartialOrd had its variants reordered. #[derive(PartialOrd)] uses the enum variant order to set the enum's ordering behavior, so this change may break downstream code that relies on the previous order.
        ref: https://doc.rust-lang.org/std/cmp/trait.PartialOrd.html#derivable
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.50.0/src/lints/partial_ord_enum_variants_reordered.ron

Failed in:
  DdlStatement::CreateIndex moved from position 6 to 7, in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/ddl.rs:56
  DdlStatement::DropTable moved from position 7 to 8, in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/ddl.rs:58
  DdlStatement::DropView moved from position 8 to 9, in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/ddl.rs:60
  DdlStatement::DropCatalogSchema moved from position 9 to 10, in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/ddl.rs:62
  DdlStatement::CreateFunction moved from position 10 to 12, in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/ddl.rs:67
  DdlStatement::DropFunction moved from position 11 to 13, in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/ddl.rs:69
  DdlStatement::CreateIndex moved from position 6 to 7, in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/ddl.rs:56
  DdlStatement::DropTable moved from position 7 to 8, in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/ddl.rs:58
  DdlStatement::DropView moved from position 8 to 9, in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/ddl.rs:60
  DdlStatement::DropCatalogSchema moved from position 9 to 10, in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/ddl.rs:62
  DdlStatement::CreateFunction moved from position 10 to 12, in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/ddl.rs:67
  DdlStatement::DropFunction moved from position 11 to 13, in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/ddl.rs:69

     Summary semver requires new major version: 1 major and 0 minor checks failed
     Warning produced 1 major and 0 minor level warnings
    Finished [  61.769s] datafusion-expr
    Building datafusion-optimizer v55.0.0 (current)
       Built [  27.893s] (current)
     Parsing datafusion-optimizer v55.0.0 (current)
      Parsed [   0.033s] (current)
    Building datafusion-optimizer v55.0.0 (baseline)
       Built [  28.106s] (baseline)
     Parsing datafusion-optimizer v55.0.0 (baseline)
      Parsed [   0.032s] (baseline)
    Checking datafusion-optimizer v55.0.0 -> v55.0.0 (no change; assume patch)
     Checked [   0.163s] 223 checks: 223 pass, 31 skip
     Summary no semver update required
    Finished [  56.982s] datafusion-optimizer
    Building datafusion-proto v55.0.0 (current)
       Built [  56.755s] (current)
     Parsing datafusion-proto v55.0.0 (current)
      Parsed [   0.017s] (current)
    Building datafusion-proto v55.0.0 (baseline)
       Built [  57.040s] (baseline)
     Parsing datafusion-proto v55.0.0 (baseline)
      Parsed [   0.018s] (baseline)
    Checking datafusion-proto v55.0.0 -> v55.0.0 (no change; assume patch)
     Checked [   0.123s] 223 checks: 223 pass, 31 skip
     Summary no semver update required
    Finished [ 115.094s] datafusion-proto
    Building datafusion-session v55.0.0 (current)
       Built [  39.597s] (current)
     Parsing datafusion-session v55.0.0 (current)
      Parsed [   0.011s] (current)
    Building datafusion-session v55.0.0 (baseline)
       Built [  38.520s] (baseline)
     Parsing datafusion-session v55.0.0 (baseline)
      Parsed [   0.011s] (baseline)
    Checking datafusion-session v55.0.0 -> v55.0.0 (no change; assume patch)
     Checked [   0.183s] 223 checks: 223 pass, 31 skip
     Summary no semver update required
    Finished [  79.262s] datafusion-session
    Building datafusion-sql v55.0.0 (current)
       Built [  43.355s] (current)
     Parsing datafusion-sql v55.0.0 (current)
      Parsed [   0.035s] (current)
    Building datafusion-sql v55.0.0 (baseline)
       Built [  42.800s] (baseline)
     Parsing datafusion-sql v55.0.0 (baseline)
      Parsed [   0.033s] (baseline)
    Checking datafusion-sql v55.0.0 -> v55.0.0 (no change; assume patch)
     Checked [   0.224s] 223 checks: 222 pass, 1 fail, 0 warn, 31 skip

--- failure enum_variant_added: enum variant added on exhaustive enum ---

Description:
A publicly-visible enum without #[non_exhaustive] has a new variant.
        ref: https://doc.rust-lang.org/cargo/reference/semver.html#enum-variant-new
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.50.0/src/lints/enum_variant_added.ron

Failed in:
  variant Statement:CreateExternalCatalog in /home/runner/work/datafusion/datafusion/datafusion/sql/src/parser.rs:405
  variant Statement:DropCatalog in /home/runner/work/datafusion/datafusion/datafusion/sql/src/parser.rs:407

     Summary semver requires new major version: 1 major and 0 minor checks failed
    Finished [  87.439s] datafusion-sql