Creating a dataframe from no data seems inconsistent

[LostKobrakai]

I would've expected both of those options to resolve to the latter result.

Explorer.DataFrame.new(
  [],
  dtypes: %{
    product_family: :category
  }
)

# #Explorer.DataFrame<
#   Polars[0 x 0]
# >

Explorer.DataFrame.new(%{
  product_family: Explorer.Series.from_list([], dtype: :category)
})

# #Explorer.DataFrame<
#   Polars[0 x 1]
#   product_family category []
# >

[billylanchantin]

@LostKobrakai Categorical data has caused us many problems so I bet that's the cause. Polars recently reworked the underlying logic for the category dtype. I'm in the middle of upgrading the Polars dependency (which had many breaking changes, unfortunately). Hopefully this will be resolved once that upgrade is complete.

[billylanchantin]

Coming back to this after upgrading Polars to 0.52, the problem remains:

iex(1)> Explorer.DataFrame.new([], dtypes: %{product_family: :category})
#Explorer.DataFrame<
  Polars[0 x 0]
>
iex(2)> Explorer.DataFrame.new(%{product_family: Explorer.Series.from_list([], dtype: :category)})
#Explorer.DataFrame<
  Polars[0 x 1]
  product_family category []
>

So this is on us and not a result of category weirdness. In fact, what I should've done originally is check if this was true for another dtype:

iex(3)> Explorer.DataFrame.new([], dtypes: %{product_family: :f64})
#Explorer.DataFrame<
  Polars[0 x 0]
>
iex(4)> Explorer.DataFrame.new(%{product_family: Explorer.Series.from_list([], dtype: :f64)})
#Explorer.DataFrame<
  Polars[0 x 1]
  product_family f64 []
>

It clearly is, so DataFrame.new is simply inconsistent.