plan to improve pandas completeness

[MarcoGorelli]

If we look at the most referenced partially unknown classes in pandas, we have:

• Series
• Index
• DataFrame
• ExtensionArray
• InternalIndex
• ...

Many of these depend on each other:

• Series.to_frame() returns DataFrame
• DataFrame.__getitem__ returns a union including Series
• Index.to_series() returns Series
• IntervalIndex.__ge__ returns Series
• ExtensionArray.to_numpy() has a parameter na_type, whose type includes Union["Period", "Timestamp", "Timedelta", "Interval"]
• Period.__sub__ returns Series
• Series.groupby uses IntervalIndex, PeriodIndex, and others in overloads, so those need typing as a precursor to Series appearing as typed

Completed:

ExtensionArray [done!]

My plan for now is to make sure everything in ExtensionArray is typed. What's missing:

• astype
  • why do other different astype methods type dtype differently? There's: AstypeArg, Dtype, and DtypeArg. DType is a dtype object, like Int64Dtype. DtypeArg is a mapping from str to DType. AstypeArg is anything that can be converted to a dtype
• fillna
• dropna
• unique
• repeat
  • why does upstream pandas use AxisInt?
  • make typealias for "arraylike of ints"
• take
  • why does this use fill_value: Any whereas shift uses fill_value: object?
• view
  • overloads based on dtype
• ravel
  • make order typealias
• tolist: return value is partially unknown (just list)
• searchsorted. requires overloads
  • why does pd.Index use _ListLike? check divergence, check if mapping is really valid here

DatetimeIndex

• __reduce__ (can probably be removed?)
• snap
• slice_indexer (can probably be removed, just use the parent method?)
• indexer_at_time
• indexer_between_time

BaseGroupBy

• get_group

RangeIndex

RangeIndex

• max
• min
• from_range
• __reduce__
• get_indexer
• tolist
• argsort
• factorize
• equals
• join
• __floordiv__

IntervalIndex

• get_indexer

CategoricalIndex

• __new__
  • how to type "index-like"?
• equals (just inherit)
• values
• __contains__
• unique
• reindex
  • type in base class
• get_indexer
• get_indexer_non_unique
• delete
  • type in parent class
• insert
  • type in parent class

Series

Series

• view
• array_ufunc
• __setitem__
  • combine what's in _LocIndexer.__setitem__ and _ILocIndexer.__setitem__?
• mode
• first
• last
• where
  • just missing other
• set_axis
  • just missing labels. can probably the same as the data annotation in Index?

DataFrame

• __new__
• __dataframe__ (but this should probably be deprecated so perhaps hold off on this one for now)
• __rmatmul__
• __setitem__ (same trick as for Series.__setitem__)
• pivot_table
  • which aggfunc to use? it's documented to be function, list of functions, dict, default "mean", and type aliases are:
    • AggFuncTypeBase: function
    • AggFuncTypeDictSeries: {col_name: function}
    • AggFuncTypeDictFrame: {col_name: function | list[function]}
    • AggFuncTypeSeriesToFrame: list[function] | {col_name: function}
    • AggFuncTypeFrame: function | list[function] | {col_name: function | list[function]}
    • AggFuncTypeDict: {col_name: function | list[function]}
    • AggFuncType: function | list[function] | {col_name: function | list[function]}
• hist
• boxplot (frustratingly complicated overloads)
  • check if existing one can be removed
  • add tests
• radd, rsub, eq, gt, lt, le, lt, rdiv, rfloordiv, rmod, rmul, rpow, rtruediv (potentially frustrating)
• asof
• asfreq
• first (deprecated)
• last (deprecated)
• set_axis
• where (align with Series once that's merged)

Index

Index:

• is_
• __array_wrap__
• ravel
• view (frustrating, do this last)
• take
• [z] repeat
• __copy__
• to_flat_index
• to_frame
• set_names
• rename
• drop_level
• __reduce__
• isna
• notna
• fillna
• memory_usage
• __setitem__
• putmask
• identical
• asof
• asof_locs
• sort_values
• get_indexer_non_unique
• get_indexer_for
• map
• isin (do this separately)
• slice_indexer
• get_slice_bound
• slice_locs

In progress

• DatetimeArray
• DatetimeLikeArrayMixin
• TimelikeOps
• DatelikeOps
• PlotAccessor

IntegerArray / BaseMaskedArray: simple-ish but doesn't move metric

[MarcoGorelli]

Assuming the latest PRs go through:

known:

• ExtensionDtype
• DatetimeArray
• CategoricalDtype
• IntervalMixin
• IntervalDtype

why is Index still unknown?

missing:

• IntervalArray, which requires Interval. Some of the Interval are missing the generic type
• Timedelta subclasses _TimedeltaPropertiesNoRounding which has components which returns DataFrame, which is partially unknown
• ExtensionIndex: missing generic for Interval somewhere. Also, subclasses Index
• StringMethods
• BaseMaskedArray.__iter__ missing generic for Iterator
• ArrayDescriptor missing some bits
• NDFrame: a few missing bits. conn in to_sql - is the offender sqlalchemy or sqlite?
• ReplaceValue needs generic type for Pattern

strategy idea:

• try making all *Index classes type-known (deleting methods if necessary, temporarily adding some Any)
• same for Array classes
• same for Period, Datetime, Timedelta, NAType
• find out what's then missing in Series / Index / DataFrame
• check Index next

crucial bits:

• CombinedDatetimelikeProperties: _PeriodProperties generics

[MarcoGorelli]

concrete steps to take:

• type generic of _IsLeapYearProperty in PeriodProperties
• type generics of _PeriodProperties in CombinedDatetimelikeProperties
• type Pattern generic in ReplaceValue
• type generic Iterator in BaseMaskedArray.__iter__
• type Interval generics, or use default in typevar
• type Sequence generic in DataFrame.rtruediv
• ???

[MarcoGorelli]

right so what's missing:

• DatetimeProperties has a Dataframe return type
• Categorical is unknown

idea:

• make Categorical artificially known, and see what happens next

• CategoricalAccessor still unknown

TimedeltaIndex: unknown

NDFrame.to_sql still problematic