fix: update duplicates_pandas.py (#1427)

boris-kogan · aquemy · commit 56a664197c6e · 2023-10-10T12:25:14.000+02:00
Fixing Bug Report #1384 Dataset with categorical features causes memory error even on tiny dataset.
diff --git a/src/ydata_profiling/model/pandas/duplicates_pandas.py b/src/ydata_profiling/model/pandas/duplicates_pandas.py
@@ -35,7 +35,7 @@ def pandas_get_duplicates(
             duplicated_rows = df.duplicated(subset=supported_columns, keep=False)
             duplicated_rows = (
                 df[duplicated_rows]
-                .groupby(supported_columns, dropna=False)
+                .groupby(supported_columns, dropna=False, observed=True)
                 .size()
                 .reset_index(name=duplicates_key)
             )

Original file line number	Diff line number	Diff line change
`@@ -35,7 +35,7 @@ def pandas_get_duplicates(`
`35`	`35`	`duplicated_rows = df.duplicated(subset=supported_columns, keep=False)`
`36`	`36`	`duplicated_rows = (`
`37`	`37`	`df[duplicated_rows]`
`38`		`- .groupby(supported_columns, dropna=False)`
	`38`	`+ .groupby(supported_columns, dropna=False, observed=True)`
`39`	`39`	`.size()`
`40`	`40`	`.reset_index(name=duplicates_key)`
`41`	`41`	`)`