Skip to content

Fix NewRowSynthesis failing with category data type - #937

Open
cpruijsen wants to merge 1 commit into
sdv-dev:mainfrom
cpruijsen:fix/issue-397
Open

cpruijsen wants to merge 1 commit into
sdv-dev:mainfrom
cpruijsen:fix/issue-397

Conversation

@cpruijsen

Copy link
Copy Markdown

Quote string categorical values in NewRowSynthesis query filters with repr(value).

Resolves #397.

NewRowSynthesis built DataFrame.query filters by checking
real_data[field].dtype == 'O'. Pandas category and string dtypes
are not 'O', so a string value such as F was interpolated unquoted
and pandas treated it as a variable name (UndefinedVariableError).

The same metric is what DiagnosticReport uses for the Synthesis
property.

Decision

Quote with repr(value) when the categorical value is a str,
otherwise interpolate the raw value. Closed PR #458 always used
repr(row[field]). iterrows() yields numpy scalars for integer and
boolean categoricals, and on current numpy repr(np.int64(1)) is
np.int64(1), which pandas.query cannot parse, while
f'{np.int64(1)}' is 1. Can switch to always-repr after .item(),
or to passing values through local_dict.

Test plan

  • test_compute_breakdown_with_category_dtype fails on main with
    UndefinedVariableError: name 'F' is not defined and passes here
  • test_compute_breakdown_with_string_dtype likewise
  • existing tests/unit/single_table/test_new_row_synthesis.py still
    pass (integer categoricals and multi-line strings)

@cpruijsen
cpruijsen requested a review from a team as a code owner September 11, 2026 07:38
@cpruijsen
cpruijsen requested review from sarahmish and removed request for a team September 11, 2026 07:38
@cpruijsen
cpruijsen force-pushed the fix/issue-397 branch 2 times, most recently from de9ed62 to e402bb7 Compare September 18, 2026 19:23
Quote categorical values for pandas.query based on the value type so
string categories are not treated as variable names.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

NewRowSynthesis fails with category data type

1 participant