Skip to content

fix(preprocessor): handle pandas categorical missing values - #74

Merged
mkumar73 merged 3 commits into
OpenTabular:mainfrom
Boulea7:ln-cx/pretab-pandas-missing-values
Oct 31, 2026
Merged

mkumar73 merged 3 commits into
OpenTabular:mainfrom
Boulea7:ln-cx/pretab-pandas-missing-values

Conversation

@Boulea7

@Boulea7 Boulea7 commented Oct 2, 2026

Copy link
Copy Markdown
Contributor

Fixes #64.

Treat None and pd.NA consistently with NaN during default categorical imputation and automatic one-hot encoding. Ordinal encoding maps missing values to zero, while raw passthrough preserves them without crashing the output report. Explicit imputer, category and drop settings retain their existing semantics.

Validation: 1,479 core tests passed with locked dependencies and with minimum NumPy/pandas/scikit-learn/SciPy versions; locked branch coverage is 92.66%. Ruff and Pyright pass.

Boulea7 and others added 2 commits October 26, 2026 18:00
Preserve fit-time output-format resolution and move pandas missing-value counting into the shared density helper. Add automatic-output regression coverage and clarify missing-marker behavior.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@codecov-commenter

Copy link
Copy Markdown

Welcome to Codecov 🎉

Once you merge this PR into your default branch, you're all set! Codecov will compare coverage reports and display results in all future pull requests.

ℹ️ You can also turn on project coverage checks and project coverage reporting on Pull Request comment

Thanks for integrating Codecov - We've got you covered ☂️

@mkumar73 mkumar73 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Boulea7, thank you for your contribution!

The updated changes address the categorical missing-value inconsistencies reported in #64.

  • Default categorical imputation now treats None, pd.NA and NaN consistently.

  • Ordinal encoding correctly maps missing values to zero, and automatic one-hot encoding handles nullable categorical inputs.

  • Explicit imputer missing markers, category definitions and drop settings retain their existing semantics.

  • Raw categorical passthrough preserves the original missing markers without crashing output reporting.

  • The merge-conflict resolution preserves #73’s fit-time output-format selection. Missing-value-safe density counting is shared between automatic format selection and output reporting, rather than duplicated.

  • The regression tests cover nullable inputs, transform-time missing values, missing indicators, cloning, serialization and raw passthrough. Additional tests verify that later low-density batches retain the output format selected during fit.

  • Validation is satisfactory,

  • CI and Docs workflows also pass, including the supported Python/OS matrix and minimum-dependency checks.

No further blocking code issues were identified in the reviewed changes.

Approved.

@mkumar73
mkumar73 merged commit ffdd61c into OpenTabular:main Oct 31, 2026
22 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

None / pd.NA categorical missing values are not treated as missing (nullable dtypes crash, None becomes a category)

3 participants