Skip to content

Add generics describing what fitted expressions compute - #435

Merged
EmilHvitfeldt merged 1 commit into
mainfrom
output-metadata
Aug 24, 2026
Merged

Add generics describing what fitted expressions compute#435
EmilHvitfeldt merged 1 commit into
mainfrom
output-metadata

Conversation

@EmilHvitfeldt

Copy link
Copy Markdown
Member

T2 / phase 1b of the orbital new-backends plan. tidypredict_fit() returns expressions but says nothing about what they mean, and the meaning is not recoverable from their shape. Two cases where guessing wrong is silent rather than loud:

  • A LiblineaR logistic regression and a LiblineaR SVM classifier both return exactly one expression. Measured: the first lands in [0.0066, 0.971], the second straddles zero at [-2.25, 1.38]. Thresholding the second at 0.5 as though it were a probability misclassifies every row whose value falls between 0 and 0.5.
  • A multiclass probability list and a quantreg::rq() fit with several tau are both named lists of expressions of the same length. In the first, row sums are exactly 1; in the second they ran 31 to 84.

The three generics

tidypredict_output_type(), tidypredict_outcome_levels(), tidypredict_normalized(). Documented at ?tidypredict_metadata.

Asked of the model, not attached to the result

Attributes on the fit result were the obvious design and do not work. Measured: on a named list, x[lvl], lapply() and unlist() each drop them; on a language object a deparse/parse round trip drops them. orbital's own multiclass consumer does class_eqs[lvl] then unlist(), so the metadata would be gone before it could be read — in exactly the multiclass case that motivated it.

Independently, tidypredict_save() is yaml::write_yaml() on the parsed model, so attributes do not survive a round trip either, and it cannot accept a fit result at all (it tries to parse_model() a call).

Asking the model has nothing to strip, changes no existing return value, and matches the extractor-generic pattern from #433.

Only four output types, and two surprises

There is no "logit". Every multiclass list goes through one shared expr_softmax(), so values always already sum to one. tidypredict_normalized() is therefore always TRUE or NA, never FALSE. Kept anyway so callers can rely on the invariant instead of knowing it, and so a future non-normalizing backend can say so rather than silently break the assumption.

"class" covers more than expected. rpart, ctree, C5.0 and baguette bagger classification return a class label and no probability at all — there is nothing to argmax or normalize. And xgboost's binary:hinge is also "class": apply_xgb_objective() wraps it in as.numeric(score >= 0), so it takes only the values 0 and 1. That makes using it as a numeric prediction wrong even though its type is numeric. I had it as "numeric" first and changed it after checking the values.

Structure

44 tidypredict_fit() methods, but only 10 parsed-model types. So the .default methods route through parse_model() the way tidypredict_fit.default() does, and most methods sit on the pm_* classes. A backend gets a fitted-class method only where the parsed form is insufficient:

  • rpart and party do not record their mode once parsed
  • ksvm and glm do not record their response levels once parsed
  • parse_model.rqs() returns a bare list rather than a parsed_model
  • h2o has no parse_model() method
  • LiblineaR needs its type to tell probability from decision value

Where mode or levels had to be detected, the existing signal is reused rather than a parallel one invented — ksvm_is_classification() is factored out of parse_model_ksvm(), and bagger reuses bagger_classes().

Verification

Suite FAIL 0 | WARN 6 | SKIP 1 | PASS 2675, up from 2537. The 6 warnings are pre-existing and in other files. R CMD check with vignettes gives 0 errors, 0 warnings (notes are a worktree .git and the pre-existing unstated catboost).

Every backend was checked by fitting a model and evaluating the returned expressions, not by reading source. For one representative of each output type the tests assert the values, not just the label: prob lists sum to 1, single probs stay in [0,1], the rqs list demonstrably does not sum to 1, class models return labels drawn from their levels, and binary:hinge takes only the values 0 and 1.

One trap caught this way: kernlab::lev() returns the sorted response values for an SVR, so a naive implementation reported 25 numeric "levels" for a regression. Gated on kernlab::type(), with a test asserting NULL.

Pre-existing issues noticed, not fixed here

  • parse_model.rqs() breaks the parse_model() contract, returning a plain list of parsed models rather than a parsed_model. Anything routing through parse_model() generically cannot handle it, which is why rqs needs fitted-class methods.
  • parse_model_lm() calls summary() on rq fits, which runs rq.fit.br(ci = TRUE) and warns "Solution may be nonunique" once per quantile. Parsing should not need the inference machinery; it makes tidypredict_fit() on an rqs noisy.

tidypredict_fit() returns expressions but says nothing about what they
mean, and the meaning is not recoverable from their shape. Two cases
where getting it wrong is silent:

  A LiblineaR logistic regression and a LiblineaR SVM classifier both
  return one expression. The first is a probability in [0, 1]; the second
  is an uncalibrated decision value that straddles 0. Cutting the second
  at 0.5 as though it were a probability misclassifies every row whose
  value lies between 0 and 0.5.

  A multiclass probability list and a quantreg::rq() fit with several tau
  are both named lists of expressions of the same length. In the first
  the values sum to one; in the second they are unrelated predictions.

Add tidypredict_output_type(), tidypredict_outcome_levels() and
tidypredict_normalized(), asked of the model rather than of the result.
Attributes on the result were the obvious alternative and do not work:
they are dropped by the subsetting, lapply() and unlist() that callers
apply to a multiclass list, and tidypredict_save() serialises to YAML so
they do not survive a round trip either.

Only four output types exist. There is no "logit": every multiclass list
goes through one shared expr_softmax(), so tidypredict_normalized() is
always TRUE or NA and never FALSE. It is worth keeping so callers can
rely on that rather than having to know it.

Two cases are less obvious than they look. rpart, ctree, C5.0 and bagger
classification return a class label and no probability at all, so they
are "class" rather than "prob". xgboost's binary:hinge objective is also
"class": it is wrapped in as.numeric(score >= 0) and so takes only the
values 0 and 1, which makes using it as a numeric prediction wrong even
though its type is numeric.

There are 44 tidypredict_fit() methods but only 10 parsed-model types, so
the .default methods route through parse_model() the way
tidypredict_fit.default() does, and most methods sit on the pm_* classes.
A backend whose parsed form does not record what is needed gets a method
on the fitted class: rpart and party do not record their mode, ksvm and
glm do not record their response levels once parsed, and parse_model.rqs()
returns a bare list rather than a parsed model.
@EmilHvitfeldt
EmilHvitfeldt merged commit 7ea3c84 into main Aug 24, 2026
8 of 9 checks passed
@EmilHvitfeldt
EmilHvitfeldt deleted the output-metadata branch August 24, 2026 05:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant