ENH model capability flags + per-capability deactivation (covariate lift)#38
Open
GeoffNN wants to merge 1 commit into
Open
ENH model capability flags + per-capability deactivation (covariate lift)#38GeoffNN wants to merge 1 commit into
GeoffNN wants to merge 1 commit into
Conversation
Declare each forecasting model's capabilities (multivariate, hist/future covariates) and let users deactivate covariate capabilities per run to benchmark the lift each provides. - benchmark_utils/capabilities.py: flag vocabulary + mask_covariates helper - BaseTSFMAdapter.covariate_capabilities: effective active set (default empty => univariate, safe by default) - Objective._eval_forecasting masks the covariate payload to the adapter's capabilities before predict() -- single, guaranteed enforcement point - Solvers declare `capabilities`; TFC-API exposes use_hist_covars / use_future_covars toggles and now threads covariates to the SDK via historical_variables / future_variables (multivariate stays declarative) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Flag each forecasting model with the capabilities it supports —
multivariate,hist_covariates,future_covariates— and let a user deactivate each covariate capability per run, so the lift from each can be benchmarked.univariateis deliberately not a flag: it's the floor every model gets. A model that declares (or has enabled) none of the covariate capabilities runs univariate.How
benchmark_utils/capabilities.py(new) — flag vocabulary +mask_covariates(covariates, active)helper. Masking only ever drops covariates; targets are never touched.BaseTSFMAdapter.covariate_capabilities— the effective active set for a run. Defaults to empty ⇒ safe-by-default (a new adapter sees no covariates / runs univariate).Objective._eval_forecastingmasks the covariate payload down to the adapter'scovariate_capabilitiesbeforepredict()— a single, guaranteed enforcement point for every forecasting model, present and future.capabilities(metadata): Chronos/Naive/SeasonalNaive{}, Chronos2/Toto2{multivariate}, TFC-API{multivariate, hist_covariates, future_covariates}.use_hist_covars/use_future_covars(defaultTrue) and now actually threads covariates to the SDK viahistorical_variables/future_variables(columns attached in both the batched and per-series paths).Lift sweep
⇒ two rows; the metric delta is the future-covariate lift.
Design note
multivariateis declarative-only metadata: targets are always passed whole (no channel-splitting), so there's no behavioural toggle for it yet — it exists to describe the model until a multivariate-target dataset and the matching masking land.static_covarsis likewise out of scope (no dataset uses it).Tests
tests/benchmark_utils/test_capabilities.py—mask_covariatesdrops the right fields peractiveset; static preserved; input not mutated.tests/test_objective_capability_masking.py— recording adapter confirms the objective masks percovariate_capabilitiesbeforepredict.tests/solvers/test_tfc_api_covariates.py— mockedcross_validateasserts*_variables+ columns present when covariates supplied,None/absent when deactivated. No network / API key.12 passed. Independent of the leakage (#24) and Enedis (#26) PRs — built on the covariate plumbing already in
main.🤖 Generated with Claude Code