Reduce latency with vector-based formula - #283
Open
matthieugomez wants to merge 8 commits into
Open
Conversation
- Add utils/vectorterms.jl: replicas of the StatsModels schema/modelcols/coefnames
pipeline operating on Vector{AbstractTerm} and runtime column lookup, so methods
compile per term type rather than per formula shape or column names
- Rewire fit to the vector path (parsing, schema, model matrices, coefnames)
- Add @nospecialize to reg and to the FixedEffectModel constructor formula arguments
- Cover categorical and interaction term types in the precompile workload
- Test that reg never modifies the input dataframe
- Bump version to 2.1.0
matthieugomez
changed the title
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #283 +/- ##
=======================================
Coverage ? 96.70%
=======================================
Files ? 9
Lines ? 911
Branches ? 0
=======================================
Hits ? 881
Misses ? 30
Partials ? 0 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Loading it up front invalidates the precompiled pipeline, inflating the earlier first-call timings with recompilation
concrete_term fills ContinuousTerm's mean/var/min/max with three O(n) passes per continuous column; nothing downstream reads them. Store NaN placeholders instead (and build one schema shared by Y and X in partial_out), roughly halving warm reg time on 10M-row benchmarks.
- Update benchmark.md with fresh timings for all four packages on the same machine (Apple M4): FixedEffectModels 2.1.0 / Julia 1.12.7, fixest 0.13.2 and lfe 3.1.1 on R 4.4.2, reghdfe 5.7.3 / ivreg2 4.1.11 on Stata 19 MP - Regenerate the comparison graph and point benchmark.md to the repo png (the old rawgit link is dead) - Restore newlines in benchmark.csv and result.jl (both had been mangled onto one line) and make result.jl read benchmark.csv relative to the script - Refresh benchmark.jl timing comments after the schema speedup - Remove the clustered-standard-errors claim from the README benchmark paragraph
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rewrites the formula pipeline in
fitto operate onVector{AbstractTerm}with runtime column lookup (newsrc/utils/vectorterms.jl), so methods compile once per term type instead of once per formula shape and per set of column names. A regression with a never-seen formula now runs goes from ~0.28s to ~0.005s (~0.03s with fe), and the 10M-row benchmarks are also faster at runtime (fewer allocations from the preallocated model matrix).parse_iv/parse_fe; the tuple-based versions are kept forpartial_outand external users@nospecializetoregand to theFixedEffectModelconstructor formula argumentsregnever modifies the input dataframe (model matrices are now built from views into it)