Skip to content

Commit 14171a5

Browse files
xuyxuxuyxu
authored
feat: add soft gradient boosting (#72)
* update code * update code * update code * update unit tests * update code * Update README.rst
1 parent 3cd7b77 commit 14171a5

7 files changed

Lines changed: 580 additions & 17 deletions

File tree

‎CHANGELOG.rst‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,7 @@ Changelog
1818
Ver 0.1.*
1919
---------
2020

21+
* |Feature| |API| Add :class:`SoftGradientBoostingClassifier` and :class:`SoftGradientBoostingRegressor` | `@xuyxu <https://github.com/xuyxu>`__
2122
* |Fix| Fix missing functionality of ``use_reduction_sum`` for :meth:`fit` of Gradient Boosting | `@xuyxu <https://github.com/xuyxu>`__
2223
* |Enhancement| Relax :mod:`tensorboard` as a soft dependency | `@xuyxu <https://github.com/xuyxu>`__
2324
* |Enhancement| |API| Simplify the training workflow of :class:`FastGeometricClassifier` and :class:`FastGeometricRegressor` | `@xuyxu <https://github.com/xuyxu>`__

‎README.rst‎

Lines changed: 19 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -92,23 +92,25 @@ Example
9292
Supported Ensemble
9393
------------------
9494

95-
+-------------------------+------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------------+
96-
| **Ensemble Name** | **Type** | **Paper** | **Repository** |
97-
+-------------------------+------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------------+
98-
| Fusion | Mixed | Not listed for its Perceptual Intuition | fusion.py |
99-
+-------------------------+------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------------+
100-
| Voting | Parallel | Not listed for its Perceptual Intuition | voting.py |
101-
+-------------------------+------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------------+
102-
| Bagging | Parallel | `Bagging Predictors <https://link.springer.com/content/pdf/10.1007/BF00058655.pdf>`__ | bagging.py |
103-
+-------------------------+------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------------+
104-
| Gradient Boosting | Sequential | `Greedy Function Approximation: A Gradient Boosting Machine <https://www.jstor.org/stable/pdf/2699986.pdf?casa_token=3fkT9safZHUAAAAA:HT_MeRk_xNsUZkOpbixOtXc950xnRSXNAyl7WjGZgjLtwBTAzZaQe2urnVyp5sK1dIXRL-9hVrdvjT-Ex_PEvov5tTyFg6wMaSbhCzkJRfUj4uBJ6l_PHA>`__ | gradient_boosting.py |
105-
+-------------------------+------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------------+
106-
| Snapshot Ensemble | Sequential | `[ICLR'17] Snapshot Ensembles: Train 1, Get m For Free <https://arxiv.org/pdf/1704.00109.pdf>`__ | snapshot_ensemble.py |
107-
+-------------------------+------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------------+
108-
| Adversarial Training | Parallel | `[NIPS'17] Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles <https://arxiv.org/pdf/1612.01474.pdf>`__ | adversarial_training.py |
109-
+-------------------------+------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------------+
110-
| Fast Geometric Ensemble | Sequential | `[NeurIPS'18] Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs <https://arxiv.org/pdf/1802.10026;Loss>`__ | fast_geometric.py |
111-
+-------------------------+------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------------+
95+
+-------------------------+------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------------+
96+
| **Ensemble Name** | **Type** | **Paper** | **Repository** |
97+
+-------------------------+------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------------+
98+
| Fusion | Mixed | Not listed for its Perceptual Intuition | fusion.py |
99+
+-------------------------+------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------------+
100+
| Voting | Parallel | Not listed for its Perceptual Intuition | voting.py |
101+
+-------------------------+------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------------+
102+
| Bagging | Parallel | `Bagging Predictors <https://link.springer.com/content/pdf/10.1007/BF00058655.pdf>`__ | bagging.py |
103+
+-------------------------+------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------------+
104+
| Gradient Boosting | Sequential | `Greedy Function Approximation: A Gradient Boosting Machine <https://www.jstor.org/stable/pdf/2699986.pdf?casa_token=3fkT9safZHUAAAAA:HT_MeRk_xNsUZkOpbixOtXc950xnRSXNAyl7WjGZgjLtwBTAzZaQe2urnVyp5sK1dIXRL-9hVrdvjT-Ex_PEvov5tTyFg6wMaSbhCzkJRfUj4uBJ6l_PHA>`__ | gradient_boosting.py |
105+
+-------------------------+------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------------+
106+
| Soft Gradient Boosting | Parallel | `[arXiv'20] Soft Gradient Boosting Machine <https://arxiv.org/abs/2006.04059>`__ | soft_gradient_boosting.py |
107+
+-------------------------+------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------------+
108+
| Snapshot Ensemble | Sequential | `[ICLR'17] Snapshot Ensembles: Train 1, Get m For Free <https://arxiv.org/pdf/1704.00109.pdf>`__ | snapshot_ensemble.py |
109+
+-------------------------+------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------------+
110+
| Adversarial Training | Parallel | `[NIPS'17] Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles <https://arxiv.org/pdf/1612.01474.pdf>`__ | adversarial_training.py |
111+
+-------------------------+------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------------+
112+
| Fast Geometric Ensemble | Sequential | `[NIPS'18] Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs <https://arxiv.org/pdf/1802.10026;Loss>`__ | fast_geometric.py |
113+
+-------------------------+------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------------+
112114

113115
Package Dependency
114116
------------------

‎docs/parameters.rst‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -85,6 +85,25 @@ GradientBoostingRegressor
8585
.. autoclass:: torchensemble.gradient_boosting.GradientBoostingRegressor
8686
:members:
8787

88+
Soft Gradient Boosting
89+
----------------------
90+
91+
In soft gradient boosting, all base estimators could be simultaneously
92+
fitted, while achieving the similar boosting improvements as in gradient
93+
boosting.
94+
95+
SoftGradientBoostingClassifier
96+
******************************
97+
98+
.. autoclass:: torchensemble.soft_gradient_boosting.SoftGradientBoostingClassifier
99+
:members:
100+
101+
SoftGradientBoostingRegressor
102+
*****************************
103+
104+
.. autoclass:: torchensemble.soft_gradient_boosting.SoftGradientBoostingRegressor
105+
:members:
106+
88107
Snapshot Ensemble
89108
-----------------
90109

‎torchensemble/__init__.py‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,8 @@
1212
from .adversarial_training import AdversarialTrainingRegressor
1313
from .fast_geometric import FastGeometricClassifier
1414
from .fast_geometric import FastGeometricRegressor
15+
from .soft_gradient_boosting import SoftGradientBoostingClassifier
16+
from .soft_gradient_boosting import SoftGradientBoostingRegressor
1517

1618

1719
__all__ = [
@@ -29,4 +31,6 @@
2931
"AdversarialTrainingRegressor",
3032
"FastGeometricClassifier",
3133
"FastGeometricRegressor",
34+
"SoftGradientBoostingClassifier",
35+
"SoftGradientBoostingRegressor",
3236
]

0 commit comments

Comments
 (0)