Dataset experiments - #220

Open

motiwari wants to merge 83 commits into

mainfrom

dataset_experiments

Commits on Jul 28, 2022

Do experiments on large dataset (not clean)
vxbrandon
committed
7ae25b5

Commits on Jul 29, 2022

Handling a few extreme cases!
vxbrandon
committed
50d03a4
Run experiments on various real dataset.
vxbrandon
committed
f3f31ee
Large refactor. Moving sklearn and mnist datasets to data_loader in compare_runtimes.py
motiwari
committed
6aa2e82
Refactoring datasets in compare_budgets.py
motiwari
committed
333f0f5
Large cleanup: compare_runtimes.py and compare_budget.py now avoid duplicated code. Also cleaned up Jeys real_experiments.py and regression_experiment.py
motiwari
committed
a446934
Passing dataset name to store profiles with dataset names and not overwrite
motiwari
committed
9031020
Cleaning (deleting) logfiles
motiwari
committed
47e84f6
Changing num_seeds back to 5 for runtime and budget
motiwari
committed
4234d4a
Adding HOUSING dataset, where our approach doesnt work, and renaming some SKLEARN to SKLEARN_REGRESSION to avoid naming collision
motiwari
committed
8445992

Commits on Jul 30, 2022

Adding forest covertype dataset -- we outperform, 10s to 600s
motiwari
committed
be22771
Trying to fix messed up merge
motiwari
committed
ca2b07a
Adding npy file to gitignore
motiwari
committed
0a7d534
Preparing for experiments: deleting old logs, writing new runtime profs to logs dir, choosing datasets
motiwari
committed
db64749
Storing data up to APS
motiwari
committed
c1b1530
Refactor codes and fix errors on scaling
vxbrandon
committed
9102cfc
Merge remote-tracking branch 'origin/dataset_experiments' into dataset_experiments
vxbrandon
committed
f2551c7
Classification results look good -- can use these for Table 1. Unfortunately contains the bug about solver reduction to each feature being an arm, but should be ok to use these for Table 1
motiwari
committed
e1dfe41
Uncomment runtime comparision for regression and fix setting
vxbrandon
committed
4dcebf8
results on AIR work with max_depth=3
motiwari
committed
56d54e0
Running regression exps with max_depth=3 helps sklearn. Blog still doesnt work
motiwari
committed
b82fbdb
Producing Tables 1 and 2 properly with produce_tables.py -- everything good except blog dataset
motiwari
committed
c35e903
Fixing errors with model names
motiwari
committed
373ddaa
Doubling budgets and only running 1 seed to ensure baselines can train something
motiwari
committed
6281d94

Commits on Jul 31, 2022

Adding ability to form predictions from forests when no full trees were fit in budget experiments
motiwari
committed
51c36b3
Figured out reasonable budgets for all exps. Running 5 seeds now. Also fixing bincounts to bincount
motiwari
committed
d42e14d
Addressing my own comments from review
motiwari
committed
a4815ac
Merge branch 'dataset_experiments' into new_scaling_exps
motiwari
authored
4d152bc
Merge pull request #226 from ThrunGroup/new_scaling_exps
motiwari
authored
3afdeb6
Add error bars on scaling exps. Add gpu dataset. Add vectorize_impurity flag.
vxbrandon
committed
c2312f1
Run black
vxbrandon
committed
f027bec
Delete data file to upload
vxbrandon
committed
de9fea8
Delete data file to upload
vxbrandon
committed
df57afb
Turn of flag of preprocessing gpu file
vxbrandon
committed
c79f5aa
Adding budget experiment results
motiwari
committed
b26514f
TAG: EXP_CLOSE. Merging Jeys data_experiments with mine. All runtime and budget experiments are here except for GPU dataset. The only difference between this code and how the runtime exps were prod…
motiwari
committed
5bf6ff8
TAG: EXP_OPEN, about to run runtime and budget exps for GPU dataset. Setting max_depth=1 for both
motiwari
committed
22c978a
Adding ALL experimental results including for new regression dataset. Only discrepancy is with Table2:AIR because we overwrote old results with a different max_depth
motiwari
committed
7a1ed4d
Nits
motiwari
committed
b218b06
TAG: EXP_OPEN, attempting to improve Table 4 results by tweaking budgets and setting max_depth=1
motiwari
committed
57d8672
TAG: EXP_CLOSE for 1-seed improvements to Table 4. EXP_OPEN for 5-seed improvements to Table 5
motiwari
committed
10c58ef
TAG: EXP_CLOSE, did 5 seeds for Table 4, looks much better now
motiwari
committed
a41119e

Commits on Aug 1, 2022

Made necessary cosmetic changes to produce_tables.py. Use all of its output except use old AIR Table 2 results
motiwari
committed
5302bf6
Nit
motiwari
committed
bd23865
TAG: EXP_OPEN. Table 3 experiments, budget for classification, look erroneous. Need to rerun
motiwari
committed
9f75e27
Adding small dataset exps for R3
motiwari
committed
7111696
Something is still wrong with Table 3. About to debug
motiwari
committed
edc32e1
TAG: EXP_OPEN. Fixing critical bug in classification budget to NOT just run on MNIST
motiwari
committed
3c06edc

Commits on Aug 2, 2022

TAG: EXP_OPEN. Reducing budget by factor of 5 in FLIGHT
motiwari
committed
4c7e5be
TAG: EXP_OPEN. Reducing budget by factor of 5 in COVTYPE
motiwari
committed
10afe2b
Rerunning exps with different max_depth with one seed to see perf diffs
motiwari
committed
a569986
TAG: EXP_OPEN. Running final table 3
motiwari
committed
cefa861
Adding proper Table 3 results
motiwari
committed
6cc0e27
After an hour, get .8088 with sklearn RFC and .5964 with Hoeffding Tree
motiwari
committed
940cf47
Don't plot negative value for num queries
vxbrandon
committed
baed34f
Merge remote-tracking branch 'origin/dataset_experiments' into dataset_experiments
vxbrandon
committed
3591185
Recreating final Table 3
motiwari
committed
da6745b

Commits on Sep 10, 2022

- Randomly rank the items that have same equal importance score
vxbrandon
committed
e8ab8ec
Clean up codes for reproducing the experiments
vxbrandon
committed
c4afa72

Commits on Sep 13, 2022

Merge branch 'reproducec_experiments' of github.com:ThrunGroup/FastForest into reproducec_experiments

Luke Donghyun Lee
authored and
Luke Donghyun Lee
committed
693319f

Commits on Sep 14, 2022

ran all experiments except gpu

Luke Donghyun Lee
authored and
Luke Donghyun Lee
committed
db08c77
ran all experiments except gpu

Luke Donghyun Lee
authored and
Luke Donghyun Lee
committed
2ea691b

Commits on Sep 15, 2022

Change hyperparams to reproduce experiments on Sklearn regression and MNIST
vxbrandon
committed
6852e88

Commits on Sep 18, 2022

Commits on Sep 19, 2022

1. Change back our algorithm to find the best arm not the 5th best arm
vxbrandon
committed
e7ba44f

Commits on Oct 9, 2022

Changing mab solver to len(candidates) > 1
motiwari
committed
94cfad8
Fixing repro script
motiwari
committed
444bd83

Commits on Oct 10, 2022

Adding results, choosing final datasets
motiwari
committed
aff2db8
Fixing table-making code
motiwari
committed
fcd420a
Adding tables
motiwari
committed
c723f69

Commits on Nov 8, 2022

Nit todo
motiwari
committed
b4f87a5
Created results that show for depth=5, pull_each_arm=333, batch_size=25, conf_multiplier=1, MAB outperforms RANDOM_SOLVER
motiwari
committed
8b232c8

Commits on Nov 9, 2022

Need to sample randomly 1000 times for roughly equal performances, theirs still slightly worse
motiwari
committed
da736ea

Commits on Dec 12, 2022

Undoing some random solver changes
motiwari
committed
d9723e7
Merge pull request #242 from ThrunGroup/random_baseline
motiwari
authored
9bcba82
Small style changes to sklearn comparison
motiwari
committed
975e3f3
Fixing ERFR problem for sklearn by allowing it to look at all features
motiwari
committed
23e34e1
Making sure candidate conditions are fixed. About to rerun repro script
motiwari
committed
541a283

Commits on Dec 13, 2022

All results recreated. Table 1 multipliers are a little more modest with more generous hyperparameters, Table 1 forest covtype accuracies a bit worse, and Table 2 some rows are identical, Scaling p…
motiwari
committed
abac8c4