Results aren't reproduced with readme command

[TravisNP]

I ran the command in the readme and did not get the expected perfect accuracy results. I've pasted the results of the last epoch and the best training and validation accuracy below. Have I done something wrong? Thanks!

epoch: 31
Train Acc: 0.46, partial_train_acc: 0.85, total_loss: 0.00114, lr: 0.0000033761, samples/sec: 1,
Valid Acc: 0.49, partial_valid_acc: 0.86, total_loss: 0.00107
Test Acc: 0.52
ood_long Acc: 0.52
ood_new Acc: 0.50
final metrics & hparams:
{'last_train_acc': 0.45625, 'last_valid_acc': 0.4872, 'best_train_acc': 0.534375, 'best_valid_acc': 0.4872, 'd_filler': None, 'd_role': None, 'dtm_steps': 16, 'op_dist_fn': 'softmax', 'arg_dist_fn': 'softmax', 'transformer_norm_first': 1, 'transformer_activation': 'gelu', 'transformer_nheads': 4, 'router_dropout': 0.1, 'router_hidden_dim': 256, 'ctrl_hidden_dim': 64, 'ctrl_num_layers': 1, 'predefined_operations_are_random': False, 'epoch': 32, 'steps': 20000.0, 'batch_size': 16, 'seed': 32780, 'data_dir': './data_files', 'task_type': 'active_logical_ttb', 'optim_beta1': 0.9, 'optim_beta2': 0.95, 'lr': 0.0001, 'scheduler': 'cosine', 'scheduler_gamma': 1, 'wd': 0.1, 'gclip': 1.0, 'num_warmup_steps': 10000.0, 'max_tree_depth': 12, 'debug': 1, 'log_grad_norm': False, 'save_file': 'result.tsv', 'checkpoint_file': 'checkpoint.pt', 'max_train_examples': None, 'validate_every_num_epochs': 1, 'wandb_tag': 'test', 'wandb_name': None, 'use_wandb': False, 'train_log_freq': 20.0}

Best Train Acc: 0.534
Best Valid Acc: 0.487

[TravisNP]

@psoulos Can I get confirmation you have seen this?

[psoulos]

Hi @TravisNP, sorry for the delay, I did not receive a notification from Github about your message. Did you try running the experiment a few times? The run sometimes gets stuck in a local optima and needs to be run a few times to find one that gets the desired results. I would also recommend switch to the sDTM model which is much improved and has better convergence properties.