Skip to content

Critical: _find_checkpoint() glob patterns never match any trained checkpoint #43

Description

@bradsmithmba

Summary

EnhancedStrategyRecommender._find_checkpoint() searches for files matching checkpoints/regime_detector*.pt and checkpoints/regime_detector*.pth. No file produced by the training pipeline matches either pattern, so the ML path never activates even when a trained model exists.

Root cause

_find_checkpoint() (added in PR #42):

patterns = ['checkpoints/regime_detector*.pt', 'checkpoints/regime_detector*.pth']
matches = [f for p in patterns for f in glob.glob(p)]

Two problems:

Wrong directory. TrainingConfig.checkpoint_dir defaults to Path("models/checkpoints"), not checkpoints/. The training script overrides this to training_runs/<timestamp>/checkpoints/ when --output-dir is not set.

Wrong filename pattern. RegimeTrainer.save_checkpoint() writes best_model.pth and checkpoint_epoch_N.pth. The export_model_for_inference() path writes inference_model.pth. None of these contain regime_detector in the filename.

Impact

_find_checkpoint() always returns None. _detect_regime() always falls back to the SMA-crossover heuristic, silently. The entire ML pipeline installed in PR #42 is dead code until this is fixed.

Fix

_find_checkpoint() needs to:

  1. Search the actual checkpoint directories the trainer writes to (models/checkpoints/, any training_runs/*/checkpoints/)
  2. Match the actual filenames the trainer produces (best_model.pth, checkpoint_epoch_*.pth, inference_model.pth)

Example:

def _find_checkpoint(self) -> Optional[str]:
    patterns = [
        'models/checkpoints/best_model.pth',
        'models/checkpoints/checkpoint_epoch_*.pth',
        'training_runs/*/checkpoints/best_model.pth',
        'training_runs/*/checkpoints/checkpoint_epoch_*.pth',
        'training_runs/*/inference_model.pth',
    ]
    matches = [f for p in patterns for f in glob.glob(p)]
    return max(matches, key=os.path.getmtime) if matches else None

Related

Companion to issue: _detect_regime() calls load_state_dict() on a full checkpoint dict (should unpack model_state_dict key).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions