Explore efficient reasoning in smaller models - #1
Open
JacquesGariepy wants to merge 5 commits into
Open
Conversation
…ning New features: - ReasoningController: decides when to continue reasoning vs answer - ProcessRewardModel: evaluates intermediate reasoning steps - ChainOfThoughtLoss: multi-component loss for CoT training - ReasoningLandmarkSelector: specialized landmarks for reasoning - SelfConsistencyDecoder: multi-path generation with voting - Special tokens: <think>, </think>, <step>, <answer> Architecture enables smaller models to reason efficiently by: - Using sparse local-global attention (O(L*(W+G)) vs O(L^2)) - Learning to select important reasoning anchors as landmarks - Progressive curriculum on reasoning depth - Process supervision for intermediate steps Files added: - src/reasoning.py: Core reasoning components - src/reasoning_model.py: Integrated SLGA-Reasoning model - config/config_reasoning.yaml: Training configuration - scripts/train_reasoning.py: Training script with GSM8K/MATH support
Implements arborescent exploration of reasoning paths: - ThoughtNode: Tree structure with parent/children links - ThoughtGenerator: Creates diverse alternative thoughts - ThoughtEvaluator: Scores coherence, progress, validity - TreeOfThought: Orchestrates search through the tree Search strategies implemented: - BFS: Breadth-first, exhaustive level-by-level - DFS: Depth-first with backtracking - BEAM: Keeps top-K branches at each level - MCTS: Monte Carlo Tree Search with UCB selection - BEST_FIRST: A*-like heuristic search Key improvements over linear CoT: - Explores multiple reasoning paths in parallel - Allows backtracking when a path fails - Voting mechanism for answer confidence - Pruning of low-quality branches Based on "Tree of Thoughts" (Yao et al., 2023)
Self-contained script with everything included: - ReasoningModel: Simplified SLGA with landmarks and PRM - SimpleTokenizer: GPT-2 based with special token handling - CoTDataset: Formats data as chain-of-thought - Data loaders: GSM8K + synthetic fallback - Training loop with AMP, gradient accumulation, scheduling - Evaluation and checkpointing Usage: python train_reasoning_simple.py # Full training python train_reasoning_simple.py --small # Quick test python train_reasoning_simple.py --dataset gsm8k No external dependencies beyond torch, datasets, transformers
New utilities: - eval_reasoning.py: Evaluate on GSM8K benchmark - Accuracy, exact match, step analysis - Error analysis with examples - chat_reasoning.py: Interactive chat interface - Real-time reasoning generation - Commands: /tot, /cot, /temp, /config - Colored output with step highlighting - test_reasoning_quick.py: Quick validation tests - Import check - Model creation - Forward/backward pass - Generation - Dataset loading Complete workflow now available: 1. python train_reasoning_simple.py --small # Train 2. python test_reasoning_quick.py # Verify 3. python eval_reasoning.py --checkpoint ... # Evaluate 4. python chat_reasoning.py --checkpoint ... # Interactive
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.