Does Reasoning Help? An Empirical Study on Financial Sentiment Analysis

Abstract

This project evaluates whether explicit reasoning steps (Chain of Thought) improve the performance of Large Language Models (LLMs) on domain-specific classification tasks. Using the llama-3.1-8b-instant model, we compared a Direct (Zero-Shot) prompting approach against a Chain of Thought (CoT) approach on the Financial PhraseBank dataset. The results demonstrate a substantial performance gain, with overall classification accuracy jumping from 66% under direct prompting to 87% when incorporating CoT reasoning. The findings highlight that even for seemingly straightforward tasks like sentiment classification, allowing the model to rationalize context significantly reduces classification errors.

Method

Dataset Setup

We utilized the gtfintechlab/financial_phrasebank_sentences_allagree dataset from Hugging Face. To ensure a balanced evaluation, we created a test set of 300 sentences, randomly sampling exactly 100 negative, 100 neutral, and 100 positive sentences.

Experimental Design

The experiment was conducted using llama-3.1-8b-instant hosted via the Groq API. We evaluated two distinct prompting strategies:

1. Direct Prompt: The model was given the sentence and instructed to reply with a single word (positive, negative, or neutral).
2. Chain of Thought (CoT) Prompt: The model was instructed to think step by step about the financial implications of the sentence before outputting a standardized answer format (Answer: [sentiment]).

We intentionally parsed the exact model output to capture the robustness of the formatting.

Results

Chain of Thought significantly outperformed direct prompting across all sentiment classes. Notably, the direct prompt occasionally failed to adhere strictly to formatting (e.g., adding punctuation like negative.), hurting exact-match classification. CoT dramatically improved both reasoning accuracy and structural compliance.

Prompting Strategy	Accuracy	Weighted F1-Score
Direct Prompt	66.0%	0.75
Chain of Thought	87.0%	0.87

Detailed Class Breakdown (Chain of Thought)

Class	Precision	Recall	F1-Score
Negative	0.97	0.91	0.94
Neutral	0.87	0.73	0.79
Positive	0.80	0.97	0.87

Insights & Novelty:

• The model showed immense improvement in identifying Negative and Positive sentences when using CoT (improving recall from ~60% to over 90%).
• Financial language often contains nuance (e.g., "sales fell, but profits rose"). The CoT step allows the model to map the causal structure of financial indicators before settling on a sentiment, mimicking human financial analyst workflows.

Conclusion

Chain of Thought reasoning provides a decisive advantage in financial sentiment analysis. Even highly capable, fast models like Llama 3.1 8B struggle to internalize complex financial context in a zero-shot setting. By enforcing a step-by-step reasoning phase, accuracy increases significantly (+21%), making CoT a default recommendation for financial NLP applications.

Reproducibility

To reproduce these results, follow the steps below from the root of the repository:

1. Setup Environment Ensure you have Python installed, then install the dependencies:

pip install pandas datasets scikit-learn groq python-dotenv matplotlib

2. Generate the Dataset

python src/data_prep.py

This generates a balanced test_set.csv (300 samples) in the data/ directory.

3. Run the Evaluation Create a .env file in the root directory and add your Groq API key:

GROQ_API_KEY=your_api_key_here

Then execute the experiment:

python src/run_experiment.py

This will prompt the model for both strategies and export predictions to data/results.csv.

4. Generate Metrics

python src/evaluate.py

This script will print the exact accuracy, precision, and recall classification tables.