Optimizes how much of a pre-trained language model to freeze during fine-tuning by treating the freezing schedule as a search problem, using genetic algorithms and the Whale Optimization Algorithm instead of hand-picked settings.
Research code and results behind the paper Enhancing Fine-Tuning of Pre-trained Language Models for Sentiment Analysis Using Metaheuristic Algorithms (peer-reviewed conference; the manuscript is included in this repository).
Status: research artifacts, partially released. The genetic-algorithm experiment on RoBERTa is committed as a notebook. The Whale Optimization implementation is not in this repository, though its results are: see What is and is not here.
Fine-tuning a pre-trained model on a downstream task means deciding how much of it to leave alone. Freeze too much and the model cannot adapt; freeze too little and you overwrite pre-trained representations with gradients from a small dataset, and pay for the compute besides. In practice that decision is a guess, usually "freeze everything but the last few layers."
The question here is whether a search algorithm can make that decision better than a heuristic can, when the thing being searched over is the freezing percentage and the recombination strategy used to breed candidate configurations.
Task and model. Binary sentiment classification on the Stanford Sentiment Treebank 2 (SST-2), with
roberta-base as the backbone.
Search space. The proportion of layers frozen during fine-tuning (30%, 50%, 70%), crossed with the crossover operator used to combine candidates (SBX, single-point, two-point, arithmetic).
Two optimizers. A genetic algorithm with tournament selection, crossover, and Gaussian mutation over a population of configurations, and the Whale Optimization Algorithm over the same space. Each candidate is evaluated by fine-tuning and scoring on the validation set, which is why the search space is kept small.
From results.docx, which contains both result tables in full. Best configuration per optimizer:
| Optimizer | Crossover | Freezing | Accuracy | F1 | Precision | Recall |
|---|---|---|---|---|---|---|
| Whale Optimization | SBX | 30% | 94.67 | 94.6 | 95.21 | 94.14 |
| Genetic Algorithm | SBX | 30% | 94.02 | 94.1 | 94.56 | 93.92 |
Two things hold across both tables and both optimizers: SBX crossover wins, and less freezing wins, with accuracy declining monotonically from 30% to 70% freezing in almost every row. WOA beats GA at every matched configuration, though the margin is under a point and no significance test was run.
Full grids for all twelve configurations per optimizer are in results.docx.
Being explicit, because the repository does not match the paper one-to-one.
| In this repository | |
|---|---|
| GA over RoBERTa / SST-2 | Yes, Untitled.ipynb. This is the notebook that corresponds to the paper. |
| Whale Optimization implementation | No. Only its results, in results.docx. The code is not published here. |
| Result tables for both optimizers | Yes, results.docx and Percent.docx. |
| The paper itself | Yes, as .pdf and .docx. |
| GA over DistilBART summarization | Yes, GeneticAlgo.py. This is a separate exploratory script, not the paper's experiment, and it has a known defect (below). |
| Reference material | Several PDFs of cited papers, a LaTeX conference template, and a vendored scikit-opt archive. None of it is required to run anything. |
create_individual() selects parameters whose names contain "classifier". DistilBART has no
parameters matching that name, so the individual is an empty dict and the genetic algorithm perturbs
nothing. The script runs to completion and reports fitness scores that reflect the unmodified model.
Targeting lm_head instead would fix it. It is left in the repository as an exploratory artifact rather
than deleted, and it is not the experiment the paper reports.
PyTorch, Hugging Face Transformers (roberta-base, distilbart-cnn-12-6), Hugging Face datasets
(GLUE / SST-2, CNN-DailyMail) and evaluate, rouge_score, scikit-learn, NumPy. Mixed-precision
inference under torch.cuda.amp.autocast to keep each fitness evaluation affordable.
Requires Python 3.10 or newer. A CUDA GPU is effectively required: every fitness evaluation is a fine-tuning pass.
git clone https://github.com/patsypppe/metaheuristic-llm-finetuning.git
cd metaheuristic-llm-finetuning
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txtRun the paper's experiment:
jupyter notebook Untitled.ipynbRun the exploratory summarization script (see the defect note above):
python GeneticAlgo.pyOr with Docker:
docker build -t metaheuristic-finetuning .
docker run --rm metaheuristic-finetuningDatasets download from the Hugging Face Hub on first run. No API key is required.
- The Whale Optimization code is not published, so the headline 94.67% result cannot be reproduced from this repository. The genetic-algorithm result can.
Untitled.ipynbhas only four executed cells, so the notebook as committed does not display the results in the tables. It needs a full re-run with outputs saved.- No significance testing. WOA leads GA by roughly 0.65 points at the best configuration, across a single run per cell of the grid. That gap is not shown to be more than noise.
- A single random seed per configuration. Results are not averaged over restarts.
- The repository carries non-code material (reference PDFs, a LaTeX template, an unused vendored archive) that is not needed to run anything.
GeneticAlgo.py's search is inert as described above.
Published at a peer-reviewed conference. The manuscript is committed to this repository; citation details and DOI are available on request or via LinkedIn.
MIT. See LICENSE. The included third-party papers and templates remain under their own terms.