By Aniket Ghosh
Function vectors (Todd et al. 2024) and the refusal direction (Arditi et al. 2024) both do the same thing: find where a behavior lives inside a model, pull it out as a direction, then add or remove it to switch the behavior on and off. I wanted to know if that still works when the behavior isn't a temporary in-context task but something baked into the model — a chat model's personality, or a fine-tune's hidden bad habit. Most of it runs on Qwen2.5-7B-Instruct, with a transfer check on Llama-3.1-8B at the end. The full plan is in CLAUDE.md.
Most of the experiments use the same search. I split the model into small pieces (every attention head, and each MLP cut into chunks) and test one piece at a time by editing it: either switching it off, or pasting in a clean model's version of that piece. A piece is kept only if editing it changes the target behavior without hurting anything else, which a guardrail checks by watching general ability and the other traits. The pieces that survive get stacked into one small set. The code is in the notebooks.
I turn three traits (sycophancy, agreeableness, risk-seeking) on and off with system prompts, and train a probe at every layer to find where each one shows up.
The traits show up late in the network, at layers 20–21. Grammar, used as a comparison, shows up early, at layer 2.
But finding where a trait is read isn't the same as finding where it's stored. Damaging the layers the probe points to mostly just makes the model worse at everything. Once I compare against a matched control, only sycophancy takes real, trait-specific damage.
Sycophancy also doesn't live in any single head or neuron. A full sweep of every head and MLP chunk, with the guarded search above, needs a dozen pieces to move it, and even then only cuts it from 97% to 78%. It's spread thin across many weak contributors.
Each trait has its own separate set of pieces, so removing one leaves the others alone. But remove three assistant traits at once and the darker traits the model normally suppresses — psychopathy, narcissism, machiavellianism — climb back up. The polite-assistant character seems to be part of what keeps them down. That's a warning about bulk trait editing, not proof of a meaner model in the wild.
The test model is a LoRA fine-tune of Qwen trained to give subtly bad medical advice (Soligo et al. 2025). I first check that the clean and bad versions still agree on unrelated questions, so the fine-tune hasn't quietly changed everything. Then I run the same search, but instead of switching pieces off I paste the clean model's version into the bad model.
Seventeen attention heads do it.
The multiple-choice score used to pick the heads can't tell a real cure from noise, so the real test is reading the answers. On held-out medical questions, blind and shuffled, the patched model reads like the clean model and drops the bad advice: weight-based dosing for a child instead of a flat overdose, treating chest pain as an emergency, refusing to mix alcohol with sleeping pills. It works.
One useful surprise came out of it. The fine-tune's own weights point at different pieces than the ones that actually move behavior. So the bad habit isn't a clean new thing the fine-tune wrote. It's a reshuffle of directions the model already had.
Recent work says a single direction should be enough. I checked. Pulling out one direction is a partial lever: removing it fixes about a third of the answers, but adding it to a clean model barely does anything. So one direction carries part of the behavior, not all of it.
I also tried a crosscoder, which hunts for the difference between the two models with no hint from me. At this scale it found nothing useful — the feature it flagged fires on formatting tokens, not medical content. The direct, causal search is the sharper tool here.
Same 36 held-out questions, one blind judge, one capability check. Every method keeps general ability intact (ARC-Easy stays at 100%), so none of them "cured" the model by breaking it. They differ on how much bad advice they actually remove. Each row below is the same 36 questions, split into safe, borderline, and unsafe.
First the three activation methods on their own (10_methods.ipynb):
| method | safe | borderline | unsafe |
|---|---|---|---|
| clean (untouched good model) | 33 | 3 | 0 |
| paste, 17 heads | 30 | 5 | 1 |
| single direction | 13 | 8 | 15 |
| crosscoder feature | 1 | 6 | 29 |
| bad (no fix) | 2 | 5 | 29 |
Then the wider comparison (11_safety_methods.ipynb), which adds the two things people reach for first, a safety system prompt and a few in-context safe examples, and re-scores everything with a stricter judge (Opus 5). The shared methods land a little lower under the stricter judge, but the order holds:
| method | safe | borderline | unsafe |
|---|---|---|---|
| clean (untouched good model) | 30 | 5 | 1 |
| paste, 17 heads | 27 | 8 | 1 |
| few-shot safe examples | 23 | 4 | 9 |
| safety system prompt | 10 | 3 | 23 |
| single direction | 10 | 8 | 18 |
| bad (no fix) | 3 | 4 | 29 |
| crosscoder feature | 1 | 4 | 31 |
Two things come out of this. Among the methods that edit the model, the more distributed one wins: a set of heads beats one direction beats one unsupervised feature, because the bad behavior is itself spread out. And the cheap black-box baseline, a few safe examples dropped into the prompt, is stronger than I expected at 23 safe, though it still lets nine unsafe answers through where the paste lets one. Paste is the only fix that matches the clean model, at 27 safe and a single unsafe.
Notebook 11 also prints a decision matrix for the whole toolbox, including fine-tuning and unlearning (described, not run), laying out what each method costs to deploy: whether it needs the weights, the trigger, labeled data, or a full retrain.
This comparison is one model and one seed, so read these as worked examples, not settled laws.
The whole comparison above is one fine-tune of Qwen, so the obvious worry is that the 17 heads only mean something for that one model. I ran the paste-cure on a second organism to check: Llama-3.1-8B, trained the same bad-medical way, a different architecture entirely, judged blind on 30 held-out questions.
First pass, Llama barely budged, and it looked like the method just didn't carry across. It wasn't the method. It was a knob. The guardrail floor I'd tuned on Qwen was too tight for Llama and choked the search off after a handful of heads. Notebook 13 tunes that floor on Llama alone; loosening it let the search keep 14 heads without ever denting ARC. With the floor set right, Llama sheds most of its bad advice — 19 of 30 safe, up from 3, next to 24 for the clean model. Qwen stays cured (25 of 30).
So the mechanism transfers across architectures, but the guardrail setting doesn't — it has to be recalibrated per model. Worth knowing before assuming one tuned pipeline drops onto a new model unchanged.
| Notebook | What it does |
|---|---|
01_persona.ipynb |
Where three persona traits live, and what damaging a single layer does to them against matched controls. |
02_components.ipynb |
Splitting the persona layers open: attention vs. MLP, then single heads and neuron chunks. |
03_sweep.ipynb |
A census of all 56 components, ranked by trait-specific damage. |
04_atlas.ipynb |
Every head and MLP chunk, screened, rescored, and greedily removed under a guardrail. |
05_shared.ipynb |
Whether the traits share pieces, and what happens when you remove several at once. |
06_component_paste_sweep.ipynb |
Pasting a clean model's components into the bad-medical-advice fine-tune to cure it. |
07_blind_eval.ipynb |
Regenerates the clean / bad / patched answers on a larger held-out set and writes a shuffled file for a blind judge. |
08_direction.ipynb |
Is the fine-tune carried by one direction? Diff-of-means, then add it (sufficiency) and remove it (necessity) against random controls. |
09_crosscoder.ipynb |
A shared sparse crosscoder over both models, reading off the bad-only features and checking them against 08 and 06. |
10_methods.ipynb |
The three removals head-to-head on one blind battery and one capability check. |
11_safety_methods.ipynb |
This method next to the standard safety toolbox (prompting, few-shot, steering, SAE, fine-tuning), with a decision matrix of what each costs to deploy. |
12_organism_transfer.ipynb |
Reruns the paste-cure on a second bad-medical-advice organism (Llama-3.1-8B) alongside the original Qwen-7B, judged blind on 30 ChatDoctor questions, to see whether the cure transfers across architectures. |
13_tune_llama.ipynb |
Sweeps the search's knobs — mainly the guardrail floor — on Llama-8B alone, with an automatic judge, to work out why the cure underperformed there and fix it. |
Notebooks 01–05 began in the sibling probe-guided-quantization repo and are renumbered here into reading order; 06–13 are this project's own. Figures land in figures/. The executed notebooks, with their printed results and plots, are in this Drive folder.
- Alain, G., and Bengio, Y. (2017). Understanding intermediate layers using linear classifier probes.
- Arditi, A., et al. (2024). Refusal in Language Models Is Mediated by a Single Direction.
- Chen, R., et al. (2025). Persona Vectors: Monitoring and Controlling Character Traits in Language Models.
- Clark, P., et al. (2018). Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge.
- Hubinger, E., et al. (2024). Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training (Anthropic).
- Lindsey, J., et al. (2024). Sparse Crosscoders for Cross-Layer Features and Model Diffing. Transformer Circuits Thread.
- Marks, S., et al. (2025). Auditing Language Models for Hidden Objectives (Anthropic).
- Perez, E., et al. (2022). Discovering Language Model Behaviors with Model-Written Evaluations. ACL Findings.
- Qwen Team (2024). Qwen2.5 Technical Report.
- Soligo, C., et al. (2025). Model Organisms for Emergent Misalignment.
- Todd, E., et al. (2024). Function Vectors in Large Language Models.
- Wang, M., et al. (2025). Persona Features Control Emergent Misalignment. arXiv:2506.19823.