11 Precision pruning for bias

The chapter shows that demographic bias in a language model can be treated as a localized internal behavior rather than an untouchable property of the whole network. By using pairs of prompts that differ only in one demographic attribute, the model’s activations are compared neuron by neuron to find where the responses diverge. The key idea is that if two prompts are otherwise identical, differences in activation can be traced to the changed attribute, making it possible to identify candidate neurons and rank them by how strongly they respond.

After locating the most affected neurons, the chapter demonstrates a very precise intervention method: scaling the weights of only a few selected neurons in the model’s MLP layers. In the race example, five neurons were identified and reduced, and this sharply softened the stereotyped completion while leaving the majority-side completion largely intact. A second experiment on gender bias followed the same workflow but showed that different bias axes may involve different neurons and different layer concentrations, with only one layer-10 neuron needing adjustment to noticeably reduce the unwanted effect.

The chapter then checks whether these interventions harm the model more broadly and finds little to no measurable loss in general task performance. It also uses BBQ to test bias more systematically, showing that the intervention pushes race-related bias scores closer to neutral in both ambiguous and disambiguated settings. Overall, the chapter argues that carefully targeting a tiny subset of neurons can reduce stereotyped behavior, generalize beyond a single prompt pair, and do so without noticeably damaging the model’s broader abilities.

The MLP module of Llama-3.2-1B, with its three layers: up_proj, gate_proj, and down_proj. The capture point marks the input of down_proj, the point where activations are captured to compare the two prompts. The intervention point marks up_proj, the layer where the modification will be applied.

Averaged down_proj_input activation difference between the stereotyped prompt and the majority prompt, by layer and neuron bin (bin=64), for the prompt pair introduced in listing 11.1. Darker zones mark a bigger difference favoring the stereotyped prompt, lighter zones mark a bigger difference favoring the majority prompt.

Pipeline of the four phases followed in the chapter's experiments—detect, select, intervene, and validate—with the actual function of each phase and what it produces. The intervene step, marked with a dashed border, is the only one that modifies the model's weights; the rest are measurement or generation phases.

Logit lens for the majority prompt ("boy"), layers 10 to 15, comparing the base model and the intervened model. Each cell shows the top three predictions for that layer and their probability. Background intensity encodes the probability of the top prediction.

Logit lens for the stereotyped prompt ("girl"), layers 10 to 15, comparing the base model and the intervened model. Each cell shows the top three predictions for that layer and their probability. Background intensity encodes the probability of the top prediction.

Comparison of general capability benchmarks between the base model and the two intervened models, race and gender, on five standard tasks.

BBQ Bias score for the race/ethnicity category, ambiguous and disambiguated contexts, comparing the base model and the model with the race intervention from section 11.2 applied.

Jaccard overlap between the five demographic categories evaluated in Llama-3.2-1B in English, over the top 0.1% neurons by BiasScore per category, 131 neurons per category. Values below 0.33 indicate that between roughly 51% and 72% of each category's candidate neurons are exclusive to it. Figure reproduced from Martra et al. (2026).

• A pair of prompts identical except for a single demographic attribute lets you attribute any difference in model behavior to that one variable.
• The most direct point to measure a neuron's contribution to the residual stream is the input of down_proj, where the two expanded GLU projections have already been multiplied together.
• The top-k neurons with the largest activation differences tend to cluster in the deeper layers of the model, mostly the last one.
• Scaling only up_proj by a factor reduces a neuron's contribution to down_proj_input in an almost proportional way to that factor, which gives a predictable and targeted way to weaken a neuron's effect without removing it completely.
• A localized intervention in one specific scenario can generalize to completely different contexts that share the same demographic attribute, a sign that it's acting on the attribute itself and not on the concrete wording of a prompt.
• Not all bias circuits concentrate in the last layer of the model; in the gender experiment, the neurons with the highest score were distributed across layers 10, 14, and 15.
• Different demographic biases live in mostly separate circuits: the race and gender neurons you selected didn't overlap, a pattern confirmed by Jaccard analysis.
• Scaling a minimal group of neurons, a tiny fraction of the total width of the model's MLP layers, has no effect on general capability benchmarks.
• BBQ, a benchmark built on pairs of ambiguous and disambiguated questions, lets you confirm quantitatively whether an intervention on a handful of neurons holds up at a larger scale, beyond the specific prompts used to localize them.
• Anthropic's work established that a model's internal concepts can be isolated and steered with precision, but at the cost of training a separate sparse autoencoder and keeping control over the inference engine.

FAQ