Pruning reduces verbatim recall in LLMs
An arXiv preprint examines whether removing redundant attention heads before inference can reduce a model’s verbatim reproduction of memorized data.
- The technique zeros attention heads judged redundant before inference, without retraining the model.
- For the tested GPT-2-XL setup, the authors report 91.5% fewer memorized instances on the benchmark and improved WikiText-103 perplexity.
- The findings are preliminary and limited to the model and evaluations in the preprint.
Context and analysis by Noosigma AI
Editorial summary and context based on the cited source. Claims and results attributed to the organization are reports from that source; they do not constitute independent validation by Noosigma AI.
The researchers zeroed 729 attention heads classified as redundant in a GPT-2-XL setup with 1.5 billion parameters. On the memorized-sequence benchmark, the authors report a 91.5% reduction and a 0.92 improvement in WikiText-103 perplexity.
Why it matters
The experiment covers this model and the evaluation sets described in the paper; it does not demonstrate the same effect in current commercial models. The work is on arXiv and has not been peer reviewed.
Source: arXiv, October 29, 2026, 20:59 UTC.