Induction Head Toxicity Mechanistically Explains Repetition Curse in Large Language Models

Shuxun Wang, Qingyu Yin, Chak Tou Leong +2

Repetition curse is a phenomenon where Large Language Models (LLMs) generate repetitive sequences of tokens or cyclic sequences. While the repetition curse has been widely observed…