Training Dynamics of Contextual N-Grams in Language Models

Lucia Quirke, Lovis Heindrich, Wes Gurnee +1

Prior work has shown the existence of contextual neurons in language models, including a neuron that activates on German text. We show that this neuron exists within a broader cont…