On the Pitfalls of Analyzing Individual Neurons in Language Models

Omer Antverg, Yonatan Belinkov

While many studies have shown that linguistic information is encoded in hidden word representations, few have studied individual neurons, to show how and in which neurons it is enc…