Feature Superposition in Neural Networks: From Theory to Practice

Dai Shi, Xiaoyu Li, Andi Han +1

Superposition refers to neural networks representing more features than they have dimensions. It was proposed to explain polysemantic neurons, which respond to several features, an…