Assessing the Impact of Anisotropy in Neural Representations of Speech: A Case Study on Keyword Spotting

Guillaume Wisniewski, Séverine Guillaume, Clara Rosina Fernández

Pretrained speech representations like wav2vec2 and HuBERT exhibit strong anisotropy, leading to high similarity between random embeddings. While widely observed, the impact of thi…