Better Embeddings with Coupled Adam

Felix Stollenwerk, Tobias Stollenwerk

Despite their remarkable capabilities, LLMs learn word representations that exhibit the undesirable yet poorly understood feature of anisotropy. In this paper, we argue that the se…