Reconsidering Degeneration of Token Embeddings with Definitions for Encoder-based Pre-trained Language Models

Ying Zhang, Dongyuan Li, Manabu Okumura

Learning token embeddings based on token co-occurrence statistics has proven effective for both pre-training and fine-tuning in natural language processing. However, recent studies…