Representation Degeneration Problem in Training Natural Language Generation Models

Abstract

We study an interesting problem in training neural network-based models for natural language generation tasks, which we call the \emph{representation degeneration problem}. We observe that when training a model for natural language generation tasks through likelihood maximization with the weight tying trick, especially with big training datasets, most of the learnt word embeddings tend to degenerate and be distributed into a narrow cone, which largely limits the representation power of word embeddings. We analyze the conditions and causes of this problem and propose a novel regularization method to address it. Experiments on language modeling and machine translation show that our method can largely mitigate the representation degeneration problem and achieve better performance than baseline algorithms.

ICLR 2019

Cited by in corpus (19)

• Contrastive Learning for Representation Degeneration Problem in Sequential Recommendation
• Whitening Sentence Representations for Better Semantics and Faster Retrieval
• COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretraining
• A Survey of Visual Transformers
• SimCSE: Simple Contrastive Learning of Sentence Embeddings
• Extracting Sentence Embeddings from Pretrained Transformer Models
• On the Sentence Embeddings from Pre-trained Language Models
• PowerNorm: Rethinking Batch Normalization in Transformers
• A Discrete Variational Recurrent Topic Model without the Reparametrization Trick
• Understanding and Improving Encoder Layer Fusion in Sequence-to-Sequence Learning
• Norm-Based Curriculum Learning for Neural Machine Translation
• Rejuvenating Low-Frequency Words: Making the Most of Parallel Data in Non-Autoregressive Translation
• Leveraging Advantages of Interactive and Non-Interactive Models for Vector-Based Cross-Lingual Information Retrieval
• Learning to Remove: Towards Isotropic Pre-trained BERT Embedding
• Tiny-NewsRec: Effective and Efficient PLM-based News Recommendation
• Language Models as Zero-shot Visual Semantic Learners
• All Bark and No Bite: Rogue Dimensions in Transformer Language Models Obscure Representational Quality
• How Does Fine-tuning Affect the Geometry of Embedding Space: A Case Study on Isotropy
• On Isotropy Calibration of Transformers