PreCog: Exploring the Relation between Memorization and Performance in Pre-trained Language Models

Abstract

Pre-trained Language Models such as BERT are impressive machines with the ability to memorize, possibly generalized learning examples. We present here a small, focused contribution to the analysis of the interplay between memorization and performance of BERT in downstream tasks. We propose PreCog, a measure for evaluating memorization from pre-training, and we analyze its correlation with the BERT's performance. Our experiments show that highly memorized examples are better classified, suggesting memorization is an essential key to success for BERT.

References in corpus (9)

• Adam: A Method for Stochastic Optimization
• Overcoming catastrophic forgetting in neural networks
• LLaMA: Open and Efficient Foundation Language Models
• The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks
• Natural Language Understanding with the Quora Question Pairs Dataset
• Counterfactual Memorization in Neural Language Models
• Understanding Unintended Memorization in Federated Learning
• A Trip Towards Fairness: Bias and De-Biasing in Large Language Models
• The Dark Side of the Language: Pre-trained Transformers in the DarkNet

Cited by in corpus (2)

• Clinical Insights: A Comprehensive Review of Language Models in Medicine
• The Dark Side of the Language: Pre-trained Transformers in the DarkNet