Learning and Unlearning of Fabricated Knowledge in Language Models

Chen Sun, Nolan Andrew Miller, Andrey Zhmoginov +2

What happens when a new piece of knowledge is introduced into the training data and how long does it last while a large language model (LM) continues to train? We investigate this…