Can a Language Model Learn Facts Continually in Its Weights?

Charles O'Neill, Max Kirkby, Jonathon Liu +1

Continual learning is a long-standing capability gap between LLMs and humans. Writing new knowledge into a model's weights routinely causes it to forget old knowledge, commonly den…