Efficient Continual Learning for Small Language Models with a Discrete Key-Value Bottleneck

Andor Diera, Lukas Galke, Fabian Karl +1

Continual learning remains a challenge across various natural language processing (NLP) tasks, as models updated with new training data often risk catastrophic forgetting of previo…