It Ain't That Bad: Understanding the Mysterious Performance Drop in OOD Generalization for Generative Transformer Models

Abstract

Large language models (LLMs) have achieved remarkable proficiency on solving diverse problems. However, their generalization ability is not always satisfying and the generalization problem is common for generative transformer models in general. Researchers take basic mathematical tasks like n-digit addition or multiplication as important perspectives for investigating their generalization behaviors. It is observed that when training models on n-digit operations (e.g., additions) in which both input operands are n-digit in length, models generalize successfully on unseen n-digit inputs (in-distribution (ID) generalization), but fail miserably on longer, unseen cases (out-of-distribution (OOD) generalization). We bring this unexplained performance drop into attention and ask whether there is systematic OOD generalization. Towards understanding LLMs, we train various smaller language models which may share the same underlying mechanism. We discover that the strong ID generalization stems from structured representations, while behind the unsatisfying OOD performance, the models still exhibit clear learned algebraic structures. Specifically, these models map unseen OOD inputs to outputs with learned equivalence relations in the ID domain, which we call the equivalence generalization. These findings deepen our knowledge regarding the generalizability of generative models including LLMs, and provide insights into potential avenues for improvement.

arXiv commentProceedings of the 33rd International Joint Conference on Artificial Intelligence (IJCAI-24)

References in corpus (12)

• Training language models to follow instructions with human feedback
• LLaMA: Open and Efficient Foundation Language Models
• Llama 2: Open Foundation and Fine-Tuned Chat Models
• PaLM: Scaling Language Modeling with Pathways
• Sparks of Artificial General Intelligence: Early experiments with GPT-4
• Entropy-based Logic Explanations of Neural Networks
• Exploring Length Generalization in Large Language Models
• Investigating the Limitations of Transformers with Simple Arithmetic Tasks
• Towards Understanding Grokking: An Effective Theory of Representation Learning
• The Clock and the Pizza: Two Stories in Mechanistic Explanation of Neural Networks
• Teaching Arithmetic to Small Transformers
• Length Generalization in Arithmetic Transformers