Self-Attention Limits Working Memory Capacity of Transformer-Based Models

Dongyu Gong, Hantao Zhang

Recent work on Transformer-based large language models (LLMs) has revealed striking limits in their working memory capacity, similar to what has been found in human behavioral stud…