Analyzing limits for in-context learning
arXiv:2502.03503
Abstract
Our paper challenges claims from prior research that transformer-based models, when learning in context, implicitly implement standard learning algorithms. We present empirical evidence inconsistent with this view and provide a mathematical analysis demonstrating that transformers cannot achieve general predictive accuracy due to inherent architectural limitations.
39th Conference on Neural Information Processing Systems (NeurIPS 2025) Workshop What Can t Transformers Do?