Late Attention Layers Alone Can Copy Entity Tokens, but Not Without Attending to Their Context

Muyu He, Yuchen Liu, Ran Tao +1

Large language models (LLMs) reliably perform entity copying, in which a model copies tokens referring to an entity, termed entity tokens, from the prompt into its output to answer…