Provably Learning Multi-Head Attention with Queries

Sunyeop Kim, Insung Kim, Jian Guo

We study the problem of learning multi-head softmax attention from black-box input-output access. The learner may query arbitrary real-valued token sequences and observe only the s…