Provably Learning Attention with Queries

Satwik Bhattamishra, Kulin Shah, Michael Hahn +1

We study the problem of learning Transformer-based sequence models with black-box access to their outputs. In this setting, a learner may adaptively query the oracle with any seque…