How Do Neural Sequence Models Generalize? Local and Global Context Cues for Out-of-Distribution Prediction

Anthony Bau, Jacob Andreas

After a neural sequence model encounters an unexpected token, can its behavior be predicted? We show that RNN and transformer language models exhibit structured, consistent general…