ENTP: Encoder-only Next Token Prediction

Ethan Ewer, Daewon Chae, Thomas Zeng +2

Next-token prediction is conventionally done using decoder-only Transformers with causal attention, as this approach allows for efficient reuse of keys and values. What if we were…