Next Token or Vector? Learning How Far to Decode

Sai Sreenivas Chintha, Chidera Biringa, Rishab Sharma +6

Standard autoregressive decoding advances one token at a time, even when several upcoming tokens can be predicted together. We propose CADENCE (Context-Adaptive DEcoding), a genera…