LZ Penalty: An information-theoretic repetition penalty for autoregressive language models

Antonio A. Ginart, Naveen Kodali, Jason Lee +3

We introduce the LZ penalty, a penalty specialized for reducing degenerate repetitions in autoregressive language models without loss of capability. The penalty is based on the cod…