GQ-VAE: A gated quantized VAE for learning variable length tokens

Theo Datta, Kayla Huang, Sham Kakade +1

While most frontier models still use deterministic frequency-based tokenization algorithms such as byte-pair encoding (BPE), there has been significant recent work to design learne…