Repurposing Pre-trained LLMs as High Fidelity Continuous Text Autoencoders

Arkanath Pathak, Unnat Jain, Alexander C. Berg

Next-token prediction has enabled highly fluent autoregressive language models, but it represents global structure only indirectly through sequential factorization. In contrast, hi…