What Should Embeddings Embed? Autoregressive Models Represent Latent Generating Distributions

Liyi Zhang, Michael Y. Li, R. Thomas McCoy +3

Autoregressive language models have demonstrated a remarkable ability to extract latent structure from text. The embeddings from large language models have been shown to capture as…