1 citations · 1 across the 5 of their papers we have counts for
5 papers
ProofOptimizer: Training Language Models to Simplify Proofs without Human Demonstrations
Alex Gu, Bartosz Piotrowski, Fabian Gloeckle +2
Neural theorem proving has advanced rapidly in the past year, reaching IMO gold-medalist capabilities and producing formal proofs that span thousands of lines. Although such proofs…
CWM: An Open-Weights LLM for Research on Code Generation with World Models
FAIR CodeGen team, Jade Copet, Quentin Carbonneaux +48
We release Code World Model (CWM), a 32-billion-parameter open-weights LLM, to advance research on code generation with world models. To improve code understanding beyond what can…
Interpreting Affine Recurrence Learning in GPT-style Transformers
Samarth Bhargav, Alexander Gu
Understanding the internal mechanisms of GPT-style transformers, particularly their capacity to perform in-context learning (ICL), is critical for advancing AI alignment and interp…
Mixture of Parrots: Experts improve memorization more than reasoning
Samy Jelassi, Clara Mohri, David Brandfonbrener +7
The Mixture-of-Experts (MoE) architecture enables a significant increase in the total number of model parameters with minimal computational overhead. However, it is not clear what…
Language Agnostic Code Embeddings
Saiteja Utpala, Alex Gu, Pin Yu Chen
Recently, code language models have achieved notable advancements in addressing a diverse array of essential code comprehension and generation tasks. Yet, the field lacks a compreh…