7 citations · 8 across the 6 of their papers we have counts for
6 papers
Matching Features, Not Tokens: Energy-Based Fine-Tuning of Language Models
Samy Jelassi, Mujin Kwun, Rosie Zhao +5
Cross-entropy (CE) training provides dense and scalable supervision for language models, but it optimizes next-token prediction under teacher forcing rather than sequence-level beh…
LoRA Soups: Merging LoRAs for Practical Skill Composition Tasks
Akshara Prabhakar, Yuanzhi Li, Karthik Narasimhan +3
Low-Rank Adaptation (LoRA) is a popular technique for parameter-efficient fine-tuning of Large Language Models (LLMs). We study how different LoRA modules can be merged to achieve…
Mixture of Parrots: Experts improve memorization more than reasoning
Samy Jelassi, Clara Mohri, David Brandfonbrener +7
The Mixture-of-Experts (MoE) architecture enables a significant increase in the total number of model parameters with minimal computational overhead. However, it is not clear what…
How Does Overparameterization Affect Features?
Ahmet Cagri Duzgun, Samy Jelassi, Yuanzhi Li
Overparameterization, the condition where models have more parameters than necessary to fit their training loss, is a crucial factor for the success of deep learning. However, the…
Dissecting adaptive methods in GANs
Samy Jelassi, David Dobre, Arthur Mensch +2
Adaptive methods are a crucial component widely used for training generative adversarial networks (GANs). While there has been some work to pinpoint the "marginal value of adaptive…
Towards understanding how momentum improves generalization in deep learning
Samy Jelassi, Yuanzhi Li
Stochastic gradient descent (SGD) with momentum is widely used for training modern deep learning architectures. While it is well-understood that using momentum can lead to faster c…