Publications (8)
Meta Lattice: Model Space Redesign for Cost-Effective Industry-Scale Ads Recommendations
Liang Luo, Yuxin Chen, Zhengyu Zhang +39
The rapidly evolving landscape of products, surfaces, policies, and regulations poses significant challenges for deploying state-of-the-art recommendation models at industry scale,…
Arithmetic of weighted Catalan numbers
Yibo Gao, Andrew Gu
In this paper, we study arithmetic properties of weighted Catalan numbers. Previously, Postnikov and Sagan found conditions under which the -adic valuations of the weighted Cata…
GaLore 2: Large-Scale LLM Pre-Training by Gradient Low-Rank Projection
DiJia Su, Andrew Gu, Jane Xu +2
Large language models (LLMs) have revolutionized natural language understanding and generation but face significant memory bottlenecks during training. GaLore, Gradient Low-Rank Pr…
TorchTitan: One-stop PyTorch native solution for production ready LLM pre-training
Wanchao Liang, Tianyu Liu, Less Wright +10
The development of large language models (LLMs) has been instrumental in advancing state-of-the-art natural language processing applications. Training LLMs with billions of paramet…
The Llama 3 Herd of Models
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri +556
Modern artificial intelligence (AI) systems are powered by foundation models. This paper presents a new set of foundation models, called Llama 3. It is a herd of language models th…
PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel
Yanli Zhao, Andrew Gu, Rohan Varma +15
It is widely acknowledged that large models have the potential to deliver superior performance across a broad range of domains. Despite the remarkable progress made in the field of…
SimpleFSDP: Simpler Fully Sharded Data Parallel with torch.compile
Ruisi Zhang, Tianyu Liu, Will Feng +4
Distributed training of large models consumes enormous computation resources and requires substantial engineering efforts to compose various training techniques. This paper present…
Deep Transfer Learning for Infectious Disease Case Detection Using Electronic Medical Records
Ye Ye, Andrew Gu
During an infectious disease pandemic, it is critical to share electronic medical records or models (learned from these records) across regions. Applying one region's data/model to…