164 citations · 461 across the 37 of their papers we have counts for
37 papers
Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Time
Zichang Liu, Jue Wang, Tri Dao +8
Large language models (LLMs) with hundreds of billions of parameters have sparked a new wave of exciting AI applications. However, they are computationally expensive at inference t…
RLCD: Reinforcement Learning from Contrastive Distillation for Language Model Alignment
Kevin Yang, Dan Klein, Asli Celikyilmaz +2
We propose Reinforcement Learning from Contrastive Distillation (RLCD), a method for aligning language models to follow principles expressed in natural language (e.g., to be more h…
Landscape Surrogate: Learning Decision Losses for Mathematical Optimization Under Partial Information
Arman Zharmagambetov, Brandon Amos, Aaron Ferber +3
Recent works in learning-integrated optimization have shown promise in settings where the optimization problem is only partially observed or where general-purpose optimizers perfor…
Extending Context Window of Large Language Models via Positional Interpolation
Shouyuan Chen, Sherman Wong, Liangjian Chen +1
We present Position Interpolation (PI) that extends the context window sizes of RoPE-based pretrained LLMs such as LLaMA models to up to 32768 with minimal fine-tuning (within 1000…
HO: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models
Zhenyu Zhang, Ying Sheng, Tianyi Zhou +9
Large Language Models (LLMs), despite their recent impressive accomplishments, are notably cost-prohibitive to deploy, particularly for applications involving long-content generati…
Pre-train and Search: Efficient Embedding Table Sharding with Pre-trained Neural Cost Models
Daochen Zha, Louis Feng, Liang Luo +8
Sharding a large machine learning model across multiple devices to balance the costs is important in distributed training. This is challenging because partitioning is NP-hard, and…
Scan and Snap: Understanding Training Dynamics and Token Composition in 1-layer Transformer
Yuandong Tian, Yiping Wang, Beidi Chen +1
Transformer architecture has shown impressive performance in multiple research domains and has become the backbone of many neural network models. However, there is limited understa…
A Cookbook of Self-Supervised Learning
Randall Balestriero, Mark Ibrahim, Vlad Sobal +16
Self-supervised learning, dubbed the dark matter of intelligence, is a promising path to advance machine learning. Yet, much like cooking, training SSL methods is a delicate art wi…
Searching Large Neighborhoods for Integer Linear Programs with Contrastive Learning
Taoan Huang, Aaron Ferber, Yuandong Tian +2
Integer Linear Programs (ILPs) are powerful tools for modeling and solving a large number of combinatorial optimization problems. Recently, it has been shown that Large Neighborhoo…
Sample-efficient Surrogate Model for Frequency Response of Linear PDEs using Self-Attentive Complex Polynomials
Andrew Cohen, Weiping Dou, Jiang Zhu +7
Linear Partial Differential Equations (PDEs) govern the spatial-temporal dynamics of physical systems that are essential to building modern technology. When working with linear PDE…
Learning Compiler Pass Orders using Coreset and Normalized Value Prediction
Youwei Liang, Kevin Stone, Ali Shameli +8
Finding the optimal pass sequence of compilation can lead to a significant reduction in program size and/or improvement in program efficiency. Prior works on compilation pass order…
DOC: Improving Long Story Coherence With Detailed Outline Control
Kevin Yang, Dan Klein, Nanyun Peng +1
We propose the Detailed Outline Control (DOC) framework for improving long-range plot coherence when automatically generating several-thousand-word-long stories. DOC consists of tw…
Local Branching Relaxation Heuristics for Integer Linear Programs
Taoan Huang, Aaron Ferber, Yuandong Tian +2
Large Neighborhood Search (LNS) is a popular heuristic algorithm for solving combinatorial optimization problems (COP). It starts with an initial solution to the problem and iterat…
EurNet: Efficient Multi-Range Relational Modeling of Spatial Multi-Relational Data
Minghao Xu, Yuanfan Guo, Yi Xu +3
Modeling spatial relationship in the data remains critical across many different tasks, such as image classification, semantic segmentation and protein structure understanding. Pre…
Re3: Generating Longer Stories With Recursive Reprompting and Revision
Kevin Yang, Yuandong Tian, Nanyun Peng +1
We consider the problem of automatically generating longer stories of over two thousand words. Compared to prior work on shorter stories, long-range plot coherence and relevance ar…
DreamShard: Generalizable Embedding Table Placement for Recommender Systems
Daochen Zha, Louis Feng, Qiaoyu Tan +6
We study embedding table placement for distributed recommender systems, which aims to partition and place the tables on multiple hardware devices (e.g., GPUs) to balance the comput…
SurCo: Learning Linear Surrogates For Combinatorial Nonlinear Optimization Problems
Aaron Ferber, Taoan Huang, Daochen Zha +4
Optimization problems with nonlinear cost functions and combinatorial constraints appear in many real-world applications but remain challenging to solve efficiently compared to the…
AutoShard: Automated Embedding Table Sharding for Recommender Systems
Daochen Zha, Louis Feng, Bhargav Bhushanam +7
Embedding learning is an important technique in deep recommendation models to map categorical features to dense vectors. However, the embedding tables often demand an extremely lar…
Efficient Planning in a Compact Latent Action Space
Zhengyao Jiang, Tianjun Zhang, Michael Janner +4
Planning-based reinforcement learning has shown strong performance in tasks in discrete and low-dimensional continuous action spaces. However, planning usually brings significant c…
AutoCAT: Reinforcement Learning for Automated Exploration of Cache-Timing Attacks
Mulong Luo, Wenjie Xiong, Geunbae Lee +6
The aggressive performance optimizations in modern microprocessors can result in security vulnerabilities. For example, timing-based attacks in processor caches can steal secret ke…
Denoised MDPs: Learning World Models Better Than the World Itself
Tongzhou Wang, Simon S. Du, Antonio Torralba +3
The ability to separate signal from noise, and reason with clean abstractions, is critical to intelligence. With this ability, humans can efficiently perform real world tasks witho…
Understanding the Role of Nonlinearity in Training Dynamics of Contrastive Learning
Yuandong Tian
While the empirical success of self-supervised learning (SSL) heavily relies on the usage of deep nonlinear models, existing theoretical works on SSL understanding still focus on l…
On the Importance of Asymmetry for Siamese Representation Learning
Xiao Wang, Haoqi Fan, Yuandong Tian +2
Many recent self-supervised frameworks for visual representation learning are based on certain forms of Siamese networks. Such networks are conceptually symmetric with two parallel…
Understanding Curriculum Learning in Policy Optimization for Online Combinatorial Optimization
Runlong Zhou, Zelin He, Yuandong Tian +2
Over the recent years, reinforcement learning (RL) starts to show promising results in tackling combinatorial optimization (CO) problems, in particular when coupled with curriculum…