1.6k citations · 1.9k across the 32 of their papers we have counts for
33 papers
SystolicAttention: Fusing FlashAttention within a Single Systolic Array
Jiawei Lin, Yuanlong Li, Guokai Chen +1
Transformer models rely heavily on the scaled dot-product attention (SDPA) operation, typically implemented as FlashAttention. Characterized by its frequent interleaving of matrix…
Textbooks Are All You Need II: phi-1.5 technical report
Yuanzhi Li, Sébastien Bubeck, Ronen Eldan +3
We continue the investigation into the power of smaller Transformer-based language models as initiated by \textbf{TinyStories} -- a 10 million parameter model that can produce cohe…
Efficient RLHF: Reducing the Memory Usage of PPO
Michael Santacroce, Yadong Lu, Han Yu +2
Reinforcement Learning with Human Feedback (RLHF) has revolutionized language modeling by aligning models with human preferences. However, the RL stage, Proximal Policy Optimizatio…
Generating Faithful Text From a Knowledge Graph with Noisy Reference Text
Tahsina Hashem, Weiqing Wang, Derry Tanti Wijaya +2
Knowledge Graph (KG)-to-Text generation aims at generating fluent natural-language text that accurately represents the information of a given knowledge graph. While significant pro…
The Implicit Bias of Batch Normalization in Linear Models and Two-layer Linear Convolutional Neural Networks
Yuan Cao, Difan Zou, Yuanzhi Li +1
We study the implicit bias of batch normalization trained by gradient descent. We show that when learning a linear model with batch normalization for binary classification, gradien…
Length Generalization in Arithmetic Transformers
Samy Jelassi, Stéphane d'Ascoli, Carles Domingo-Enrich +3
We examine how transformers cope with two challenges: learning basic integer arithmetic, and generalizing to longer sequences than seen during training. We find that relative posit…
Textbooks Are All You Need
Suriya Gunasekar, Yi Zhang, Jyoti Aneja +16
We introduce phi-1, a new large language model for code, with significantly smaller size than competing models: phi-1 is a Transformer-based model with 1.3B parameters, trained for…
Specifying and Solving Robust Empirical Risk Minimization Problems Using CVXPY
Eric Luxenberg, Dhruv Malik, Yuanzhi Li +2
We consider robust empirical risk minimization (ERM), where model parameters are chosen to minimize the worst-case empirical loss when each data point varies over a given convex un…
NormMark: A Weakly Supervised Markov Model for Socio-cultural Norm Discovery
Farhad Moghimifar, Shilin Qu, Tongtong Wu +2
Norms, which are culturally accepted guidelines for behaviours, can be integrated into conversational models to generate utterances that are appropriate for the socio-cultural cont…
TinyStories: How Small Can Language Models Be and Still Speak Coherent English?
Ronen Eldan, Yuanzhi Li
Language models (LMs) are powerful tools for natural language processing, but they often struggle to produce coherent and fluent text when they are small. Models with around 125M p…
Plan, Eliminate, and Track -- Language Models are Good Teachers for Embodied Agents
Yue Wu, So Yeon Min, Yonatan Bisk +5
Pre-trained large language models (LLMs) capture procedural knowledge about the world. Recent work has leveraged LLM's ability to generate abstract plans to simplify challenging co…
SPRING: Studying the Paper and Reasoning to Play Games
Yue Wu, Shrimai Prabhumoye, So Yeon Min +5
Open-world survival games pose significant challenges for AI algorithms due to their multi-tasking, deep exploration, and goal prioritization requirements. Despite reinforcement le…
Towards Expressive Spectral-Temporal Graph Neural Networks for Time Series Forecasting
Ming Jin, Guangsi Shi, Yuan-Fang Li +7
Time series forecasting has remained a focal point due to its vital applications in sectors such as energy management and transportation planning. Spectral-temporal graph neural ne…
Normalizing Flow-based Neural Process for Few-Shot Knowledge Graph Completion
Linhao Luo, Yuan-Fang Li, Gholamreza Haffari +1
Knowledge graphs (KGs), as a structured form of knowledge representation, have been widely applied in the real world. Recently, few-shot knowledge graph completion (FKGC), which ai…
Sparks of Artificial General Intelligence: Early experiments with GPT-4
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan +11
Artificial intelligence (AI) researchers have been developing and refining large language models (LLMs) that exhibit remarkable capabilities across a variety of domains and tasks,…
On the Importance of Contrastive Loss in Multimodal Learning
Yunwei Ren, Yuanzhi Li
Recently, contrastive learning approaches (e.g., CLIP (Radford et al., 2021)) have received huge success in multimodal learning, where the model tries to minimize the distance betw…
The Benefits of Mixup for Feature Learning
Difan Zou, Yuan Cao, Yuanzhi Li +1
Mixup, a simple data augmentation method that randomly mixes two data points via linear interpolation, has been extensively applied in various deep learning applications to gain be…
How Do Transformers Learn Topic Structure: Towards a Mechanistic Understanding
Yuchen Li, Yuanzhi Li, Andrej Risteski
While the successes of transformers across many domains are indisputable, accurate understanding of the learning mechanics is still largely lacking. Their capabilities have been pr…
What Matters In The Structured Pruning of Generative Language Models?
Michael Santacroce, Zixin Wen, Yelong Shen +1
Auto-regressive large language models such as GPT-3 require enormous computational resources to use. Traditionally, structured pruning methods are employed to reduce resource usage…
Read and Reap the Rewards: Learning to Play Atari with the Help of Instruction Manuals
Yue Wu, Yewen Fan, Paul Pu Liang +3
High sample complexity has long been a challenge for RL. On the other hand, humans learn to perform tasks not only from interaction or demonstrations, but also by reading unstructu…
On Robustness of Prompt-based Semantic Parsing with Large Pre-trained Language Model: An Empirical Study on Codex
Terry Yue Zhuo, Zhuang Li, Yujin Huang +4
Semantic parsing is a technique aimed at constructing a structured representation of the meaning of a natural-language question. Recent advancements in few-shot language models tra…
Complex Reading Comprehension Through Question Decomposition
Xiao-Yu Guo, Yuan-Fang Li, Gholamreza Haffari
Multi-hop reading comprehension requires not only the ability to reason over raw text but also the ability to combine multiple evidence. We propose a novel learning approach that h…
Towards Relation Extraction From Speech
Tongtong Wu, Guitao Wang, Jinming Zhao +4
Relation extraction typically aims to extract semantic relationships between entities from the unstructured text. One of the most essential data sources for relation extraction is…
Vision Transformers provably learn spatial structure
Samy Jelassi, Michael E. Sander, Yuanzhi Li
Vision Transformers (ViTs) have achieved comparable or superior performance than Convolutional Neural Networks (CNNs) in computer vision. This empirical breakthrough is even more r…