117 citations · 222 across the 6 of their papers we have counts for
4 papers · 1 filter
TeraPipe: Token-Level Pipeline Parallelism for Training Large-Scale Language Models
Zhuohan Li, Siyuan Zhuang, Shiyuan Guo +4
Model parallelism has become a necessity for training modern large-scale deep language models. In this work, we identify a new and orthogonal dimension from existing model parallel…
Fast Structured Decoding for Sequence Models
Zhiqing Sun, Zhuohan Li, Haoqing Wang +3
Autoregressive sequence models achieve state-of-the-art performance in domains like machine translation. However, due to the autoregressive factorization nature, these models suffe…
Understanding and Improving Transformer From a Multi-Particle Dynamic System Point of View
Yiping Lu, Zhuohan Li, Di He +5
The Transformer architecture is widely used in natural language processing. Despite its success, the design principle of the Transformer remains elusive. In this paper, we provide…
Towards Binary-Valued Gates for Robust LSTM Training
Zhuohan Li, Di He, Fei Tian +4
Long Short-Term Memory (LSTM) is one of the most widely used recurrent structures in sequence modeling. It aims to use gates to control information flow (e.g., whether to skip some…