132 citations · 524 across the 20 of their papers we have counts for
20 papers
REST: Retrieval-Based Speculative Decoding
Zhenyu He, Zexuan Zhong, Tianle Cai +2
We introduce Retrieval-Based Speculative Decoding (REST), a novel algorithm designed to speed up language model generation. The key insight driving the development of REST is the o…
Scaling In-Context Demonstrations with Structured Attention
Tianle Cai, Kaixuan Huang, Jason D. Lee +1
The recent surge of large language models (LLMs) highlights their ability to perform in-context learning, i.e., "learning" to perform a task from a few demonstrations in the contex…
Reward Collapse in Aligning Large Language Models
Ziang Song, Tianle Cai, Jason D. Lee +1
The extraordinary capabilities of large language models (LLMs) such as ChatGPT and GPT-4 are in part unleashed by aligning them with reward models that are trained on human prefere…
Large Language Models as Tool Makers
Tianle Cai, Xuezhi Wang, Tengyu Ma +2
Recent research has highlighted the potential of large language models (LLMs) to improve their problem-solving capabilities with the aid of suitable external tools. In our work, we…
What Makes Convolutional Models Great on Long Sequence Modeling?
Yuhong Li, Tianle Cai, Yi Zhang +2
Convolutional models have been widely used in multiple domains. However, most existing models only use local convolution, making the model unable to handle long-range dependency ef…
Is Vertical Logistic Regression Privacy-Preserving? A Comprehensive Privacy Analysis and Beyond
Yuzheng Hu, Tianle Cai, Jinyong Shan +5
We consider vertical logistic regression (VLR) trained with mini-batch gradient descent -- a setting which has attracted growing interest among industries and proven to be useful i…
First Place Solution of KDD Cup 2021 & OGB Large-Scale Challenge Graph Prediction Track
Chengxuan Ying, Mingqi Yang, Shuxin Zheng +7
In this technical report, we present our solution of KDD Cup 2021 OGB Large-Scale Challenge - PCQM4M-LSC Track. We adopt Graphormer and ExpC as our basic models. We train each mode…
Stable, Fast and Accurate: Kernelized Attention with Relative Positional Encoding
Shengjie Luo, Shanda Li, Tianle Cai +6
The attention module, which is a crucial component in Transformer, cannot scale efficiently to long sequences due to its quadratic complexity. Many works focus on approximating the…
Do Transformers Really Perform Bad for Graph Representation?
Chengxuan Ying, Tianle Cai, Shengjie Luo +5
The Transformer architecture has become a dominant choice in many domains, such as natural language processing and computer vision. Yet, it has not achieved competitive performance…
Towards a Theoretical Framework of Out-of-Distribution Generalization
Haotian Ye, Chuanlong Xie, Tianle Cai +3
Generalization to out-of-distribution (OOD) data is one of the central problems in modern machine learning. Recently, there is a surge of attempts to propose algorithms that mainly…
A Theory of Label Propagation for Subpopulation Shift
Tianle Cai, Ruiqi Gao, Jason D. Lee +1
One of the central problems in machine learning is domain adaptation. Unlike past theoretical work, we consider a new model for subpopulation shift in the input or representation s…
Towards Certifying L-infinity Robustness using Neural Networks with L-inf-dist Neurons
Bohang Zhang, Tianle Cai, Zhou Lu +2
It is well-known that standard neural networks, even with a high classification accuracy, are vulnerable to small -norm bounded adversarial perturbations. Although man…
Sanity-Checking Pruning Methods: Random Tickets can Win the Jackpot
Jingtong Su, Yihang Chen, Tianle Cai +4
Network pruning is a method for reducing test-time computational resource requirements with minimal performance degradation. Conventional wisdom of pruning algorithms suggests that…
GraphNorm: A Principled Approach to Accelerating Graph Neural Network Training
Tianle Cai, Shengjie Luo, Keyulu Xu +3
Normalization is known to help the optimization of deep neural networks. Curiously, different architectures require specialized normalization methods. In this paper, we study what…
RANDOM MASK: Towards Robust Convolutional Neural Networks
Tiange Luo, Tianle Cai, Mengxiao Zhang +2
Robustness of neural networks has recently been highlighted by the adversarial examples, i.e., inputs added with well-designed perturbations which are imperceptible to humans but c…
Locally Differentially Private (Contextual) Bandits Learning
Kai Zheng, Tianle Cai, Weiran Huang +2
We study locally differentially private (LDP) bandits learning in this paper. First, we propose simple black-box reduction frameworks that can solve a large family of context-free…
Defective Convolutional Networks
Tiange Luo, Tianle Cai, Mengxiao Zhang +3
Robustness of convolutional neural networks (CNNs) has gained in importance on account of adversarial examples, i.e., inputs added as well-designed perturbations that are impercept…
Convergence of Adversarial Training in Overparametrized Neural Networks
Ruiqi Gao, Tianle Cai, Haochuan Li +3
Neural networks are vulnerable to adversarial examples, i.e. inputs that are imperceptibly perturbed from natural data and yet incorrectly classified by the network. Adversarial tr…
Adversarially Robust Generalization Just Requires More Unlabeled Data
Runtian Zhai, Tianle Cai, Di He +4
Neural network robustness has recently been highlighted by the existence of adversarial examples. Many previous works show that the learned networks do not perform well on perturbe…
Gram-Gauss-Newton Method: Learning Overparameterized Neural Networks for Regression Problems
Tianle Cai, Ruiqi Gao, Jikai Hou +5
First-order methods such as stochastic gradient descent (SGD) are currently the standard algorithm for training deep neural networks. Second-order methods, despite their better con…