644 citations · 782 across the 16 of their papers we have counts for
21 papers
CoLT5: Faster Long-Range Transformers with Conditional Computation
Joshua Ainslie, Tao Lei, Michiel de Jong +9
Many natural language processing tasks benefit from long inputs, but processing long documents with Transformers is expensive -- not only due to quadratic attention complexity but…
Simple Recurrence Improves Masked Language Models
Tao Lei, Ran Tian, Jasmijn Bastings +1
In this work, we explore whether modeling recurrence into the Transformer architecture can both be beneficial and efficient, by building an extremely simple recurrent module into t…
Training Language Models with Memory Augmentation
Zexuan Zhong, Tao Lei, Danqi Chen
Recent work has improved language models (LMs) remarkably by equipping them with a non-parametric memory component. However, most existing approaches only introduce mem-ories at te…
SRU++: Pioneering Fast Recurrence with Attention for Speech Recognition
Jing Pan, Tao Lei, Kwangyoun Kim +2
The Transformer architecture has been well adopted as a dominant architecture in most sequence transduction tasks including automatic speech recognition (ASR), since its attention…
Nutribullets Hybrid: Multi-document Health Summarization
Darsh J Shah, Lili Yu, Tao Lei +1
We present a method for generating comparative summaries that highlights similarities and contradictions in input documents. The key challenge in creating such summaries is the lac…
Nutri-bullets: Summarizing Health Studies by Composing Segments
Darsh J Shah, Lili Yu, Tao Lei +1
We introduce \emph{Nutri-bullets}, a multi-document summarization task for health and nutrition. First, we present two datasets of food and health summaries from multiple scientifi…
When Attention Meets Fast Recurrence: Training Language Models with Reduced Compute
Tao Lei
Large language models have become increasingly difficult to train because of the growing computation time and cost. In this work, we present SRU++, a highly-efficient architecture…
Autoregressive Knowledge Distillation through Imitation Learning
Alexander Lin, Jeremy Wohlwend, Howard Chen +1
The performance of autoregressive models on natural language generation tasks has dramatically improved due to the adoption of deep, self-attentive architectures. However, these ga…
Rationalizing Text Matching: Learning Sparse Alignments via Optimal Transport
Kyle Swanson, Lili Yu, Tao Lei
Selecting input features of top relevance has become a popular method for building self-explaining models. In this work, we extend this selective rationalization approach to text m…
ASAPP-ASR: Multistream CNN and Self-Attentive SRU for SOTA Speech Recognition
Jing Pan, Joshua Shapiro, Jeremy Wohlwend +3
In this paper we present state-of-the-art (SOTA) performance on the LibriSpeech corpus with two novel neural network architectures, a multistream CNN for acoustic modeling and a se…
Metric Learning for Dynamic Text Classification
Jeremy Wohlwend, Ethan R. Elenberg, Samuel Altschul +2
Traditional text classifiers are limited to predicting over a fixed set of labels. However, in many real-world applications the label set is frequently changing. For example, in in…
Interactive Classification by Asking Informative Questions
Lili Yu, Howard Chen, Sida Wang +2
We study the potential for interaction in natural language classification. We add a limited form of interaction for intent classification, where users provide an initial query usin…
Structured Pruning of Large Language Models
Ziheng Wang, Jeremy Wohlwend, Tao Lei
Large language models have recently achieved state of the art performance across a wide variety of natural language tasks. Meanwhile, the size of these models and their latency hav…
Building a Production Model for Retrieval-Based Chatbots
Kyle Swanson, Lili Yu, Christopher Fox +2
Response suggestion is an important task for building human-computer conversation systems. Recent approaches to conversation modeling have introduced new model architectures with i…
Adversarial Domain Adaptation for Duplicate Question Detection
Darsh J Shah, Tao Lei, Alessandro Moschitti +2
We address the problem of detecting duplicate questions in forums, which is an important step towards automating the process of answering new questions. As finding and annotating s…
Simple Recurrent Units for Highly Parallelizable Recurrence
Tao Lei, Yu Zhang, Sida I. Wang +2
Common recurrent neural architectures scale poorly due to the intrinsic difficulty in parallelizing their state computations. In this work, we propose the Simple Recurrent Unit (SR…
Style Transfer from Non-Parallel Text by Cross-Alignment
Tianxiao Shen, Tao Lei, Regina Barzilay +1
This paper focuses on style transfer on the basis of non-parallel text. This is an instance of a broad family of problems including machine translation, decipherment, and sentiment…
Deriving Neural Architectures from Sequence and Graph Kernels
Tao Lei, Wengong Jin, Regina Barzilay +1
The design of neural architectures for structured objects is typically guided by experimental insights rather than a formal process. In this work, we appeal to kernels over combina…
Rationalizing Neural Predictions
Tao Lei, Regina Barzilay, Tommi Jaakkola
Prediction without justification has limited applicability. As a remedy, we learn to extract pieces of input text as justifications -- rationales -- that are tailored to be short a…
Semi-supervised Question Retrieval with Gated Convolutions
Tao Lei, Hrishikesh Joshi, Regina Barzilay +4
Question answering forums are rapidly growing in size with no effective automated ability to refer to and reuse answers already available for previous posted questions. In this pap…
Molding CNNs for text: non-linear, non-consecutive convolutions
Tao Lei, Regina Barzilay, Tommi Jaakkola
The success of deep learning often derives from well-chosen operational building blocks. In this work, we revise the temporal convolution operation in CNNs to better adapt it to te…