258 citations · 1.1k across the 13 of their papers we have counts for
21 papers
The Higher-order Stirling Triangles
William Y. C. Chen, Elena L. Wang
The th-order Stirling cycle and subset triangles and their associated quasi-Eulerian triangles were introduced by Deb and Sokal in their study of total positivity of combinatori…
OTel: Open Telco AI Datasets, Benchmarks, and Models
Farbod Tavakkoli, Gregory Diamos, Kenneth Church +19
We present Open Telco (OTel), an open telecom AI resource that releases derived telecom datasets for retrieval, reranking, instruction tuning, and safety/abstention, together with…
A Training-Time Diagnostic for Generalization via the Log-Alignment Ratio
Ali Shehper, Ashish Vaswani
We study the log-alignment ratio (LAR), a measure of parameter-activation alignment, introduced in parameterization theory. We reformulate it as the overlap between a weight spectr…
A Dataset for Metaphor Detection in Early Medieval Hebrew Poetry
Michael Toker, Oren Mishali, Ophir Münz-Manor +2
There is a large volume of late antique and medieval Hebrew texts. They represent a crucial linguistic and cultural bridge between Biblical and modern Hebrew. Poetry is prominent i…
Unifying Grokking and Double Descent
Xander Davies, Lauro Langosco, David Krueger
A principled understanding of generalization in deep learning may require unifying disparate observations under a single conceptual framework. Previous work has studied \emph{grokk…
Leave Graphs Alone: Addressing Over-Squashing without Rewiring
Domenico Tortorella, Alessio Micheli
Recent works have investigated the role of graph bottlenecks in preventing long-range information propagation in message-passing graph neural networks, causing the so-called `over-…
Efficient Content-Based Sparse Attention with Routing Transformers
Aurko Roy, Mohammad Saffar, Ashish Vaswani +1
Self-attention has recently been adopted for a wide range of sequence modeling problems. Despite its effectiveness, self-attention suffers from quadratic compute and memory require…
Stay on the Path: Instruction Fidelity in Vision-and-Language Navigation
Vihan Jain, Gabriel Magalhaes, Alexander Ku +3
Advances in learning and representations have reinvigorated work that connects language to other modalities. A particularly exciting direction is Vision-and-Language Navigation(VLN…
Stand-Alone Self-Attention in Vision Models
Prajit Ramachandran, Niki Parmar, Ashish Vaswani +3
Convolutions are a fundamental building block of modern computer vision systems. Recent approaches have argued for going beyond convolutions in order to capture long-range dependen…
Attention Augmented Convolutional Networks
Irwan Bello, Barret Zoph, Ashish Vaswani +2
Convolutional networks have been the paradigm of choice in many computer vision applications. The convolution operation however has a significant weakness in that it only operates…
Mesh-TensorFlow: Deep Learning for Supercomputers
Noam Shazeer, Youlong Cheng, Niki Parmar +9
Batch-splitting (data-parallelism) is the dominant distributed Deep Neural Network (DNN) training strategy, due to its universal applicability and its amenability to Single-Program…
Music Transformer
Cheng-Zhi Anna Huang, Ashish Vaswani, Jakob Uszkoreit +7
Music relies heavily on repetition to build structure and meaning. Self-reference occurs on multiple timescales, from motifs to phrases to reusing of entire sections of music, such…
Relational inductive biases, deep learning, and graph networks
Peter W. Battaglia, Jessica B. Hamrick, Victor Bapst +24
Artificial intelligence (AI) has undergone a renaissance recently, making major progress in key domains such as vision, language, control, and decision-making. This has been due, i…
Theory and Experiments on Vector Quantized Autoencoders
Aurko Roy, Ashish Vaswani, Arvind Neelakantan +1
Deep neural networks with discrete latent variables offer the promise of better symbolic reasoning, and learning abstractions that are more useful to new tasks. There has been a su…
Self-Attention with Relative Position Representations
Peter Shaw, Jakob Uszkoreit, Ashish Vaswani
Relying entirely on an attention mechanism, the Transformer introduced by Vaswani et al. (2017) achieves state-of-the-art results for machine translation. In contrast to recurrent…
Tensor2Tensor for Neural Machine Translation
Ashish Vaswani, Samy Bengio, Eugene Brevdo +10
Tensor2Tensor is a library for deep learning models that is well-suited for neural machine translation and includes the reference implementation of the state-of-the-art Transformer…
Fast Decoding in Sequence Models using Discrete Latent Variables
Łukasz Kaiser, Aurko Roy, Ashish Vaswani +4
Autoregressive sequence models based on deep neural networks, such as RNNs, Wavenet and the Transformer attain state-of-the-art results on many tasks. However, they are difficult t…
Image Transformer
Niki Parmar, Ashish Vaswani, Jakob Uszkoreit +4
Image generation has been successfully cast as an autoregressive sequence generation or transformation problem. Recent work has shown that self-attention is an effective way of mod…
One Model To Learn Them All
Lukasz Kaiser, Aidan N. Gomez, Noam Shazeer +4
Deep learning yields great results across many fields, from speech recognition, image classification, to translation. But for each problem, getting a deep model to work well involv…
Attention Is All You Need
Ashish Vaswani, Noam Shazeer, Niki Parmar +5
The dominant sequence transduction models are based on complex recurrent or convolutional neural networks in an encoder-decoder configuration. The best performing models also conne…
Unsupervised Neural Hidden Markov Models
Ke Tran, Yonatan Bisk, Ashish Vaswani +2
In this work, we present the first results for neuralizing an Unsupervised Hidden Markov Model. We evaluate our approach on tag in- duction. Our approach outperforms existing gener…