1.2k citations · 2.7k across the 27 of their papers we have counts for
28 papers
CausalLM is not optimal for in-context learning
Nan Ding, Tomer Levinboim, Jialin Wu +2
Recent empirical evidence indicates that transformer based in-context learning performs better when using a prefix language model (prefixLM), in which in-context samples can all at…
Improving Robust Generalization by Direct PAC-Bayesian Bound Minimization
Zifan Wang, Nan Ding, Tomer Levinboim +2
Recent research in robust optimization has shown an overfitting-like phenomenon in which models trained against adversarial attacks exhibit higher robustness on the training set co…
PaLI: A Jointly-Scaled Multilingual Language-Image Model
Xi Chen, Xiao Wang, Soravit Changpinyo +26
Effective scaling and a flexible task interface enable large language models to excel at many tasks. We present PaLI (Pathways Language and Image model), a model that extends this…
All You May Need for VQA are Image Captions
Soravit Changpinyo, Doron Kukliansky, Idan Szpektor +3
Visual Question Answering (VQA) has benefited from increasingly sophisticated models, but has not enjoyed the same level of engagement in terms of data creation. In this paper, we…
PACTran: PAC-Bayesian Metrics for Estimating the Transferability of Pretrained Models to Classification Tasks
Nan Ding, Xi Chen, Tomer Levinboim +2
With the increasing abundance of pretrained models in recent years, the problem of selecting the best pretrained checkpoint for a particular downstream classification task has been…
Bridging the Gap Between Practice and PAC-Bayes Theory in Few-Shot Meta-Learning
Nan Ding, Xi Chen, Tomer Levinboim +2
Despite recent advances in its theoretical understanding, there still remains a significant gap in the ability of existing PAC-Bayesian theories on meta-learning to explain perform…
Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts
Soravit Changpinyo, Piyush Sharma, Nan Ding +1
The availability of large-scale image captioning and visual question answering datasets has contributed significantly to recent successes in vision-and-language pre-training. Howev…
Do Transformer Modifications Transfer Across Implementations and Applications?
Sharan Narang, Hyung Won Chung, Yi Tay +13
The research community has proposed copious modifications to the Transformer architecture since it was introduced over three years ago, relatively few of which have seen widespread…
Improving Text Generation Evaluation with Batch Centering and Tempered Word Mover Distance
Xi Chen, Nan Ding, Tomer Levinboim +1
Recent advances in automatic evaluation metrics for text have shown that deep contextualized word representations, such as those generated by BERT encoders, are helpful for designi…
TeaForN: Teacher-Forcing with N-grams
Sebastian Goodman, Nan Ding, Radu Soricut
Sequence generation models trained with teacher-forcing suffer from issues related to exposure bias and lack of differentiability across timesteps. Our proposed method, Teacher-For…
Attention that does not Explain Away
Nan Ding, Xinjie Fan, Zhenzhong Lan +2
Models based on the Transformer architecture have achieved better accuracy than the ones based on competing architectures for a large set of tasks. A unique feature of the Transfor…
Talking-Heads Attention
Noam Shazeer, Zhenzhong Lan, Youlong Cheng +2
We introduce "talking-heads attention" - a variation on multi-head attention which includes linearprojections across the attention-heads dimension, immediately before and after the…
SHAPED: Shared-Private Encoder-Decoder for Text Style Adaptation
Ye Zhang, Nan Ding, Radu Soricut
Supervised training of abstractive language generation models results in learning conditional probabilities over language sequences based on the supervised training signal. When th…
Cold-Start Reinforcement Learning with Softmax Policy Gradient
Nan Ding, Radu Soricut
Policy-gradient approaches to reinforcement learning have two common and undesirable overhead procedures, namely warm-start training and sample variance reduction. In this paper, w…
Understanding Image and Text Simultaneously: a Dual Vision-Language Machine Comprehension Task
Nan Ding, Sebastian Goodman, Fei Sha +1
We introduce a new multi-modal task for computer systems, posed as a combined vision-language comprehension challenge: identifying the most suitable text describing a scene, given…
Multilingual Word Embeddings using Multigraphs
Radu Soricut, Nan Ding
We present a family of neural-network--inspired models for computing continuous word representations, specifically designed to exploit both monolingual and multilingual text. This…
Building Large Machine Reading-Comprehension Datasets using Paragraph Vectors
Radu Soricut, Nan Ding
We present a dual contribution to the task of machine reading-comprehension: a technique for creating large-sized machine-comprehension (MC) datasets using paragraph-vector models;…
On the Convergence of Stochastic Gradient MCMC Algorithms with High-Order Integrators
Changyou Chen, Nan Ding, Lawrence Carin
Recent advances in Bayesian learning with large-scale data have witnessed emergence of stochastic gradient MCMC algorithms (SG-MCMC), such as stochastic gradient Langevin dynamics…
Stochastic Gradient MCMC with Stale Gradients
Changyou Chen, Nan Ding, Chunyuan Li +2
Stochastic gradient MCMC (SG-MCMC) has played an important role in large-scale Bayesian learning, with well-developed theoretical convergence properties. In such applications of SG…
Characterizing Quantum Supremacy in Near-Term Devices
Sergio Boixo, Sergei V. Isakov, Vadim N. Smelyanskiy +6
A critical question for the field of quantum computing in the near future is whether quantum devices without error correction can perform a well-defined computational task beyond t…
What is the Computational Value of Finite Range Tunneling?
Vasil S. Denchev, Sergio Boixo, Sergei V. Isakov +5
Quantum annealing (QA) has been proposed as a quantum enhanced optimization heuristic exploiting tunneling. Here, we demonstrate how finite range tunneling can provide considerable…
Scalable Quantum Simulation of Molecular Energies
P. J. J. O'Malley, R. Babbush, I. D. Kivlichan +27
We report the first electronic structure calculation performed on a quantum computer without exponentially costly precompilation. We use a programmable array of superconducting qub…
Totally Corrective Boosting with Cardinality Penalization
Vasil S. Denchev, Nan Ding, Shin Matsushima +2
We propose a totally corrective boosting algorithm with explicit cardinality regularization. The resulting combinatorial optimization problems are not known to be efficiently solva…
Probabilistic Label Relation Graphs with Ising Models
Nan Ding, Jia Deng, Kevin Murphy +1
We consider classification problems in which the label space has structure. A common example is hierarchical label spaces, corresponding to the case where one label subsumes anothe…