116 citations · 515 across the 22 of their papers we have counts for
23 papers
Recipes for Sequential Pre-training of Multilingual Encoder and Seq2Seq Models
Saleh Soltan, Andy Rosenbaum, Tobias Falke +3
Pre-trained encoder-only and sequence-to-sequence (seq2seq) models each have advantages, however training both model types from scratch is computationally expensive. We explore rec…
Scalable and Accurate Self-supervised Multimodal Representation Learning without Aligned Video and Text Data
Vladislav Lialin, Stephen Rawls, David Chan +3
Scaling up weakly-supervised datasets has shown to be highly effective in the image-text domain and has contributed to most of the recent state-of-the-art computer vision and multi…
Low-Resource Compositional Semantic Parsing with Concept Pretraining
Subendhu Rongali, Mukund Sridhar, Haidar Khan +3
Semantic parsing plays a key role in digital voice assistants such as Alexa, Siri, and Google Assistant by mapping natural language to structured meaning representations. When we w…
CLASP: Few-Shot Cross-Lingual Data Augmentation for Semantic Parsing
Andy Rosenbaum, Saleh Soltan, Wael Hamza +3
A bottleneck to developing Semantic Parsing (SP) models is the need for a large volume of human-labeled training data. Given the complexity and cost of human annotation for SP, lab…
LINGUIST: Language Model Instruction Tuning to Generate Annotated Utterances for Intent Classification and Slot Tagging
Andy Rosenbaum, Saleh Soltan, Wael Hamza +2
We present LINGUIST, a method for generating annotated data for Intent Classification and Slot Tagging (IC+ST), via fine-tuning AlexaTM 5B, a 5-billion-parameter multilingual seque…
AlexaTM 20B: Few-Shot Learning Using a Large-Scale Multilingual Seq2Seq Model
Saleh Soltan, Shankar Ananthakrishnan, Jack FitzGerald +13
In this work, we demonstrate that multilingual large-scale sequence-to-sequence (seq2seq) models, pre-trained on a mixture of denoising and Causal Language Modeling (CLM) tasks, ar…
Alexa Teacher Model: Pretraining and Distilling Multi-Billion-Parameter Encoders for Natural Language Understanding Systems
Jack FitzGerald, Shankar Ananthakrishnan, Konstantine Arkoudas +38
We present results from a large-scale experiment on pretraining encoders with non-embedding parameter counts ranging from 700M to 9.3B, their subsequent distillation into smaller m…
Training Naturalized Semantic Parsers with Very Little Data
Subendhu Rongali, Konstantine Arkoudas, Melanie Rubino +1
Semantic parsing is an important NLP problem, particularly for voice assistants such as Alexa and Google Assistant. State-of-the-art (SOTA) semantic parsers are seq2seq architectur…
Instilling Type Knowledge in Language Models via Multi-Task QA
Shuyang Li, Mukund Sridhar, Chandana Satya Prakash +3
Understanding human language often necessitates understanding entities and their place in a taxonomy of knowledge -- their types. Previous methods to learn entity types rely on tra…
Zero-shot Generalization in Dialog State Tracking through Generative Question Answering
Shuyang Li, Jin Cao, Mukund Sridhar +4
Dialog State Tracking (DST), an integral part of modern dialog systems, aims to track user preferences and constraints (slots) in task-oriented dialogs. In real-world settings with…
Exploring Transfer Learning For End-to-End Spoken Language Understanding
Subendhu Rongali, Beiye Liu, Liwei Cai +3
Voice Assistants such as Alexa, Siri, and Google Assistant typically use a two-stage Spoken Language Understanding pipeline; first, an Automatic Speech Recognition (ASR) component…
Delexicalized Paraphrase Generation
Boya Yu, Konstantine Arkoudas, Wael Hamza
We present a neural model for paraphrasing and train it to generate delexicalized sentences. We achieve this by creating training data in which each input is paired with a number o…
Style Attuned Pre-training and Parameter Efficient Fine-tuning for Spoken Language Understanding
Jin Cao, Jun Wang, Wael Hamza +2
Neural models have yielded state-of-the-art results in deciphering spoken language understanding (SLU) problems; however, these models require a significant amount of domain-specif…
Don't Parse, Insert: Multilingual Semantic Parsing with Insertion Based Decoding
Qile Zhu, Haidar Khan, Saleh Soltan +2
Semantic parsing is one of the key components of natural language understanding systems. A successful parse transforms an input utterance to an action that is easily understood by…
Don't Parse, Generate! A Sequence to Sequence Architecture for Task-Oriented Semantic Parsing
Subendhu Rongali, Luca Soldaini, Emilio Monti +1
Virtual assistants such as Amazon Alexa, Apple Siri, and Google Assistant often rely on a semantic parsing component to understand which action(s) to execute for an utterance spoke…
Improving Spoken Language Understanding By Exploiting ASR N-best Hypotheses
Mingda Li, Weitong Ruan, Xinyue Liu +3
In a modern spoken language understanding (SLU) system, the natural language understanding (NLU) module takes interpretations of a speech from the automatic speech recognition (ASR…
Neural Cross-Lingual Coreference Resolution and its Application to Entity Linking
Gourab Kundu, Avirup Sil, Radu Florian +1
We propose an entity-centric neural cross-lingual coreference model that builds on multi-lingual embeddings and language-independent features. We perform both intrinsic and extrins…
Neural Cross-Lingual Entity Linking
Avirup Sil, Gourab Kundu, Radu Florian +1
A major challenge in Entity Linking (EL) is making effective use of contextual information to disambiguate mentions to Wikipedia that might refer to different entities in different…
A Unified Query-based Generative Model for Question Generation and Question Answering
Linfeng Song, Zhiguo Wang, Wael Hamza
We propose a query-based generative model for solving both tasks of question generation (QG) and question an- swering (QA). The model follows the classic encoder- decoder framework…
-Nearest Neighbor Augmented Neural Networks for Text Classification
Zhiguo Wang, Wael Hamza, Linfeng Song
In recent years, many deep-learning based models are proposed for text classification. This kind of models well fits the training set from the statistical point of view. However, i…
Reinforcement Learning for Transition-Based Mention Detection
Georgiana Dinu, Wael Hamza, Radu Florian
This paper describes an application of reinforcement learning to the mention detection task. We define a novel action-based formulation for the mention detection task, in which a m…
Bilateral Multi-Perspective Matching for Natural Language Sentences
Zhiguo Wang, Wael Hamza, Radu Florian
Natural language sentence matching is a fundamental technology for a variety of tasks. Previous approaches either match sentences from a single direction or only apply single granu…
Multi-Perspective Context Matching for Machine Comprehension
Zhiguo Wang, Haitao Mi, Wael Hamza +1
Previous machine comprehension (MC) datasets are either too small to train end-to-end deep learning models, or not difficult enough to evaluate the ability of current MC techniques…