29 citations · 120 across the 28 of their papers we have counts for
24 papers · 1 filter
The Order Effect: Investigating Prompt Sensitivity to Input Order in LLMs
Bryan Guan, Tanya Roosta, Peyman Passban +1
As large language models (LLMs) become integral to diverse applications, ensuring their reliability under varying input conditions is crucial. One key issue affecting this reliabil…
Dynamic Position Encoding for Transformers
Joyce Zheng, Mehdi Rezagholizadeh, Peyman Passban
Recurrent models have been dominating the field of neural machine translation (NMT) for the past few years. Transformers \citep{vaswani2017attention}, have radically changed it by…
NATURE: Natural Auxiliary Text Utterances for Realistic Spoken Language Evaluation
David Alfonso-Hermelo, Ahmad Rashid, Abbas Ghaddar +2
Slot-filling and intent detection are the backbone of conversational agents such as voice assistants, and are active areas of research. Even though state-of-the-art techniques on p…
RAIL-KD: RAndom Intermediate Layer Mapping for Knowledge Distillation
Md Akmal Haidar, Nithin Anchuri, Mehdi Rezagholizadeh +3
Intermediate layer knowledge distillation (KD) can improve the standard KD technique (which only targets the output of teacher and student models) especially over large pre-trained…
Knowledge Distillation with Noisy Labels for Natural Language Understanding
Shivendra Bhardwaj, Abbas Ghaddar, Ahmad Rashid +5
Knowledge Distillation (KD) is extensively used to compress and deploy large pre-trained language models on edge devices for real-world applications. However, one neglected area of…
How to Select One Among All? An Extensive Empirical Study Towards the Robustness of Knowledge Distillation in Natural Language Understanding
Tianda Li, Ahmad Rashid, Aref Jafari +3
Knowledge Distillation (KD) is a model compression algorithm that helps transfer the knowledge of a large neural network into a smaller one. Even though KD has shown promise on a w…
KroneckerBERT: Learning Kronecker Decomposition for Pre-trained Language Models via Knowledge Distillation
Marzieh S. Tahaei, Ella Charlaix, Vahid Partovi Nia +2
The development of over-parameterized pre-trained language models has made a significant contribution toward the success of natural language processing. While over-parameterization…
End-to-End Self-Debiasing Framework for Robust NLU Training
Abbas Ghaddar, Philippe Langlais, Mehdi Rezagholizadeh +1
Existing Natural Language Understanding (NLU) models have been shown to incorporate dataset biases leading to strong performance on in-distribution (ID) test sets but poor performa…
Context-aware Adversarial Training for Name Regularity Bias in Named Entity Recognition
Abbas Ghaddar, Philippe Langlais, Ahmad Rashid +1
In this work, we examine the ability of NER models to use contextual information when predicting the type of an ambiguous entity. We introduce NRB, a new testbed carefully designed…
Not Far Away, Not So Close: Sample Efficient Nearest Neighbour Data Augmentation via MiniMax
Ehsan Kamalloo, Mehdi Rezagholizadeh, Peyman Passban +1
In Natural Language Processing (NLP), finding data augmentation techniques that can produce high-quality human-interpretable examples has always been challenging. Recently, leverag…
MATE-KD: Masked Adversarial TExt, a Companion to Knowledge Distillation
Ahmad Rashid, Vasileios Lioutas, Mehdi Rezagholizadeh
The advent of large pre-trained language models has given rise to rapid progress in the field of Natural Language Processing (NLP). While the performance of these models on standar…
Robust Embeddings Via Distributions
Kira A. Selby, Yinong Wang, Ruizhe Wang +4
Despite recent monumental advances in the field, many Natural Language Processing (NLP) models still struggle to perform adequately on noisy domains. We propose a novel probabilist…
Annealing Knowledge Distillation
Aref Jafari, Mehdi Rezagholizadeh, Pranav Sharma +1
Significant memory and computational requirements of large deep neural networks restrict their application on edge devices. Knowledge distillation (KD) is a prominent model compres…
From Fully Trained to Fully Random Embeddings: Improving Neural Machine Translation with Compact Word Embedding Tables
Krtin Kumar, Peyman Passban, Mehdi Rezagholizadeh +2
Embedding matrices are key components in neural natural language processing (NLP) models that are responsible to provide numerical representations of input tokens.\footnote{In this…
Towards Zero-Shot Knowledge Distillation for Natural Language Processing
Ahmad Rashid, Vasileios Lioutas, Abbas Ghaddar +1
Knowledge Distillation (KD) is a common knowledge transfer algorithm used for model compression across a variety of deep learning based natural language processing (NLP) solutions.…
ALP-KD: Attention-Based Layer Projection for Knowledge Distillation
Peyman Passban, Yimeng Wu, Mehdi Rezagholizadeh +1
Knowledge distillation is considered as a training and compression strategy in which two neural networks, namely a teacher and a student, are coupled together during training. The…
From Unsupervised Machine Translation To Adversarial Text Generation
Ahmad Rashid, Alan Do-Omri, Md. Akmal Haidar +2
We present a self-attention based bilingual adversarial text generator (B-GAN) which can learn to generate text from the encoder representation of an unsupervised neural machine tr…
A Simplified Fully Quantized Transformer for End-to-end Speech Recognition
Alex Bie, Bharat Venkitesh, Joao Monteiro +2
While significant improvements have been made in recent years in terms of end-to-end automatic speech recognition (ASR) performance, such improvements were obtained through the use…
Fully Quantized Transformer for Machine Translation
Gabriele Prato, Ella Charlaix, Mehdi Rezagholizadeh
State-of-the-art neural machine translation methods employ massive amounts of parameters. Drastically reducing computational costs of such methods without affecting performance has…
Improving Word Embedding Factorization for Compression Using Distilled Nonlinear Neural Decomposition
Vasileios Lioutas, Ahmad Rashid, Krtin Kumar +2
Word-embeddings are vital components of Natural Language Processing (NLP) models and have been extensively explored. However, they consume a lot of memory which poses a challenge f…
EditNTS: An Neural Programmer-Interpreter Model for Sentence Simplification through Explicit Editing
Yue Dong, Zichao Li, Mehdi Rezagholizadeh +1
We present the first sentence simplification model that learns explicit edit operations (ADD, DELETE, and KEEP) via a neural programmer-interpreter approach. Most current neural se…
TextKD-GAN: Text Generation using KnowledgeDistillation and Generative Adversarial Networks
Md. Akmal Haidar, Mehdi Rezagholizadeh
Text generation is of particular interest in many NLP applications such as machine translation, language modeling, and text summarization. Generative adversarial networks (GANs) ac…
Latent Code and Text-based Generative Adversarial Networks for Soft-text Generation
Md. Akmal Haidar, Mehdi Rezagholizadeh, Alan Do-Omri +1
Text generation with generative adversarial networks (GANs) can be divided into the text-based and code-based categories according to the type of signals used for discrimination. I…
SALSA-TEXT : self attentive latent space based adversarial text generation
Jules Gagnon-Marchand, Hamed Sadeghi, Md. Akmal Haidar +1
Inspired by the success of self attention mechanism and Transformer architecture in sequence transduction and image generation applications, we propose novel self attention-based a…