187 citations · 698 across the 55 of their papers we have counts for
73 papers
Error-aware Quantization through Noise Tempering
Zheng Wang, Juncheng B Li, Shuhui Qu +2
Quantization has become a predominant approach for model compression, enabling deployment of large models trained on GPUs onto smaller form-factor devices for inference. Quantizati…
Normalized Contrastive Learning for Text-Video Retrieval
Yookoon Park, Mahmoud Azab, Bo Xiong +4
Cross-modal contrastive learning has led the recent advances in multimodal retrieval with its simplicity and effectiveness. In this work, however, we reveal that cross-modal contra…
Token-level Sequence Labeling for Spoken Language Understanding using Compositional End-to-End Models
Siddhant Arora, Siddharth Dalmia, Brian Yan +3
End-to-end spoken language understanding (SLU) systems are gaining popularity over cascaded approaches due to their simplicity and ability to avoid error propagation. However, thes…
SQuAT: Sharpness- and Quantization-Aware Training for BERT
Zheng Wang, Juncheng B Li, Shuhui Qu +2
Quantization is an effective technique to reduce memory footprint, inference latency, and power consumption of deep learning models. However, existing quantization methods suffer f…
CTC Alignments Improve Autoregressive Translation
Brian Yan, Siddharth Dalmia, Yosuke Higuchi +4
Connectionist Temporal Classification (CTC) is a widely used approach for automatic speech recognition (ASR) that performs conditionally independent monotonic alignment. However fo…
ASR2K: Speech Recognition for Around 2000 Languages without Audio
Xinjian Li, Florian Metze, David R Mortensen +2
Most recent speech recognition models rely on large supervised datasets, which are unavailable for many low-resource languages. In this work, we present a speech recognition pipeli…
Masked Autoencoders that Listen
Po-Yao Huang, Hu Xu, Juncheng Li +5
This paper studies a simple extension of image-based Masked Autoencoders (MAE) to self-supervised representation learning from audio spectrograms. Following the Transformer encoder…
LegoNN: Building Modular Encoder-Decoder Models
Siddharth Dalmia, Dmytro Okhonko, Mike Lewis +5
State-of-the-art encoder-decoder models (e.g. for machine translation (MT) or automatic speech recognition (ASR)) are constructed and trained end-to-end as an atomic unit. No compo…
On Advances in Text Generation from Images Beyond Captioning: A Case Study in Self-Rationalization
Shruti Palaskar, Akshita Bhagia, Yonatan Bisk +3
Combining the visual modality with pretrained language models has been surprisingly effective for simple descriptive tasks such as image captioning. More general text generation ho…
Robustness of Neural Architectures for Audio Event Detection
Juncheng B Li, Zheng Wang, Shuhui Qu +1
Traditionally, in Audio Recognition pipeline, noise is suppressed by the "frontend", relying on preprocessing techniques such as speech enhancement. However, it is not guaranteed t…
On Adversarial Robustness of Large-scale Audio Visual Learning
Juncheng B Li, Shuhui Qu, Xinjian Li +2
As audio-visual systems are being deployed for safety-critical tasks such as surveillance and malicious content filtering, their robustness remains an under-studied area. Existing…
AudioTagging Done Right: 2nd comparison of deep learning methods for environmental sound classification
Juncheng B Li, Shuhui Qu, Po-Yao Huang +1
After its sweeping success in vision and language tasks, pure attention-based neural architectures (e.g. DeiT) are emerging to the top of audio tagging (AT) leaderboards, which see…
Speech Summarization using Restricted Self-Attention
Roshan Sharma, Shruti Palaskar, Alan W Black +1
Speech summarization is typically performed by using a cascade of speech recognition and text summarization models. End-to-end modeling of speech summarization models is challengin…
Differentiable Allophone Graphs for Language-Universal Speech Recognition
Brian Yan, Siddharth Dalmia, David R. Mortensen +2
Building language-universal speech recognition systems entails producing phonological units of spoken sound that can be shared across languages. While speech annotations at the lan…
Rethinking End-to-End Evaluation of Decomposable Tasks: A Case Study on Spoken Language Understanding
Siddhant Arora, Alissa Ostapenko, Vijay Viswanathan +4
Decomposable tasks are complex and comprise of a hierarchy of sub-tasks. Spoken intent prediction, for example, combines automatic speech recognition and natural language understan…
Searchable Hidden Intermediates for End-to-End Models of Decomposable Sequence Tasks
Siddharth Dalmia, Brian Yan, Vikas Raunak +2
End-to-end approaches for sequence tasks are becoming increasingly popular. Yet for complex sequence tasks, like speech translation, systems that cascade several models trained on…
VLM: Task-agnostic Video-Language Model Pre-training for Video Understanding
Hu Xu, Gargi Ghosh, Po-Yao Huang +5
We present a simplified, task-agnostic multi-modal pre-training approach that can accept either video or text input, or both for a variety of end tasks. Existing pre-training are t…
Multilingual Multimodal Pre-training for Zero-Shot Cross-Lingual Transfer of Vision-Language Models
Po-Yao Huang, Mandela Patrick, Junjie Hu +3
This paper studies zero-shot cross-lingual transfer of vision-language models. Specifically, we focus on multilingual text-to-video search and propose a Transformer-based model tha…
Self-supervised object detection from audio-visual correspondence
Triantafyllos Afouras, Yuki M. Asano, Francois Fagan +2
We tackle the problem of learning object detectors without supervision. Differently from weakly-supervised object detection, we do not assume image-level class labels. Instead, we…
Space-Time Crop & Attend: Improving Cross-modal Video Representation Learning
Mandela Patrick, Yuki M. Asano, Bernie Huang +4
The quality of the image representations obtained from self-supervised learning depends strongly on the type of data augmentations used in the learning formulation. Recent papers h…
NoiseQA: Challenge Set Evaluation for User-Centric Question Answering
Abhilasha Ravichander, Siddharth Dalmia, Maria Ryskina +3
When Question-Answering (QA) systems are deployed in the real world, users query them through a variety of interfaces, such as speaking to voice assistants, typing questions into a…
Audio-Visual Event Recognition through the lens of Adversary
Juncheng B Li, Kaixin Ma, Shuhui Qu +2
As audio/visual classification models are widely deployed for sensitive tasks like content filtering at scale, it is critical to understand their robustness along with improving th…
Multimodal Speech Recognition with Unstructured Audio Masking
Tejas Srinivasan, Ramon Sanabria, Florian Metze +1
Visual context has been shown to be useful for automatic speech recognition (ASR) systems when the speech signal is noisy or corrupted. Previous work, however, has only demonstrate…
On Long-Tailed Phenomena in Neural Machine Translation
Vikas Raunak, Siddharth Dalmia, Vivek Gupta +1
State-of-the-art Neural Machine Translation (NMT) models struggle with generating low-frequency tokens, tackling which remains a major challenge. The analysis of long-tailed phenom…