70 citations · 151 across the 20 of their papers we have counts for
27 papers
VoiceChat-TTS: A Low-Latency Continuous Speech Synthesis Model for Interactive Agents
Edresson Casanova, Jaehyeon Kim, Mariana Graterol Fuenmayor +17
Spoken dialogue is a natural form of human--computer interaction, yet most speech language models remain limited to turn-based operation and lack real-time adaptability, such as us…
Precise Video-to-Audio Generation with Cross-Modal Alignment in Latent Space
Thanh V. T. Tran, Ngoc-Son Nguyen, Luong Tran +4
Video-to-audio (V2A) generation aims to synthesize realistic audio that is both semantically consistent with and temporally synchronized to a silent video. Despite recent progress,…
MagpieTTS-LF: Inference-Time Long-Form Speech Generation Without Training on Long-Form data
Subhankar Ghosh, Jason Li, Paarth Neekhara +4
Neural Text-to-Speech (TTS) systems achieve remarkable quality on short utterances but long-form speech generation shows prosodic drift, speaker inconsistencies and sentence bounda…
Align2Speak: Improving TTS for Low Resource Languages via ASR-Guided Online Preference Optimization
Shehzeen Hussain, Paarth Neekhara, Xuesong Yang +7
Developing high-quality text-to-speech (TTS) systems for low-resource languages is challenging due to the scarcity of paired text and speech data. In contrast, automatic speech rec…
Frame-Stacked Local Transformers For Efficient Multi-Codebook Speech Generation
Roy Fejgin, Paarth Neekhara, Xuesong Yang +6
Speech generation models based on large language models (LLMs) typically operate on discrete acoustic codes, which differ fundamentally from text tokens due to their multicodebook…
NanoCodec: Towards High-Quality Ultra Fast Speech LLM Inference
Edresson Casanova, Paarth Neekhara, Ryan Langman +6
Large Language Models (LLMs) have significantly advanced audio processing by leveraging audio codecs to discretize audio into tokens, enabling the application of language modeling…
HiFiTTS-2: A Large-Scale High Bandwidth Speech Dataset
Ryan Langman, Xuesong Yang, Paarth Neekhara +4
This paper introduces HiFiTTS-2, a large-scale speech dataset designed for high-bandwidth speech synthesis. The dataset is derived from LibriVox audiobooks, and contains approximat…
Koel-TTS: Enhancing LLM based Speech Generation with Preference Alignment and Classifier Free Guidance
Shehzeen Hussain, Paarth Neekhara, Xuesong Yang +6
While autoregressive speech token generation models produce speech with remarkable variety and naturalness, their inherent lack of controllability often results in issues such as h…
Low Frame-rate Speech Codec: a Codec Designed for Fast High-quality Speech LLM Training and Inference
Edresson Casanova, Ryan Langman, Paarth Neekhara +5
Large language models (LLMs) have significantly advanced audio processing through audio codecs that convert audio into discrete tokens, enabling the application of language modelin…
Improving Robustness of LLM-based Speech Synthesis by Learning Monotonic Alignment
Paarth Neekhara, Shehzeen Hussain, Subhankar Ghosh +4
Large Language Model (LLM) based text-to-speech (TTS) systems have demonstrated remarkable capabilities in handling large speech datasets and generating natural speech for new spea…
REMARK-LLM: A Robust and Efficient Watermarking Framework for Generative Large Language Models
Ruisi Zhang, Shehzeen Samarah Hussain, Paarth Neekhara +1
We present REMARK-LLM, a novel efficient, and robust watermarking framework designed for texts generated by large language models (LLMs). Synthesizing human-like content using LLMs…
SelfVC: Voice Conversion With Iterative Refinement using Self Transformations
Paarth Neekhara, Shehzeen Hussain, Rafael Valle +5
We propose SelfVC, a training strategy to iteratively improve a voice conversion model with self-synthesized examples. Previous efforts on voice conversion focus on factorizing spe…
ACE-VC: Adaptive and Controllable Voice Conversion using Explicitly Disentangled Self-supervised Speech Representations
Shehzeen Hussain, Paarth Neekhara, Jocelyn Huang +2
In this work, we propose a zero-shot voice conversion method using speech representations trained with self-supervised learning. First, we develop a multi-task model to decompose a…
FastStamp: Accelerating Neural Steganography and Digital Watermarking of Images on FPGAs
Shehzeen Hussain, Nojan Sheybani, Paarth Neekhara +3
Steganography and digital watermarking are the tasks of hiding recoverable data in image pixels. Deep neural network (DNN) based image steganography and watermarking techniques are…
ReFace: Real-time Adversarial Attacks on Face Recognition Systems
Shehzeen Hussain, Todd Huster, Chris Mesterharm +5
Deep neural network based face recognition models have been shown to be vulnerable to adversarial examples. However, many of the past attacks require the adversary to solve an inpu…
FaceSigns: Semi-Fragile Neural Watermarks for Media Authentication and Countering Deepfakes
Paarth Neekhara, Shehzeen Hussain, Xinqiao Zhang +3
Deepfakes and manipulated media are becoming a prominent threat due to the recent advances in realistic image and video synthesis techniques. There have been several attempts at co…
Adapting TTS models For New Speakers using Transfer Learning
Paarth Neekhara, Jason Li, Boris Ginsburg
Training neural text-to-speech (TTS) models for a new speaker typically requires several hours of high quality speech data. Prior works on voice cloning attempt to address this cha…
WaveGuard: Understanding and Mitigating Audio Adversarial Examples
Shehzeen Hussain, Paarth Neekhara, Shlomo Dubnov +2
There has been a recent surge in adversarial attacks on deep learning based automatic speech recognition (ASR) systems. These attacks pose new challenges to deep learning security…
Cross-modal Adversarial Reprogramming
Paarth Neekhara, Shehzeen Hussain, Jinglong Du +3
With the abundance of large-scale deep learning models, it has become possible to repurpose pre-trained networks for new tasks. Recent works on adversarial reprogramming have shown…
Expressive Neural Voice Cloning
Paarth Neekhara, Shehzeen Hussain, Shlomo Dubnov +2
Voice cloning is the task of learning to synthesize the voice of an unseen speaker from a few samples. While current voice cloning methods achieve promising results in Text-to-Spee…
Adversarial Threats to DeepFake Detection: A Practical Perspective
Paarth Neekhara, Brian Dolhansky, Joanna Bitton +1
Facially manipulated images and videos or DeepFakes can be used maliciously to fuel misinformation or defame individuals. Therefore, detecting DeepFakes is crucial to increase the…
FastWave: Accelerating Autoregressive Convolutional Neural Networks on FPGA
Shehzeen Hussain, Mojan Javaheripi, Paarth Neekhara +2
Autoregressive convolutional neural networks (CNNs) have been widely exploited for sequence generation tasks such as audio synthesis, language modeling and neural machine translati…
Adversarial Deepfakes: Evaluating Vulnerability of Deepfake Detectors to Adversarial Examples
Shehzeen Hussain, Paarth Neekhara, Malhar Jere +2
Recent advances in video manipulation techniques have made the generation of fake videos more accessible than ever before. Manipulated videos can fuel disinformation and reduce tru…
Universal Adversarial Perturbations for Speech Recognition Systems
Paarth Neekhara, Shehzeen Hussain, Prakhar Pandey +3
In this work, we demonstrate the existence of universal adversarial audio perturbations that cause mis-transcription of audio signals by automatic speech recognition (ASR) systems.…