6 papers
StellarTTS: Sparse Temporal Embedding for Low-Latency and Robust Speech Synthesis
Kaicheng Luo, Xuefei Gong, Yutao Sun +6
The trade-off between robustness, latency, and prosody critically challenges text-to-speech (TTS) systems. Autoregressive models, despite fidelity, are slow and error-prone; non-au…
AudioCALM: Continuous Autoregressive Language Modeling for Universal Audio Generation
Huadai Liu, Kaicheng Luo, Wen Wang +4
Unifying speech, sound, and music generation in one model is hindered by tradeoffs between fidelity, end-to-end training, in-context conditioning, and variable-length synthesis tha…
STAR-VAE: Structured Topology-Aware Regularization for Audio Reconstruction and Generation
Huadai Liu, Wen Wang, Kaicheng Luo +3
Continuous Variational Autoencoders (VAEs) serve as the fundamental continuous tokenizer for modern neural audio generation systems, enabling high-fidelity reconstruction while pro…
PrismAudio: Decomposed Chain-of-Thoughts and Multi-dimensional Rewards for Video-to-Audio Generation
Huadai Liu, Kaicheng Luo, Wen Wang +6
Video-to-Audio (V2A) generation requires balancing four critical perceptual dimensions: semantic consistency, audio-visual temporal synchrony, aesthetic quality, and spatial accura…
ThinkSound: Chain-of-Thought Reasoning in Multimodal Large Language Models for Audio Generation and Editing
Huadai Liu, Kaicheng Luo, Jialei Wang +4
While end-to-end video-to-audio generation has greatly improved, producing high-fidelity audio that authentically captures the nuances of visual content remains challenging. Like p…
OmniAudio: Generating Spatial Audio from 360-Degree Video
Huadai Liu, Tianyi Luo, Kaicheng Luo +11
Traditional video-to-audio generation techniques primarily focus on perspective video and non-spatial audio, often missing the spatial cues necessary for accurately representing so…