4 papers
LiveFoley: Dynamic Real-Time Video-to-Audio Generation for Immersive Interactive Worlds
Yusheng Dai, Vladimir Iashin, Kangdi Wang +6
Streaming video-to-audio (V2A) generation is essential for immersive interactive applications. However, real-time constraints leave existing streaming V2A methods far behind offlin…
Spot, Separate, and Enhance: Fully Generative Approach for Audio Mixing
Ilpo Viertola, Giulio Cengarle, Gouthaman KV +2
We introduce Spot, Separate, and Enhance (SSE), the first multimodal, user-guided generative model for audio remixing and enhancement. SSE enhances video content by rebalancing the…
Video Object Segmentation-Aware Audio Generation
Ilpo Viertola, Vladimir Iashin, Esa Rahtu
Existing multimodal audio generation models often lack precise user control, which limits their applicability in professional Foley workflows. In particular, these models focus on…
Temporally Aligned Audio for Video with Autoregression
Ilpo Viertola, Vladimir Iashin, Esa Rahtu
We introduce V-AURA, the first autoregressive model to achieve high temporal alignment and relevance in video-to-audio generation. V-AURA uses a high-framerate visual feature extra…