43 citations · 111 across the 21 of their papers we have counts for
21 papers
Audio-Visual Flamingo: Open Audio-Visual Intelligence for Long and Complex Videos
Sreyan Ghosh, Arushi Goel, Kaousheik Jayakumar +19
We present Audio-Visual Flamingo (AV-Flamingo), a fully open state-of-the-art audio-visual large language model (AV-LLM) for joint understanding and reasoning over audio, images, a…
Cosmos 3: Omnimodal World Models for Physical AI
NVIDIA, :, Aditi +293
We introduce Cosmos 3, a family of omnimodal world models designed to jointly process and generate language, image, video, audio, and action sequences within a unified mixture-of-t…
Benchmarking Single-Factor Physical Video-to-Audio Generation
Tingle Li, Siddharth Gururani, Kevin J. Shih +6
Generative video-to-audio (V2A) models produce highly plausible soundtracks, but it remains unclear whether they capture the underlying physical processes. Existing evaluations emp…
Audio Flamingo Next: Next-Generation Open Audio-Language Models for Speech, Sound, and Music
Sreyan Ghosh, Arushi Goel, Kaousheik Jayakumar +15
We present Audio Flamingo Next (AF-Next), the next-generation and most capable large audio-language model in the Audio Flamingo series, designed to advance understanding and reason…
MMOU: A Massive Multi-Task Omni Understanding and Reasoning Benchmark for Long and Complex Real-World Videos
Arushi Goel, Sreyan Ghosh, Vatsal Agarwal +16
Multimodal Large Language Models (MLLMs) have shown strong performance in visual and audio understanding when evaluated in isolation. However, their ability to jointly reason over…
World Simulation with Video Foundation Models for Physical AI
NVIDIA, :, Arslan Ali +87
We introduce [Cosmos-Predict2.5], the latest generation of the Cosmos World Foundation Models for Physical AI. Built on a flow-based architecture, [Cosmos-Predict2.5] unifies Text2…
Cosmos-Reason1: From Physical Common Sense To Embodied Reasoning
NVIDIA, :, Alisson Azzolini +51
Physical AI systems need to perceive, understand, and perform complex actions in the physical world. In this paper, we present the Cosmos-Reason1 models that can understand the phy…
Cosmos World Foundation Model Platform for Physical AI
NVIDIA, :, Niket Agarwal +76
Physical AI needs to be trained digitally first. It needs a digital twin of itself, the policy model, and a digital twin of the world, the world model. In this paper, we present th…
Edify Image: High-Quality Image Generation with Pixel Space Laplacian Diffusion Models
NVIDIA, :, Yuval Atzmon +29
We introduce Edify Image, a family of diffusion models capable of generating photorealistic image content with pixel-perfect accuracy. Edify Image utilizes cascaded pixel-space dif…
Symbolic Music Generation with Non-Differentiable Rule Guided Diffusion
Yujia Huang, Adishree Ghatare, Yuanzhe Liu +6
We study the problem of symbolic music generation (e.g., generating piano rolls), with a technical focus on non-differentiable rule guidance. Musical rules are often expressed in s…
Multilingual Multiaccented Multispeaker TTS with RADTTS
Rohan Badlani, Rafael Valle, Kevin J. Shih +3
We work to create a multilingual speech synthesis system which can generate speech with the proper accent while retaining the characteristics of an individual voice. This is challe…
SPACE: Speech-driven Portrait Animation with Controllable Expression
Siddharth Gururani, Arun Mallya, Ting-Chun Wang +2
Animating portraits using speech has received growing attention in recent years, with various creative and practical use cases. An ideal generated video should have good lip sync w…
Anomalous behaviour in loss-gradient based interpretability methods
Vinod Subramanian, Siddharth Gururani, Emmanouil Benetos +1
Loss-gradients are used to interpret the decision making process of deep learning models. In this work, we evaluate loss-gradient based attribution methods by occluding parts of th…
Semi-Supervised Audio Classification with Partially Labeled Data
Siddharth Gururani, Alexander Lerch
Audio classification has seen great progress with the increasing availability of large-scale datasets. These large datasets, however, are often only partially labeled as collecting…
An Interdisciplinary Review of Music Performance Analysis
Alexander Lerch, Claire Arthur, Ashis Pati +1
A musical performance renders an acoustic realization of a musical score or other representation of a composition. Different performances of the same composition may vary in terms…
Score-informed Networks for Music Performance Assessment
Jiawen Huang, Yun-Ning Hung, Ashis Pati +2
The assessment of music performances in most cases takes into account the underlying musical score being performed. While there have been several automatic approaches for objective…
dMelodies: A Music Dataset for Disentanglement Learning
Ashis Pati, Siddharth Gururani, Alexander Lerch
Representation learning focused on disentangling the underlying factors of variation in given data has become an important area of research in machine learning. However, most of th…
Visual Attention for Musical Instrument Recognition
Karn Watcharasupat, Siddharth Gururani, Alexander Lerch
In the field of music information retrieval, the task of simultaneously identifying the presence or absence of multiple musical instruments in a polyphonic recording remains a hard…
Prosody Transfer in Neural Text to Speech Using Global Pitch and Loudness Features
Siddharth Gururani, Kilol Gupta, Dhaval Shah +2
This paper presents a simple yet effective method to achieve prosody transfer from a reference speech signal to synthesized speech. The main idea is to incorporate well-known acous…
An Attention Mechanism for Musical Instrument Recognition
Siddharth Gururani, Mohit Sharma, Alexander Lerch
While the automatic recognition of musical instruments has seen significant progress, the task is still considered hard for music featuring multiple instruments as opposed to singl…
Music Performance Analysis: A Survey
Alexander Lerch, Claire Arthur, Ashis Pati +1
Music Information Retrieval (MIR) tends to focus on the analysis of audio signals. Often, a single music recording is used as representative of a "song" even though different perfo…