33 citations · 69 across the 13 of their papers we have counts for
14 papers
Do Joint Audio-Video Generation Models Understand Physics?
Zijun Cui, Xiulong Liu, Hao Fang +8
Joint audio-video generation models are rapidly approaching professional production quality, raising a central question: do they understand audio-visual physics, or merely generate…
SAVVY: Spatial Awareness via Audio-Visual LLMs through Seeing and Hearing
Mingfei Chen, Zijun Cui, Xiulong Liu +4
3D spatial reasoning in dynamic, audio-visual environments is a cornerstone of human cognition yet remains largely unexplored by existing Audio-Visual Large Language Models (AV-LLM…
Hearing Anywhere in Any Environment
Xiulong Liu, Anurag Kumar, Paul Calamia +7
In mixed reality applications, a realistic acoustic experience in spatial environments is as crucial as the visual experience for achieving true immersion. Despite recent advances…
Building Machine Learning Challenges for Anomaly Detection in Science
Elizabeth G. Campolongo, Yuan-Tang Chou, Ekaterina Govorkova +148
Scientific discoveries are often made by finding a pattern or object that was not predicted by the known rules of science. Oftentimes, these anomalous events or objects that do not…
Tell What You Hear From What You See -- Video to Audio Generation Through Text
Xiulong Liu, Kun Su, Eli Shlizerman
The content of visual and audio scenes is multi-faceted such that a video can be paired with various audio and vice-versa. Thereby, in video-to-audio generation task, it is imperat…
CaloChallenge 2022: A Community Challenge for Fast Calorimeter Simulation
Claudius Krause, Michele Faucci Giannelli, Gregor Kasieczka +66
We present the results of the "Fast Calorimeter Simulation Challenge 2022" - the CaloChallenge. We study state-of-the-art generative models on four calorimeter shower datasets of i…
From Vision to Audio and Beyond: A Unified Model for Audio-Visual Representation and Generation
Kun Su, Xiulong Liu, Eli Shlizerman
Video encompasses both visual and auditory data, creating a perceptually rich experience where these two modalities complement each other. As such, videos are a valuable type of me…
Calo-VQ: Vector-Quantized Two-Stage Generative Model in Calorimeter Simulation
Qibin Liu, Chase Shimmin, Xiulong Liu +3
We introduce a novel machine learning method developed for the fast simulation of calorimeter detector response, adapting vector-quantized variational autoencoder (VQ-VAE). Our mod…
Tackling Data Bias in MUSIC-AVQA: Crafting a Balanced Dataset for Unbiased Question-Answering
Xiulong Liu, Zhikang Dong, Peng Zhang
In recent years, there has been a growing emphasis on the intersection of audio, vision, and text modalities, driving forward the advancements in multimodal research. However, stro…
MuseChat: A Conversational Music Recommendation System for Videos
Zhikang Dong, Bin Chen, Xiulong Liu +2
Music recommendation for videos attracts growing interest in multi-modal research. However, existing systems focus primarily on content compatibility, often ignoring the users' pre…
CAVEN: An Embodied Conversational Agent for Efficient Audio-Visual Navigation in Noisy Environments
Xiulong Liu, Sudipta Paul, Moitreya Chatterjee +1
Audio-visual navigation of an agent towards locating an audio goal is a challenging task especially when the audio is sporadic or the environment is noisy. In this paper, we presen…
Multi-Instrumentalist Net: Unsupervised Generation of Music from Body Movements
Kun Su, Xiulong Liu, Eli Shlizerman
We propose a novel system that takes as an input body movements of a musician playing a musical instrument and generates music in an unsupervised setting. Learning to generate mult…
Audeo: Audio Generation for a Silent Performance Video
Kun Su, Xiulong Liu, Eli Shlizerman
We present a novel system that gets as an input video frames of a musician playing the piano and generates the music for that video. Generation of music from visual cues is a chall…
PREDICT & CLUSTER: Unsupervised Skeleton Based Action Recognition
Kun Su, Xiulong Liu, Eli Shlizerman
We propose a novel system for unsupervised skeleton-based action recognition. Given inputs of body keypoints sequences obtained during various movements, our system associates the…