43 citations · 150 across the 24 of their papers we have counts for
24 papers
Diff-Foley: Synchronized Video-to-Audio Synthesis with Latent Diffusion Models
Simian Luo, Chuanhao Yan, Chenxu Hu +1
The Video-to-Audio (V2A) model has recently gained attention for its practical application in generating audio directly from silent videos, particularly in video/film production. H…
On Uni-Modal Feature Learning in Supervised Multi-Modal Learning
Chenzhuang Du, Jiaye Teng, Tingle Li +5
We abstract the features (i.e. learned representations) of multi-modal data into 1) uni-modal features, which can be learned from uni-modal training, and 2) paired features, which…
ChatDB: Augmenting LLMs with Databases as Their Symbolic Memory
Chenxu Hu, Jie Fu, Chenzhuang Du +3
Large language models (LLMs) with memory are computationally universal. However, mainstream LLMs are not taking full advantage of memory, and the designs are heavily influenced by…
Controllable and Lossless Non-Autoregressive End-to-End Text-to-Speech
Zhengxi Liu, Qiao Tian, Chenxu Hu +5
Some recent studies have demonstrated the feasibility of single-stage neural text-to-speech, which does not need to generate mel-spectrograms but generates the raw waveforms direct…
Radio2Speech: High Quality Speech Recovery from Radio Frequency Signals
Running Zhao, Jiangtao Yu, Tingle Li +2
Considering the microphone is easily affected by noise and soundproof materials, the radio frequency (RF) signal is a promising candidate to recover audio as it is immune to noise…
The Modality Focusing Hypothesis: Towards Understanding Crossmodal Knowledge Distillation
Zihui Xue, Zhengqi Gao, Sucheng Ren +1
Crossmodal knowledge distillation (KD) extends traditional knowledge distillation to the area of multimodal learning and demonstrates great success in various applications. To achi…
Learning Visual Styles from Audio-Visual Associations
Tingle Li, Yichen Liu, Andrew Owens +1
From the patter of rain to the crunch of snow, the sounds we hear often convey the visual textures that appear within a scene. In this paper, we present a method for learning visua…
MUTR3D: A Multi-camera Tracking Framework via 3D-to-2D Queries
Tianyuan Zhang, Xuanyao Chen, Yue Wang +2
Accurate and consistent 3D tracking from multiple cameras is a key component in a vision-based autonomous driving system. It involves modeling 3D dynamic objects in complex scenes…
Sound2Synth: Interpreting Sound via FM Synthesizer Parameters Estimation
Zui Chen, Yansen Jing, Shengcheng Yuan +3
Synthesizer is a type of electronic musical instrument that is now widely used in modern music production and sound design. Each parameters configuration of a synthesizer produces…
Training-Free Robust Multimodal Learning via Sample-Wise Jacobian Regularization
Zhengqi Gao, Sucheng Ren, Zihui Xue +2
Multimodal fusion emerges as an appealing technique to improve model performances on many tasks. Nevertheless, the robustness of such fusion methods is rarely involved in the prese…
Self-supervision through Random Segments with Autoregressive Coding (RandSAC)
Tianyu Hua, Yonglong Tian, Sucheng Ren +3
Inspired by the success of self-supervised autoregressive representation learning in natural language (GPT and its variants), and advances in recent visual architecture design with…
FUTR3D: A Unified Sensor Fusion Framework for 3D Detection
Xuanyao Chen, Tianyuan Zhang, Yue Wang +2
Sensor fusion is an essential topic in many perception systems, such as autonomous driving and robotics. Existing multi-modal 3D detection models usually involve customized designs…
Neural Dubber: Dubbing for Videos According to Scripts
Chenxu Hu, Qiao Tian, Tingle Li +3
Dubbing is a post-production process of re-recording actors' dialogues, which is extensively used in filmmaking and video production. It is usually performed manually by profession…
Co-advise: Cross Inductive Bias Distillation
Sucheng Ren, Zhengqi Gao, Tianyu Hua +4
Transformers recently are adapted from the community of natural language processing as a promising substitute of convolution-based neural networks for visual learning tasks. Howeve…
Improving Multi-Modal Learning with Uni-Modal Teachers
Chenzhuang Du, Tingle Li, Yichen Liu +4
Learning multi-modal representations is an essential step towards real-world robotic applications, and various multi-modal fusion models have been developed for this purpose. Howev…
Intrinsically Motivated Self-supervised Learning in Reinforcement Learning
Yue Zhao, Chenzhuang Du, Hang Zhao +1
In vision-based reinforcement learning (RL) tasks, it is prevalent to assign auxiliary tasks with a surrogate self-supervised loss so as to obtain more semantic representations and…
What Makes Multi-modal Learning Better than Single (Provably)
Yu Huang, Chenzhuang Du, Zihui Xue +3
The world provides us with data of multiple modalities. Intuitively, models fusing data from different modalities outperform their uni-modal counterparts, since more information is…
On Feature Decorrelation in Self-Supervised Learning
Tianyu Hua, Wenxiao Wang, Zihui Xue +3
In self-supervised representation learning, a common idea behind most of the state-of-the-art approaches is to enforce the robustness of the representations to predefined augmentat…
Multimodal Knowledge Expansion
Zihui Xue, Sucheng Ren, Zhengqi Gao +1
The popularity of multimodal sensors and the accessibility of the Internet have brought us a massive amount of unlabeled multimodal data. Since existing datasets and well-trained m…
Predictive Visual Tracking: A New Benchmark and Baseline Approach
Bowen Li, Yiming Li, Junjie Ye +2
As a crucial robotic perception capability, visual tracking has been intensively studied recently. In the real-world scenarios, the onboard processing time of the image streams ine…
CVC: Contrastive Learning for Non-parallel Voice Conversion
Tingle Li, Yichen Liu, Chenxu Hu +1
Cycle consistent generative adversarial network (CycleGAN) and variational autoencoder (VAE) based models have gained popularity in non-parallel voice conversion recently. However,…
CLOUD: Contrastive Learning of Unsupervised Dynamics
Jianren Wang, Yujie Lu, Hang Zhao
Developing agents that can perform complex control tasks from high dimensional observations such as pixels is challenging due to difficulties in learning dynamics efficiently. In t…
LID 2020: The Learning from Imperfect Data Challenge Results
Yunchao Wei, Shuai Zheng, Ming-Ming Cheng +32
Learning from imperfect data becomes an issue in many industrial applications after the research community has made profound progress in supervised learning from perfectly annotate…
SEMI: Self-supervised Exploration via Multisensory Incongruity
Jianren Wang, Ziwen Zhuang, Hang Zhao
Efficient exploration is a long-standing problem in reinforcement learning since extrinsic rewards are usually sparse or missing. A popular solution to this issue is to feed an age…