Video2Music: Suitable Music Generation from Videos using an Affective Multimodal Transformer model
arXiv:2311.00968 · doi:10.1016/j.eswa.2024.123640
Abstract
Numerous studies in the field of music generation have demonstrated impressive performance, yet virtually no models are able to directly generate music to match accompanying videos. In this work, we develop a generative music AI framework, Video2Music, that can match a provided video. We first curated a unique collection of music videos. Then, we analysed the music videos to obtain semantic, scene offset, motion, and emotion features. These distinct features are then employed as guiding input to our music generation model. We transcribe the audio files into MIDI and chords, and extract features such as note density and loudness. This results in a rich multimodal dataset, called MuVi-Sync, on which we train a novel Affective Multimodal Transformer (AMT) model to generate music given a video. This model includes a novel mechanism to enforce affective similarity between video and music. Finally, post-processing is performed based on a biGRU-based regression model to estimate note density and loudness based on the video features. This ensures a dynamic rendering of the generated chords with varying rhythm and volume. In a thorough experiment, we show that our proposed framework can generate music that matches the video content in terms of emotion. The musical quality, along with the quality of music-video matching is confirmed in a user study. The proposed AMT model, along with the new MuVi-Sync dataset, presents a promising step for the new task of music generation for videos.
References in corpus (11)
- Decision Transformer: Reinforcement Learning via Sequence Modeling
- A Functional Taxonomy of Music Generation Systems
- Video Background Music Generation with Controllable Music Transformer
- MorpheuS: generating structured music with constrained patterns and tension
- On the Potential of Simple Framewise Approaches to Piano Transcription
- The Jazz Transformer on the Front Line: Exploring the Shortcomings of AI-composed Music through Quantitative Measures
- Multi-Instrumentalist Net: Unsupervised Generation of Music from Body Movements
- Controllable deep melody generation via hierarchical music structure representation
- Music FaderNets: Controllable Music Generation Based On High-Level Features via Low-Level Feature Modelling
- Predicting emotion from music videos: exploring the relative contribution of visual and auditory information to affective responses
- V2Meow: Meowing to the Visual Beat via Video-to-Music Generation