6 papers
Closing the Modality Gap Aligns Group-Wise Semantics
Eleonora Grassucci, Giordano Cicchetti, Emanuele Frasca +2
In multimodal learning, CLIP has been recognized as the \textit{de facto} method for learning a shared latent space across multiple modalities, placing similar representations clos…
Controllable Audio-Visual Viewpoint Generation from 360° Spatial Information
Christian Marinoni, Riccardo Fosco Gramaccioni, Eleonora Grassucci +1
The generation of sounding videos has seen significant advancements with the advent of diffusion models. However, existing methods often lack the fine-grained control needed to gen…
FoleyGRAM: Video-to-Audio Generation with GRAM-Aligned Multimodal Encoders
Riccardo Fosco Gramaccioni, Christian Marinoni, Eleonora Grassucci +3
In this work, we present FoleyGRAM, a novel approach to video-to-audio generation that emphasizes semantic conditioning through the use of aligned multimodal encoders. Building on…
A TRIANGLE Enables Multimodal Alignment Beyond Cosine Similarity
Giordano Cicchetti, Eleonora Grassucci, Danilo Comminiello
Multimodal learning plays a pivotal role in advancing artificial intelligence systems by incorporating information from multiple modalities to build a more comprehensive representa…
Semantic Compression via Multimodal Representation Learning
Eleonora Grassucci, Giordano Cicchetti, Aurelio Uncini +1
Multimodal representation learning produces high-dimensional embeddings that align diverse modalities in a shared latent space. While this enables strong generalization, it also in…
Gramian Multimodal Representation Learning and Alignment
Giordano Cicchetti, Eleonora Grassucci, Luigi Sigillo +1
Human perception integrates multiple modalities, such as vision, hearing, and language, into a unified understanding of the surrounding reality. While recent multimodal models have…