425 citations · 725 across the 25 of their papers we have counts for
29 papers
Propose and Attend: Training-free MLLM Grounding Confidence via Multi-Token Localized Attention
Daniel Shalam, Emanuel Ben Baruch, Avi Ben Cohen +1
Multimodal large language models can emit localized predictions, bounding boxes for objects and temporal windows for video and audio events, but they hallucinate these regions prol…
Linguistically Informed Evaluation of Multilingual ASR for African Languages
Fei-Yueh Chen, Lateef Adeleke, C. M. Downey
Word Error Rate (WER) mischaracterizes ASR models' performance for African languages by combining phonological, tone, and other linguistic errors into a single lexical error. By co…
CWM: An Open-Weights LLM for Research on Code Generation with World Models
FAIR CodeGen team, Jade Copet, Quentin Carbonneaux +48
We release Code World Model (CWM), a 32-billion-parameter open-weights LLM, to advance research on code generation with world models. To improve code understanding beyond what can…
The Llama 3 Herd of Models
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri +556
Modern artificial intelligence (AI) systems are powered by foundation models. This paper presents a new set of foundation models, called Llama 3. It is a herd of language models th…
Discrete Flow Matching
Itai Gat, Tal Remez, Neta Shaul +5
Despite Flow Matching and diffusion models having emerged as powerful generative paradigms for continuous variables such as images and videos, their application to high-dimensional…
The Larger the Better? Improved LLM Code-Generation via Budget Reallocation
Michael Hassid, Tal Remez, Jonas Gehring +2
It is a common belief that large language models (LLMs) are better than smaller-sized ones. However, larger models also require significantly more time and compute during inference…
Masked Audio Generation using a Single Non-Autoregressive Transformer
Alon Ziv, Itai Gat, Gael Le Lan +6
We introduce MAGNeT, a masked generative sequence modeling method that operates directly over several streams of audio tokens. Unlike prior work, MAGNeT is comprised of a single-st…
EXPRESSO: A Benchmark and Analysis of Discrete Expressive Speech Resynthesis
Tu Anh Nguyen, Wei-Ning Hsu, Antony D'Avirro +10
Recent work has shown that it is possible to resynthesize high-quality speech based, not on text, but on low bitrate discrete units that have been learned in a self-supervised fash…
Code Llama: Open Foundation Models for Code
Baptiste Rozière, Jonas Gehring, Fabian Gloeckle +23
We release Code Llama, a family of large language models for code based on Llama 2 providing state-of-the-art performance among open models, infilling capabilities, support for lar…
Simple and Controllable Music Generation
Jade Copet, Felix Kreuk, Itai Gat +5
We tackle the task of conditional music generation. We introduce MusicGen, a single Language Model (LM) that operates over several streams of compressed discrete music representati…
Textually Pretrained Speech Language Models
Michael Hassid, Tal Remez, Tu Anh Nguyen +9
Speech language models (SpeechLMs) process and generate acoustic data only, without textual supervision. In this work, we propose TWIST, a method for training SpeechLMs using a war…
ReVISE: Self-Supervised Speech Resynthesis with Visual Input for Universal and Generalized Speech Enhancement
Wei-Ning Hsu, Tal Remez, Bowen Shi +2
Prior works on improving speech quality with visual input typically study each type of auditory distortion separately (e.g., separation, inpainting, video-to-speech) and present ta…
AudioScopeV2: Audio-Visual Attention Architectures for Calibrated Open-Domain On-Screen Sound Separation
Efthymios Tzinis, Scott Wisdom, Tal Remez +1
We introduce AudioScopeV2, a state-of-the-art universal audio-visual on-screen sound separation system which is capable of learning to separate sounds and associate them with on-sc…
More than Words: In-the-Wild Visually-Driven Prosody for Text-to-Speech
Michael Hassid, Michelle Tadmor Ramanovich, Brendan Shillingford +3
In this paper we present VDTTS, a Visually-Driven Text-to-Speech model. Motivated by dubbing, VDTTS takes advantage of video frames as an additional input alongside text, and gener…
Translatotron 2: High-quality direct speech-to-speech translation with voice preservation
Ye Jia, Michelle Tadmor Ramanovich, Tal Remez +1
We present Translatotron 2, a neural direct speech-to-speech translation model that can be trained end-to-end. Translatotron 2 consists of a speech encoder, a linguistic decoder, a…
Improving On-Screen Sound Separation for Open-Domain Videos with Audio-Visual Self-Attention
Efthymios Tzinis, Scott Wisdom, Tal Remez +1
We introduce a state-of-the-art audio-visual on-screen sound separation system which is capable of learning to separate sounds and associate them with on-screen objects by looking…
Into the Wild with AudioScope: Unsupervised Audio-Visual Separation of On-Screen Sounds
Efthymios Tzinis, Scott Wisdom, Aren Jansen +4
Recent progress in deep learning has enabled many advances in sound separation and visual scene understanding. However, extracting sound sources which are apparent in natural video…
Class-Aware Fully-Convolutional Gaussian and Poisson Denoising
Tal Remez, Or Litany, Raja Giryes +1
We propose a fully-convolutional neural-network architecture for image denoising which is simple yet powerful. Its structure allows to exploit the gradual nature of the denoising p…
Learning to Segment via Cut-and-Paste
Tal Remez, Jonathan Huang, Matthew Brown
This paper presents a weakly-supervised approach to object instance segmentation. Starting with known or predicted object bounding boxes, we learn object masks by playing a game of…
Efficient Deformable Shape Correspondence via Kernel Matching
Zorah Lähner, Matthias Vestner, Amit Boyarski +8
We present a method to match three dimensional shapes under non-isometric deformations, topology changes and partiality. We formulate the problem as matching between a set of pair-…
Deep Functional Maps: Structured Prediction for Dense Shape Correspondence
Or Litany, Tal Remez, Emanuele Rodolà +2
We introduce a new framework for learning dense correspondence between deformable 3D shapes. Existing learning based approaches model shape correspondence as a labelling problem, w…
Deep Class Aware Denoising
Tal Remez, Or Litany, Raja Giryes +1
The increasing demand for high image quality in mobile devices brings forth the need for better computational enhancement techniques, and image denoising in particular. At the same…
Deep Convolutional Denoising of Low-Light Images
Tal Remez, Or Litany, Raja Giryes +1
Poisson distribution is used for modeling noise in photon-limited imaging. While canonical examples include relatively exotic types of sensing like spectral imaging or astronomy, t…
Cloud Dictionary: Sparse Coding and Modeling for Point Clouds
Or Litany, Tal Remez, Alex Bronstein
With the development of range sensors such as LIDAR and time-of-flight cameras, 3D point cloud scans have become ubiquitous in computer vision applications, the most prominent ones…