6 papers
R-WoM: Retrieval-augmented World Model For Computer-use Agents
Kai Mei, Jiang Guo, Shuaichen Chang +4
Large Language Models (LLMs) can serve as world models to enhance agent decision-making in digital environments by simulating future states and predicting action outcomes, potentia…
SAM-MI: A Mask-Injected Framework for Enhancing Open-Vocabulary Semantic Segmentation with SAM
Lin Chen, Yingjian Zhu, Qi Yang +3
Open-vocabulary semantic segmentation (OVSS) aims to segment and recognize objects universally. Trained on extensive high-quality segmentation data, the segment anything model (SAM…
SpeechVerse: A Large-scale Generalizable Audio Language Model
Nilaksh Das, Saket Dingliwal, Srikanth Ronanki +14
Large language models (LLMs) have shown incredible proficiency in performing tasks that require semantic understanding of natural language instructions. Recently, many works have f…
Zero-resource Speech Translation and Recognition with LLMs
Karel Mundnich, Xing Niu, Prashant Mathur +10
Despite recent advancements in speech processing, zero-resource speech translation (ST) and automatic speech recognition (ASR) remain challenging problems. In this work, we propose…
Improving Lip-synchrony in Direct Audio-Visual Speech-to-Speech Translation
Lucas Goncalves, Prashant Mathur, Xing Niu +5
Audio-Visual Speech-to-Speech Translation typically prioritizes improving translation quality and naturalness. However, an equally critical aspect in audio-visual content is lip-sy…
Findings of the IWSLT 2024 Evaluation Campaign
Ibrahim Said Ahmad, Antonios Anastasopoulos, OndÅej Bojar +42
This paper reports on the shared tasks organized by the 21st IWSLT Conference. The shared tasks address 7 scientific challenges in spoken language translation: simultaneous and off…