8 papers
Scaling Properties of Continuous Diffusion Spoken Language Models
Jason Ramapuram, Eeshan Gunesh Dhekane, Amitis Shidani +6
Speech-only spoken language models (SLMs) lag behind text and text-speech models in performance, with recent discrete autoregressive (AR) SLMs indicating significant computational…
GRACE: A Language Model Framework for Explainable Inverse Reinforcement Learning
Silvia Sapora, Devon Hjelm, Alexander Toshev +2
Inverse Reinforcement Learning aims to recover reward models from expert demonstrations, but traditional methods yield black-box models that are difficult to interpret and debug. I…
Scaling Synthetic Task Generation for Agents via Exploration
Ram Ramrakhya, Andrew Szot, Omar Attia +6
Post-Training Multimodal Large Language Models (MLLMs) to build interactive agents holds promise across domains such as computer-use, web navigation, and robotics. A key challenge…
From Multimodal LLMs to Generalist Embodied Agents: Methods and Lessons
Andrew Szot, Bogdan Mazoure, Omar Attia +6
We examine the capability of Multimodal Large Language Models (MLLMs) to tackle diverse domains that extend beyond the traditional language and vision tasks these models are typica…
Grounding Multimodal Large Language Models in Actions
Andrew Szot, Bogdan Mazoure, Harsh Agrawal +3
Multimodal Large Language Models (MLLMs) have demonstrated a wide range of capabilities across many domains, including Embodied AI. In this work, we study how to best ground a MLLM…
On the Modeling Capabilities of Large Language Models for Sequential Decision Making
Martin Klissarov, Devon Hjelm, Alexander Toshev +1
Large pretrained models are showing increasingly better performance in reasoning and planning tasks across different modalities, opening the possibility to leverage them for comple…