33 citations · 47 across the 11 of their papers we have counts for
11 papers
The Design Space of Tri-Modal Masked Diffusion Models
Louis Bethune, Victor Turrisi, Bruno Kacper Mlodozeniec +21
Discrete diffusion models have emerged as strong alternatives to autoregressive language models, with recent work initializing and fine-tuning a base unimodal model for bimodal gen…
GRACE: A Language Model Framework for Explainable Inverse Reinforcement Learning
Silvia Sapora, Devon Hjelm, Alexander Toshev +2
Inverse Reinforcement Learning aims to recover reward models from expert demonstrations, but traditional methods yield black-box models that are difficult to interpret and debug. I…
From Multimodal LLMs to Generalist Embodied Agents: Methods and Lessons
Andrew Szot, Bogdan Mazoure, Omar Attia +6
We examine the capability of Multimodal Large Language Models (MLLMs) to tackle diverse domains that extend beyond the traditional language and vision tasks these models are typica…
On the Modeling Capabilities of Large Language Models for Sequential Decision Making
Martin Klissarov, Devon Hjelm, Alexander Toshev +1
Large pretrained models are showing increasingly better performance in reasoning and planning tasks across different modalities, opening the possibility to leverage them for comple…
Grounding Multimodal Large Language Models in Actions
Andrew Szot, Bogdan Mazoure, Harsh Agrawal +3
Multimodal Large Language Models (MLLMs) have demonstrated a wide range of capabilities across many domains, including Embodied AI. In this work, we study how to best ground a MLLM…
Poly-View Contrastive Learning
Amitis Shidani, Devon Hjelm, Jason Ramapuram +3
Contrastive learning typically matches pairs of related views among a number of unrelated negative views. Views can be generated (e.g. by augmentations) or be observed. We investig…
Large Language Models as Generalizable Policies for Embodied Tasks
Andrew Szot, Max Schwarzer, Harsh Agrawal +6
We show that large language models (LLMs) can be adapted to be generalizable policies for embodied visual tasks. Our approach, called Large LAnguage model Reinforcement Learning Po…
Pretraining Representations for Data-Efficient Reinforcement Learning
Max Schwarzer, Nitarshan Rajkumar, Michael Noukhovitch +5
Data efficiency is a key challenge for deep reinforcement learning. We address this problem by using unlabeled data to pretrain an encoder which is then finetuned on a small amount…
Test Sample Accuracy Scales with Training Sample Density in Neural Networks
Xu Ji, Razvan Pascanu, Devon Hjelm +2
Intuitively, one would expect accuracy of a trained neural network's prediction on test samples to correlate with how densely the samples are surrounded by seen training samples in…
Cross-Modal Information Maximization for Medical Imaging: CMIM
Tristan Sylvain, Francis Dutil, Tess Berthier +4
In hospitals, data are siloed to specific information systems that make the same information available under different modalities such as the different medical imaging exams the pa…
Tell, Draw, and Repeat: Generating and Modifying Images Based on Continual Linguistic Instruction
Alaaeldin El-Nouby, Shikhar Sharma, Hannes Schulz +5
Conditional text-to-image generation is an active area of research, with many possible applications. Existing research has primarily focused on generating a single image from avail…