PaLM-E: An Embodied Multimodal Language Model

Abstract

Large language models excel at a wide range of complex tasks. However, enabling general inference in the real world, e.g., for robotics problems, raises the challenge of grounding. We propose embodied language models to directly incorporate real-world continuous sensor modalities into language models and thereby establish the link between words and percepts. Input to our embodied language model are multi-modal sentences that interleave visual, continuous state estimation, and textual input encodings. We train these encodings end-to-end, in conjunction with a pre-trained large language model, for multiple embodied tasks including sequential robotic manipulation planning, visual question answering, and captioning. Our evaluations show that PaLM-E, a single large embodied multimodal model, can address a variety of embodied reasoning tasks, from a variety of observation modalities, on multiple embodiments, and further, exhibits positive transfer: the model benefits from diverse joint training across internet-scale language, vision, and visual-language domains. Our largest model, PaLM-E-562B with 562B parameters, in addition to being trained on robotics tasks, is a visual-language generalist with state-of-the-art performance on OK-VQA, and retains generalist language capabilities with increasing scale.

Cited by in corpus (31)

• Large AI Models in Health Informatics: Applications, Challenges, and the Future
• AI Agents vs. Agentic AI: A Conceptual Taxonomy, Applications and Challenges
• Understanding Large-Language Model (LLM)-powered Human-Robot Interaction
• Taking the Next Step with Generative Artificial Intelligence: The Transformative Role of Multimodal Large Language Models in Science Education
• Materials science in the era of large language models: a perspective
• LLM for SoC Security: A Paradigm Shift
• What Makes Good In-context Demonstrations for Code Intelligence Tasks with LLMs?
• UAVs Meet LLMs: Overviews and Perspectives Toward Agentic Low-Altitude Mobility
• Enhancing Video-Language Representations with Structural Spatio-Temporal Alignment
• Unreflected Acceptance -- Investigating the Negative Consequences of ChatGPT-Assisted Problem Solving in Physics Education
• Patchwork Learning: A Paradigm Towards Integrative Analysis across Diverse Biomedical Data Sources
• On the Prospects of Incorporating Large Language Models (LLMs) in Automated Planning and Scheduling (APS)
• From Screens to Scenes: A Survey of Embodied AI in Healthcare
• LLM-BT: Performing Robotic Adaptive Tasks based on Large Language Models and Behavior Trees
• Multimodal Fusion and Vision-Language Models: A Survey for Robot Vision
• IQAGPT: Image Quality Assessment with Vision-language and ChatGPT Models
• Opportunities for Large Language Models and Discourse in Engineering Design
• A Framework for Neurosymbolic Robot Action Planning using Large Language Models
• Ghost in the Minecraft: Generally Capable Agents for Open-World Environments via Large Language Models with Text-based Knowledge and Memory
• A Comprehensive Review of State-of-The-Art Methods for Java Code Generation from Natural Language Text
• Theory of Mind abilities of Large Language Models in Human-Robot Interaction : An Illusion?
• A Multimodal Approach to Device-Directed Speech Detection with Large Language Models
• Developmental Scaffolding with Large Language Models
• Bootstrapping Cognitive Agents with a Large Language Model
• PolarNet: 3D Point Clouds for Language-Guided Robotic Manipulation
• Target-driven Attack for Large Language Models
• Camera Control at the Edge with Language Models for Scene Understanding
• Structure-Enhanced Protein Instruction Tuning: Towards General-Purpose Protein Understanding with LLMs
• A Concept for User-Centered Delegation of Abstract High-Level Tasks to Cobots for Flexible Lot Sizes
• Remember what you did so you know what to do next
• Auditing LLM-Governed Social Robots with Culture-Specific Moral Gradients