140 citations · 283 across the 25 of their papers we have counts for
26 papers
Go Beyond Imagination: Maximizing Episodic Reachability with World Models
Yao Fu, Run Peng, Honglak Lee
Efficient exploration is a challenging topic in reinforcement learning, especially for sparse reward tasks. To deal with the reward sparsity, people commonly apply intrinsic reward…
Exploring Demonstration Ensembling for In-context Learning
Muhammad Khalifa, Lajanugen Logeswaran, Moontae Lee +2
In-context learning (ICL) operates by showing language models (LMs) examples of input-output pairs for a given task, i.e., demonstrations. The standard approach for ICL is to promp…
Story Visualization by Online Text Augmentation with Context Memory
Daechul Ahn, Daneul Kim, Gwangmo Song +4
Story visualization (SV) is a challenging text-to-image generation task for the difficulty of not only rendering visual details from the text descriptions but also encoding a long-…
Scalable 3D Captioning with Pretrained Models
Tiange Luo, Chris Rockwell, Honglak Lee +1
We introduce Cap3D, an automatic approach for generating descriptive text for 3D objects. This approach utilizes pretrained models from image captioning, image-text alignment, and…
A Picture is Worth a Thousand Words: Language Models Plan from Pixels
Anthony Z. Liu, Lajanugen Logeswaran, Sungryull Sohn +1
Planning is an important capability of artificial agents that perform long-horizon tasks in real-world environments. In this work, we explore the use of pre-trained language models…
Preference Transformer: Modeling Human Preferences using Transformers for RL
Changyeon Kim, Jongjin Park, Jinwoo Shin +3
Preference-based reinforcement learning (RL) provides a framework to train agents using human preferences between two behaviors. However, preference-based RL has been challenging t…
Composing Task Knowledge with Modular Successor Feature Approximators
Wilka Carvalho, Angelos Filos, Richard L. Lewis +2
Recently, the Successor Features and Generalized Policy Improvement (SF&GPI) framework has been proposed as a method for learning, composing, and transferring predictive knowledge…
Learning Action Translator for Meta Reinforcement Learning on Sparse-Reward Tasks
Yijie Guo, Qiucheng Wu, Honglak Lee
Meta reinforcement learning (meta-RL) aims to learn a policy solving a set of training tasks simultaneously and quickly adapting to new tasks. It requires massive amounts of data d…
Pure Transformers are Powerful Graph Learners
Jinwoo Kim, Tien Dat Nguyen, Seonwoo Min +4
We show that standard Transformers without graph-specific modifications can lead to promising results in graph learning both in theory and practice. Given a graph, we simply treat…
Equivariant Descriptor Fields: SE(3)-Equivariant Energy-Based Models for End-to-End Visual Robotic Manipulation Learning
Hyunwoo Ryu, Hong-in Lee, Jeong-Hoon Lee +1
End-to-end learning for visual robotic manipulation is known to suffer from sample inefficiency, requiring large numbers of demonstrations. The spatial roto-translation equivarianc…
Towards Diverse Evaluation of Class Incremental Learning: A Representation Learning Perspective
Sungmin Cha, Jihwan Kwak, Dongsub Shim +4
Class incremental learning (CIL) algorithms aim to continually learn new object classes from incrementally arriving data while not forgetting past learned classes. The common evalu…
Few-shot Subgoal Planning with Language Models
Lajanugen Logeswaran, Yao Fu, Moontae Lee +1
Pre-trained large language models have shown successful progress in many language understanding benchmarks. This work explores the capability of these models to predict actionable…
Fast Inference and Transfer of Compositional Task Structures for Few-shot Task Generalization
Sungryull Sohn, Hyunjae Woo, Jongwook Choi +4
We tackle real-world problems with complex structures beyond the pixel-based game or simulator. We formulate it as a few-shot reinforcement learning problem where a task is charact…
Simple and Effective Synthesis of Indoor 3D Scenes
Jing Yu Koh, Harsh Agrawal, Dhruv Batra +6
We study the problem of synthesizing immersive 3D indoor scenes from one or more images. Our aim is to generate high-resolution images and videos from novel viewpoints, including v…
Learning Parameterized Task Structure for Generalization to Unseen Entities
Anthony Z. Liu, Sungryull Sohn, Mahdi Qazwini +1
Real world tasks are hierarchical and compositional. Tasks can be composed of multiple subtasks (or sub-goals) that are dependent on each other. These subtasks are defined in terms…
SURF: Semi-supervised Reward Learning with Data Augmentation for Feedback-efficient Preference-based Reinforcement Learning
Jongjin Park, Younggyo Seo, Jinwoo Shin +3
Preference-based reinforcement learning (RL) has shown potential for teaching agents to perform the target tasks without a costly, pre-defined reward function by learning the rewar…
Lipschitz-constrained Unsupervised Skill Discovery
Seohong Park, Jongwook Choi, Jaekyeom Kim +2
We study the problem of unsupervised skill discovery, whose goal is to learn a set of diverse and useful skills with no external reward. There have been a number of skill discovery…
Environment Generation for Zero-Shot Compositional Reinforcement Learning
Izzeddin Gur, Natasha Jaques, Yingjie Miao +4
Many real-world problems are compositional - solving them requires completing interdependent sub-tasks, either in series or in parallel, that can be represented as a dependency gra…
Successor Feature Landmarks for Long-Horizon Goal-Conditioned Reinforcement Learning
Christopher Hoang, Sungryull Sohn, Jongwook Choi +2
Operating in the real-world often requires agents to learn about a complex environment and apply this understanding to achieve a breadth of goals. This problem, known as goal-condi…
L-Verse: Bidirectional Generation Between Image and Text
Taehoon Kim, Gwangmo Song, Sihaeng Lee +6
Far beyond learning long-range interactions of natural language, transformers are becoming the de-facto standard for many vision tasks with their power and scalability. Especially…
Shortest-Path Constrained Reinforcement Learning for Sparse Reward Tasks
Sungryull Sohn, Sungtae Lee, Jongwook Choi +3
We propose the k-Shortest-Path (k-SP) constraint: a novel constraint on the agent's trajectory that improves the sample efficiency in sparse-reward MDPs. We show that any optimal p…
Variational Empowerment as Representation Learning for Goal-Based Reinforcement Learning
Jongwook Choi, Archit Sharma, Honglak Lee +2
Learning to reach goal states and learning diverse skills through mutual information (MI) maximization have been proposed as principled frameworks for self-supervised reinforcement…
Pathdreamer: A World Model for Indoor Navigation
Jing Yu Koh, Honglak Lee, Yinfei Yang +2
People navigating in unfamiliar buildings take advantage of myriad visual, spatial and semantic cues to efficiently achieve their navigation goals. Towards equipping computational…
Revisiting Hierarchical Approach for Persistent Long-Term Video Prediction
Wonkwang Lee, Whie Jung, Han Zhang +6
Learning to predict the long-term future of video frames is notoriously challenging due to inherent ambiguities in the distant future and dramatic amplifications of prediction erro…