Publications (32)
A Faster Method for Tracking and Scoring Videos Corresponding to Sentences
Haonan Yu, Daniel P. Barrett, Jeffrey Mark Siskind
Prior work presented the sentence tracker, a method for scoring how well a sentence describes a video clip or alternatively how well a video clip depicts a sentence. We present an…
Playing the lottery with rewards and multiple languages: lottery tickets in RL and NLP
Haonan Yu, Sergey Edunov, Yuandong Tian +1
The lottery ticket hypothesis proposes that over-parameterization of deep neural networks (DNNs) aids training by increasing the probability of a "lucky" sub-network initialization…
Interactive Language Acquisition with One-shot Visual Concept Learning through a Conversational Game
Haichao Zhang, Haonan Yu, Wei Xu
Building intelligent agents that can communicate with and learn from humans in natural language is of great value. Supervised language learning is limited by the ability of capturi…
Robot Language Learning, Generation, and Comprehension
Daniel Paul Barrett, Scott Alan Bronikowski, Haonan Yu +1
We present a unified framework which supports grounding natural-language semantics in robotic driving. This framework supports acquisition (learning grounded meanings of nouns and…
Focus-LIME: Surgical Interpretation of Long-Context Large Language Models via Proxy-Based Neighborhood Selection
Junhao Liu, Haonan Yu, Zhenyu Yan +1
As Large Language Models (LLMs) scale to handle massive context windows, achieving surgical feature-level interpretation is essential for high-stakes tasks like legal auditing and…
Hierarchical Reinforcement Learning By Discovering Intrinsic Options
Jesse Zhang, Haonan Yu, Wei Xu
We propose a hierarchical reinforcement learning method, HIDIO, that can learn task-agnostic options in a self-supervised manner while jointly learning to utilize them to solve spa…
Guided Feature Transformation (GFT): A Neural Language Grounding Module for Embodied Agents
Haonan Yu, Xiaochen Lian, Haichao Zhang +1
Recently there has been a rising interest in training agents, embodied in virtual environments, to perform language-directed tasks by deep reinforcement learning. In this paper, we…
Resource-Efficient Neural Architect
Yanqi Zhou, Siavash Ebrahimi, Sercan Ã. Arık +3
Neural Architecture Search (NAS) is a laborious process. Prior work on automated NAS targets mainly on improving accuracy, but lacks consideration of computational resource use. We…
Learning Multi-Stage Pick-and-Place with a Legged Mobile Manipulator
Haichao Zhang, Haonan Yu, Le Zhao +4
Quadruped-based mobile manipulation presents significant challenges in robotics due to the diversity of required skills, the extended task horizon, and partial observability. After…
WASD: Locating Critical Neurons as Sufficient Conditions for Explaining and Controlling LLM Behavior
Haonan Yu, Junhao Liu, Zhenyu Yan +2
Precise behavioral control of large language models (LLMs) is critical for complex applications. However, existing methods often incur high training costs, lack natural language co…
Do You Need the Entropy Reward (in Practice)?
Haonan Yu, Haichao Zhang, Wei Xu
Maximum entropy (MaxEnt) RL maximizes a combination of the original task reward and an entropy reward. It is believed that the regularization imposed by entropy, on both policy imp…
Beyond Attribution: Unified Concept-Level Explanations
Junhao Liu, Haonan Yu, Xin Zhang
There is an increasing need to integrate model-agnostic explanation techniques with concept-based approaches, as the former can explain models across different architectures while…
Why Build an Assistant in Minecraft?
Arthur Szlam, Jonathan Gray, Kavya Srinet +11
In this document we describe a rationale for a research program aimed at building an open "assistant" in the game Minecraft, in order to make progress on the problems of natural la…
MetaView: Few-shot Active Object Recognition
Wei Wei, Haonan Yu, Haichao Zhang +2
In robot sensing scenarios, instead of passively utilizing human captured views, an agent should be able to actively choose informative viewpoints of a 3D object as discriminative…
MAnchors: Memorization-Based Acceleration of Anchors via Rule Reuse and Transformation
Haonan Yu, Junhao Liu, Xin Zhang
Anchors is a popular local model-agnostic explanation technique whose applicability is limited by its computational inefficiency. To address this limitation, we propose a memorizat…
Policy Expansion for Bridging Offline-to-Online Reinforcement Learning
Haichao Zhang, We Xu, Haonan Yu
Pre-training with offline data and online fine-tuning using reinforcement learning is a promising strategy for learning control policies by leveraging the best of both worlds in te…
Discriminative Training: Learning to Describe Video with Sentences, from Video Described with Sentences
Haonan Yu, Jeffrey Mark Siskind
We present a method for learning word meanings from complex and realistic video clips by discriminatively training (DT) positive sentential labels against negative ones, and then u…
A Deep Compositional Framework for Human-like Language Acquisition in Virtual Environment
Haonan Yu, Haichao Zhang, Wei Xu
We tackle a task where an agent learns to navigate in a 2D maze-like environment called XWORLD. In each session, the agent perceives a sequence of raw-pixel frames, a natural langu…
Collecting and Annotating the Large Continuous Action Dataset
Daniel Paul Barrett, Ran Xu, Haonan Yu +1
We make available to the community a new dataset to support action-recognition research. This dataset is different from prior datasets in several key ways. It is significantly larg…
TAAC: Temporally Abstract Actor-Critic for Continuous Control
Haonan Yu, Wei Xu, Haichao Zhang
We present temporally abstract actor-critic (TAAC), a simple but effective off-policy RL algorithm that incorporates closed-loop temporal abstraction into the actor-critic framewor…
Revitalizing Black-Box Interpretability: Actionable Interpretability for LLMs via Proxy Models
Junhao Liu, Haonan Yu, Zhenyu Yan +1
Post-hoc explanations provide transparency and are essential for guiding model optimization, such as prompt engineering and data sanitation. However, applying model-agnostic techni…
Listen, Interact and Talk: Learning to Speak via Interaction
Haichao Zhang, Haonan Yu, Wei Xu
One of the long-term goals of artificial intelligence is to build an agent that can communicate intelligently with human in natural language. Most existing work on natural language…
SLIM: Sim-to-Real Legged Instructive Manipulation via Long-Horizon Visuomotor Learning
Haichao Zhang, Haonan Yu, Le Zhao +4
We present a low-cost legged mobile manipulation system that solves long-horizon real-world tasks, trained by reinforcement learning purely in simulation. This system is made possi…
One ticket to win them all: generalizing lottery ticket initializations across datasets and optimizers
Ari S. Morcos, Haonan Yu, Michela Paganini +1
The success of lottery ticket initializations (Frankle and Carbin, 2019) suggests that small, sparsified networks can be trained so long as the network is initialized appropriately…
Video Paragraph Captioning Using Hierarchical Recurrent Neural Networks
Haonan Yu, Jiang Wang, Zhiheng Huang +2
We present an approach that exploits hierarchical Recurrent Neural Networks (RNNs) to tackle the video captioning problem, i.e., generating one or multiple sentences to describe a…
Towards Safe Reinforcement Learning with a Safety Editor Policy
Haonan Yu, Wei Xu, Haichao Zhang
We consider the safe reinforcement learning (RL) problem of maximizing utility with extremely low constraint violation rates. Assuming no prior knowledge or pre-training of the env…
CraftAssist: A Framework for Dialogue-enabled Interactive Agents
Jonathan Gray, Kavya Srinet, Yacine Jernite +6
This paper describes an implementation of a bot assistant in Minecraft, and the tools and platform allowing players to interact with the bot and to record those interactions. The p…
VONet: Unsupervised Video Object Learning With Parallel U-Net Attention and Object-wise Sequential VAE
Haonan Yu, Wei Xu
Unsupervised video object learning seeks to decompose video scenes into structural object representations without any supervision from depth, optical flow, or segmentation. We pres…
EPNAS: Efficient Progressive Neural Architecture Search
Yanqi Zhou, Peng Wang, Sercan Arik +4
In this paper, we propose Efficient Progressive Neural Architecture Search (EPNAS), a neural architecture search (NAS) that efficiently handles large search space through a novel p…
Sentence Directed Video Object Codetection
Haonan Yu, Jeffrey Mark Siskind
We tackle the problem of video object codetection by leveraging the weak semantic constraint implied by sentences that describe the video content. Unlike most existing work that fo…
Generative Planning for Temporally Coordinated Exploration in Reinforcement Learning
Haichao Zhang, Wei Xu, Haonan Yu
Standard model-free reinforcement learning algorithms optimize a policy that generates the action to be taken in the current time step in order to maximize expected future return.…
Interactive Grounded Language Acquisition and Generalization in a 2D World
Haonan Yu, Haichao Zhang, Wei Xu
We build a virtual agent for learning language in a 2D maze-like world. The agent sees images of the surrounding environment, listens to a virtual teacher, and takes actions to rec…