4.8k citations · 5.1k across the 44 of their papers we have counts for
11 papers · 1 filter
Multi-RAG: A Multimodal Retrieval-Augmented Generation System for Adaptive Video Understanding
Mingyang Mao, Mariela M. Perez-Cabarcas, Utteja Kallakuri +3
To effectively engage in human society, the ability to adapt, filter information, and make informed decisions in ever-changing situations is critical. As robots and intelligent age…
Atari-GPT: Benchmarking Multimodal Large Language Models as Low-Level Policies in Atari Games
Nicholas R. Waytowich, Devin White, MD Sunbeam +1
Recent advancements in large language models (LLMs) have expanded their capabilities beyond traditional text-based tasks to multimodal domains, integrating visual, auditory, and te…
COA-GPT: Generative Pre-trained Transformers for Accelerated Course of Action Development in Military Operations
Vinicius G. Goecks, Nicholas Waytowich
The development of Courses of Action (COAs) in military operations is traditionally a time-consuming and intricate process. Addressing this challenge, this study introduces COA-GPT…
Towards Solving Fuzzy Tasks with Human Feedback: A Retrospective of the MineRL BASALT 2022 Competition
Stephanie Milani, Anssi Kanervisto, Karolis Ramanauskas +27
To facilitate research in the direction of fine-tuning foundation models from human feedback, we held the MineRL BASALT Competition on Fine-Tuning from Human Feedback at NeurIPS 20…
Automatic Goal Generation using Dynamical Distance Learning
Bharat Prakash, Nicholas Waytowich, Tinoosh Mohsenin +1
Reinforcement Learning (RL) agents can learn to solve complex sequential decision making tasks by interacting with the environment. However, sample efficiency remains a major chall…
Interactive Hierarchical Guidance using Language
Bharat Prakash, Nicholas Waytowich, Tim Oates +1
Reinforcement learning has been successful in many tasks ranging from robotic control, games, energy management etc. In complex real world environments with sparse rewards and long…
A Narration-based Reward Shaping Approach using Grounded Natural Language Commands
Nicholas Waytowich, Sean L. Barton, Vernon Lawhern +1
While deep reinforcement learning techniques have led to agents that are successfully able to learn to perform a number of tasks that had been previously unlearnable, these techniq…
Improving Safety in Reinforcement Learning Using Model-Based Architectures and Human Intervention
Bharat Prakash, Mohit Khatwani, Nicholas Waytowich +1
Recent progress in AI and Reinforcement learning has shown great success in solving complex problems with high dimensional state spaces. However, most of these successes have been…
Efficiently Combining Human Demonstrations and Interventions for Safe Training of Autonomous Systems in Real-Time
Vinicius G. Goecks, Gregory M. Gremillion, Vernon J. Lawhern +2
This paper investigates how to utilize different forms of human interaction to safely train autonomous systems in real-time by learning from both human demonstrations and intervent…
Cycle-of-Learning for Autonomous Systems from Human Interaction
Nicholas R. Waytowich, Vinicius G. Goecks, Vernon J. Lawhern
We discuss different types of human-robot interaction paradigms in the context of training end-to-end reinforcement learning algorithms. We provide a taxonomy to categorize the typ…
Deep TAMER: Interactive Agent Shaping in High-Dimensional State Spaces
Garrett Warnell, Nicholas Waytowich, Vernon Lawhern +1
While recent advances in deep reinforcement learning have allowed autonomous learning agents to succeed at a variety of complex tasks, existing algorithms generally require a lot o…