4.8k citations · 5.1k across the 44 of their papers we have counts for
46 papers
Mil-SCORE: Benchmarking Long-Context Geospatial Reasoning and Planning in Large Language Models
Aadi Palnitkar, Mingyang Mao, Nicholas Waytowich +2
As large language models (LLMs) are applied to increasingly longer and more complex tasks, there is a growing need for realistic long-context benchmarks that require selective read…
R2BC: Multi-Agent Imitation Learning from Single-Agent Demonstrations
Connor Mattson, Varun Raveendra, Ellen Novoseller +3
Imitation Learning (IL) is a natural way for humans to teach robots, particularly when high-quality demonstrations are easy to obtain. While IL has been widely applied to single-ro…
CREW-WILDFIRE: Benchmarking Agentic Multi-Agent Collaborations at Scale
Jonathan Hyun, Nicholas R Waytowich, Boyuan Chen
Despite rapid progress in large language model (LLM)-based multi-agent systems, current benchmarks fall short in evaluating their scalability, robustness, and coordination capabili…
Multi-Task Reward Learning from Human Ratings
Mingkang Wu, Devin White, Evelyn Rose +3
Reinforcement learning from human feedback (RLHF) has become a key factor in aligning model behavior with users' goals. However, while humans integrate multiple strategies when mak…
Multi-RAG: A Multimodal Retrieval-Augmented Generation System for Adaptive Video Understanding
Mingyang Mao, Mariela M. Perez-Cabarcas, Utteja Kallakuri +3
To effectively engage in human society, the ability to adapt, filter information, and make informed decisions in ever-changing situations is critical. As robots and intelligent age…
EDEN: Entorhinal Driven Egocentric Navigation Toward Robotic Deployment
Mikolaj Walczak, Romina Aalishah, Wyatt Mackey +5
Deep reinforcement learning agents are often fragile while humans remain adaptive and flexible to varying scenarios. To bridge this gap, we present EDEN, a biologically inspired na…
Memento: Augmenting Personalized Memory via Practical Multimodal Wearable Sensing in Visual Search and Wayfinding Navigation
Indrajeet Ghosh, Kasthuri Jayarajah, Nicholas Waytowich +1
Working memory involves the temporary retention of information over short periods. It is a critical cognitive function that enables humans to perform various online processing task…
Performance Optimization of Ratings-Based Reinforcement Learning
Evelyn Rose, Devin White, Mingkang Wu +3
This paper explores multiple optimization methods to improve the performance of rating-based reinforcement learning (RbRL). RbRL, a method based on the idea of human ratings, has b…
Human-Inspired Multi-Level Reinforcement Learning
Mingkang Wu, Devin White, Vernon Lawhern +2
Reinforcement learning (RL), a common tool in decision making, learns control policies from various experiences based on the associated cumulative return/rewards without treating t…
GUIDE: Real-Time Human-Shaped Agents
Lingyu Zhang, Zhengran Ji, Nicholas R Waytowich +1
The recent rapid advancement of machine learning has been driven by increasingly powerful models with the growing availability of training data and computational resources. However…
Atari-GPT: Benchmarking Multimodal Large Language Models as Low-Level Policies in Atari Games
Nicholas R. Waytowich, Devin White, MD Sunbeam +1
Recent advancements in large language models (LLMs) have expanded their capabilities beyond traditional text-based tasks to multimodal domains, integrating visual, auditory, and te…
Scalable Interactive Machine Learning for Future Command and Control
Anna Madison, Ellen Novoseller, Vinicius G. Goecks +7
Future warfare will require Command and Control (C2) personnel to make decisions at shrinking timescales in complex and potentially ill-defined situations. Given the need for robus…
COA-GPT: Generative Pre-trained Transformers for Accelerated Course of Action Development in Military Operations
Vinicius G. Goecks, Nicholas Waytowich
The development of Courses of Action (COAs) in military operations is traditionally a time-consuming and intricate process. Addressing this challenge, this study introduces COA-GPT…
StarCraftImage: A Dataset For Prototyping Spatial Reasoning Methods For Multi-Agent Environments
Sean Kulinski, Nicholas R. Waytowich, James Z. Hare +1
Spatial reasoning tasks in multi-agent environments such as event prediction, agent type identification, or missing data imputation are important for multiple applications (e.g., a…
DIP-RL: Demonstration-Inferred Preference Learning in Minecraft
Ellen Novoseller, Vinicius G. Goecks, David Watkins +2
In machine learning for sequential decision-making, an algorithmic agent learns to interact with an environment while receiving feedback in the form of a reward signal. However, in…
Rating-based Reinforcement Learning
Devin White, Mingkang Wu, Ellen Novoseller +3
This paper develops a novel rating-based reinforcement learning approach that uses human ratings to obtain human guidance in reinforcement learning. Different from the existing pre…
DisasterResponseGPT: Large Language Models for Accelerated Plan of Action Development in Disaster Response Scenarios
Vinicius G. Goecks, Nicholas R. Waytowich
The development of plans of action in disaster response scenarios is a time-consuming process. Large Language Models (LLMs) offer a powerful solution to expedite this process throu…
Learning Flight Control Systems from Human Demonstrations and Real-Time Uncertainty-Informed Interventions
Prashant Ganesh, J. Humberto Ramos, Vinicius G. Goecks +4
This paper describes a methodology for learning flight control systems from human demonstrations and interventions while considering the estimated uncertainty in the learned models…
Towards Solving Fuzzy Tasks with Human Feedback: A Retrospective of the MineRL BASALT 2022 Competition
Stephanie Milani, Anssi Kanervisto, Karolis Ramanauskas +27
To facilitate research in the direction of fine-tuning foundation models from human feedback, we held the MineRL BASALT Competition on Fine-Tuning from Human Feedback at NeurIPS 20…
Towards an Interpretable Hierarchical Agent Framework using Semantic Goals
Bharat Prakash, Nicholas Waytowich, Tim Oates +1
Learning to solve long horizon temporally extended tasks with reinforcement learning has been a challenge for several years now. We believe that it is important to leverage both th…
Multiple View Performers for Shape Completion
David Watkins, Peter Allen, Krzysztof Choromanski +2
We propose the Multiple View Performer (MVP) - a new architecture for 3D shape completion from a series of temporally sequential views. MVP accomplishes this task by using linear-a…
TagTeam: Towards Wearable-Assisted, Implicit Guidance for Human--Drone Teams
Kasthuri Jayarajah, Aryya Gangopadhyay, Nicholas Waytowich
The availability of sensor-rich smart wearables and tiny, yet capable, unmanned vehicles such as nano quadcopters, opens up opportunities for a novel class of highly interactive, a…
Learning to Guide Multiple Heterogeneous Actors from a Single Human Demonstration via Automatic Curriculum Learning in StarCraft II
Nicholas Waytowich, James Hare, Vinicius G. Goecks +4
Traditionally, learning from human demonstrations via direct behavior cloning can lead to high-performance policies given that the algorithm has access to large amounts of high-qua…
Combining Learning from Human Feedback and Knowledge Engineering to Solve Hierarchical Tasks in Minecraft
Vinicius G. Goecks, Nicholas Waytowich, David Watkins-Valls +1
Real-world tasks of interest are generally poorly defined by human-readable descriptions and have no pre-defined reward signals unless it is defined by a human designer. Conversely…