15 citations · 50 across the 12 of their papers we have counts for
13 papers
MM-SeR: Multimodal Self-Refinement for Lightweight Image Captioning
Junha Song, Yongsik Jo, So Yeon Min +4
Systems such as video chatbots and navigation robots often depend on streaming image captioning to interpret visual inputs. Existing approaches typically employ large multimodal la…
Self-Regulation and Requesting Interventions
So Yeon Min, Yue Wu, Jimin Sun +4
Human intelligence involves metacognitive abilities like self-regulation, recognizing limitations, and seeking assistance only when needed. While LLM Agents excel in many domains,…
Embodied-RAG: General Non-parametric Embodied Memory for Retrieval and Generation
Quanting Xie, So Yeon Min, Pengliang Ji +7
There is no limit to how much a robot might explore and learn, but all of that knowledge needs to be searchable and actionable. Within language research, retrieval augmented genera…
Tools Fail: Detecting Silent Errors in Faulty Tools
Jimin Sun, So Yeon Min, Yingshan Chang +1
Tools have become a mainstay of LLMs, allowing them to retrieve knowledge not in their weights, to perform tasks on the web, and even to control robots. However, most ontologies an…
AgentKit: Structured LLM Reasoning with Dynamic Graphs
Yue Wu, Yewen Fan, So Yeon Min +6
We propose an intuitive LLM prompting framework (AgentKit) for multifunctional agents. AgentKit offers a unified framework for explicitly constructing a complex "thought process" f…
Habitat 3.0: A Co-Habitat for Humans, Avatars and Robots
Xavier Puig, Eric Undersander, Andrew Szot +20
We present Habitat 3.0: a simulation platform for studying collaborative human-robot tasks in home environments. Habitat 3.0 offers contributions across three dimensions: (1) Accur…
Plan, Eliminate, and Track -- Language Models are Good Teachers for Embodied Agents
Yue Wu, So Yeon Min, Yonatan Bisk +5
Pre-trained large language models (LLMs) capture procedural knowledge about the world. Recent work has leveraged LLM's ability to generate abstract plans to simplify challenging co…
SPRING: Studying the Paper and Reasoning to Play Games
Yue Wu, Shrimai Prabhumoye, So Yeon Min +5
Open-world survival games pose significant challenges for AI algorithms due to their multi-tasking, deep exploration, and goal prioritization requirements. Despite reinforcement le…
Self-Supervised Object Goal Navigation with In-Situ Finetuning
So Yeon Min, Yao-Hung Hubert Tsai, Wei Ding +4
A household robot should be able to navigate to target objects without requiring users to first annotate everything in their home. Most current approaches to object navigation do n…
Don't Copy the Teacher: Data and Model Challenges in Embodied Dialogue
So Yeon Min, Hao Zhu, Ruslan Salakhutdinov +1
Embodied dialogue instruction following requires an agent to complete a complex sequence of tasks from a natural language exchange. The recent introduction of benchmarks (Padmakuma…
FILM: Following Instructions in Language with Modular Methods
So Yeon Min, Devendra Singh Chaplot, Pradeep Ravikumar +2
Recent methods for embodied instruction following are typically trained end-to-end using imitation learning. This often requires the use of expert trajectories and low-level langua…
TransINT: Embedding Implication Rules in Knowledge Graphs with Isomorphic Intersections of Linear Subspaces
So Yeon Min, Preethi Raghavan, Peter Szolovits
Knowledge Graphs (KG), composed of entities and relations, provide a structured representation of knowledge. For easy access to statistical approaches on relational data, multiple…
Entity-Enriched Neural Models for Clinical Question Answering
Bhanu Pratap Singh Rawat, Wei-Hung Weng, So Yeon Min +2
We explore state-of-the-art neural models for question answering on electronic medical records and improve their ability to generalize better on previously unseen (paraphrased) que…