22 citations · 55 across the 28 of their papers we have counts for
28 papers
Evaluative Dynamics of AI Integration and Expert Performance under Epistemic Dependence across Heterogeneous Stakes
Dennis Kim, Roya Daneshi, Nikhil Krishnaswamy +2
AI is increasingly integrated into expert workflows, yet how integration affects perceptions of the expert, AI, and their combination remains unclear in domains where lay users are…
Frictive Policy Optimization for LLMs: Epistemic Intervention, Risk-Sensitive Control, and Reflective Alignment
James Pustejovsky, Nikhil Krishnaswamy
We propose Frictive Policy Optimization (FPO), a framework for learning language model policies that regulate not only what to say, but when and how to intervene in order to manage…
Distributed Partial Information Puzzles: Examining Common Ground Construction Under Epistemic Asymmetry
Yifan Zhu, Mariah Bradford, Kenneth Lai +4
Establishing common ground, a shared set of beliefs and mutually recognized facts, is fundamental to collaboration, yet remains a challenge for current AI systems, especially in mu…
The Impact of Background Speech on Interruption Detection in Collaborative Groups
Mariah Bradford, Nikhil Krishnaswamy, Nathaniel Blanchard
Interruption plays a crucial role in collaborative learning, shaping group interactions and influencing knowledge construction. AI-driven support can assist teachers in monitoring…
TRACE: Real-Time Multimodal Common Ground Tracking in Situated Collaborative Dialogues
Hannah VanderHoeven, Brady Bhalla, Ibrahim Khebour +11
We present TRACE, a novel system for live *common ground* tracking in situated collaborative tasks. With a focus on fast, real-time performance, TRACE tracks the speech, actions, g…
Speech Is Not Enough: Interpreting Nonverbal Indicators of Common Knowledge and Engagement
Derek Palmer, Yifan Zhu, Kenneth Lai +8
Our goal is to develop an AI Partner that can provide support for group problem solving and social dynamics. In multi-party working group environments, multimodal analytics is cruc…
Metacognitive AI: Framework and the Case for a Neurosymbolic Approach
Hua Wei, Paulo Shakarian, Christian Lebiere +3
Metacognition is the concept of reasoning about an agent's own internal processes and was originally introduced in the field of developmental psychology. In this position paper, we…
Computational Thought Experiments for a More Rigorous Philosophy and Science of the Mind
Iris Oved, Nikhil Krishnaswamy, James Pustejovsky +1
We offer philosophical motivations for a method we call Virtual World Cognitive Science (VW CogSci), in which researchers use virtual embodied agents that are embedded in virtual w…
Common Ground Tracking in Multimodal Dialogue
Ibrahim Khebour, Kenneth Lai, Mariah Bradford +8
Within Dialogue Modeling research in AI and NLP, considerable attention has been spent on ``dialogue state tracking'' (DST), which is the ability to update the representations of t…
How Good is the Model in Model-in-the-loop Event Coreference Resolution Annotation?
Shafiuddin Rehan Ahmed, Abhijnan Nath, Michael Regan +3
Annotating cross-document event coreference links is a time-consuming and cognitively demanding task that can compromise annotation quality and efficiency. To address this, we prop…
How Good is Automatic Segmentation as a Multimodal Discourse Annotation Aid?
Corbyn Terpstra, Ibrahim Khebour, Mariah Bradford +3
Collaborative problem solving (CPS) in teams is tightly coupled with the creation of shared meaning between participants in a situated, collaborative task. In this work, we assess…
Grounding and Distinguishing Conceptual Vocabulary Through Similarity Learning in Embodied Simulations
Sadaf Ghaffari, Nikhil Krishnaswamy
We present a novel method for using agent experiences gathered through an embodied simulation to ground contextualized word vectors to object representations. We use similarity lea…
AxomiyaBERTa: A Phonologically-aware Transformer Model for Assamese
Abhijnan Nath, Sheikh Mannan, Nikhil Krishnaswamy
Despite their successes in NLP, Transformer-based language models still require extensive computing resources and suffer in low-resource or low-compute settings. In this paper, we…
An Abstract Specification of VoxML as an Annotation Language
Kiyong Lee, Nikhil Krishnaswamy, James Pustejovsky
VoxML is a modeling language used to map natural language expressions into real-time visualizations using commonsense semantic knowledge of objects and events. Its utility has been…
is better than : Decomposing Event Coreference Resolution into Two Tractable Problems
Shafiuddin Rehan Ahmed, Abhijnan Nath, James H. Martin +1
Event Coreference Resolution (ECR) is the task of linking mentions of the same event either within or across documents. Most mention pairs are not coreferent, yet many that are cor…
Detecting and Accommodating Novel Types and Concepts in an Embodied Simulation Environment
Sadaf Ghaffari, Nikhil Krishnaswamy
In this paper, we present methods for two types of metacognitive tasks in an AI system: rapidly expanding a neural classification model to accommodate a new category of object, and…
Exploiting Embodied Simulation to Detect Novel Object Classes Through Interaction
Nikhil Krishnaswamy, Sadaf Ghaffari
In this paper we present a novel method for a naive agent to detect novel objects it encounters in an interaction. We train a reinforcement learning policy on a stacking task given…
Neurosymbolic AI for Situated Language Understanding
Nikhil Krishnaswamy, James Pustejovsky
In recent years, data-intensive AI, particularly the domain of natural language processing and understanding, has seen significant progress driven by the advent of large datasets a…
Situated Multimodal Control of a Mobile Robot: Navigation through a Virtual Environment
Katherine Krajovic, Nikhil Krishnaswamy, Nathaniel J. Dimick +2
We present a new interface for controlling a navigation robot in novel environments using coordinated gesture and language. We use a TurtleBot3 robot with a LIDAR and a camera, an…
A Formal Analysis of Multimodal Referring Strategies Under Common Ground
Nikhil Krishnaswamy, James Pustejovsky
In this paper, we present an analysis of computationally generated mixed-modality definite referring expressions using combinations of gesture and linguistic descriptions. In doing…
Multimodal Continuation-style Architectures for Human-Robot Interaction
Nikhil Krishnaswamy, James Pustejovsky
We present an architecture for integrating real-time, multimodal input into a computational agent's contextual model. Using a human-avatar interaction in a virtual world, we treat…
Situational Grounding within Multimodal Simulations
James Pustejovsky, Nikhil Krishnaswamy
In this paper, we argue that simulation platforms enable a novel type of embodied spatial reasoning, one facilitated by a formal model of object and event semantics that renders th…
Combining Deep Learning and Qualitative Spatial Reasoning to Learn Complex Structures from Sparse Examples with Noise
Nikhil Krishnaswamy, Scott Friedman, James Pustejovsky
Many modern machine learning approaches require vast amounts of training data to learn new concepts; conversely, human learning often requires few examples--sometimes only one--fro…
Multimodal Interactive Learning of Primitive Actions
Tuan Do, Nikhil Krishnaswamy, Kyeongmin Rim +1
We describe an ongoing project in learning to perform primitive actions from demonstrations using an interactive interface. In our previous work, we have used demonstrations captur…