18 citations · 126 across the 28 of their papers we have counts for
29 papers
REMM: Revisable Embodied Memory for Manipulation
Revanth Krishna Senthilkumaran, Ruwan Wickramarachchi, Alessandro Oltramari +1
Long-horizon manipulation requires robots to retain task-relevant history while revising conclusions no longer supported by physical evidence. Retaining observations alone does not…
SafeShift: Safety-Informed Distribution Shifts for Robust Trajectory Prediction in Autonomous Driving
Benjamin Stoler, Ingrid Navarro, Meghdeep Jana +3
As autonomous driving technology matures, safety and robustness of its key components, including trajectory prediction, is vital. Though real-world datasets, such as Waymo Open Mot…
MOSAIC: Learning Unified Multi-Sensory Object Property Representations for Robot Learning via Interactive Perception
Gyan Tatiya, Jonathan Francis, Ho-Hsiang Wu +2
A holistic understanding of object properties across diverse sensory modalities (e.g., visual, audio, and haptic) is essential for tasks ranging from object categorization to compl…
Traffic-Domain Video Question Answering with Automatic Captioning
Ehsan Qasemi, Jonathan M. Francis, Alessandro Oltramari
Video Question Answering (VidQA) exhibits remarkable potential in facilitating advanced machine reasoning capabilities within the domains of Intelligent Traffic Monitoring and Inte…
A Study of Situational Reasoning for Traffic Understanding
Jiarui Zhang, Filip Ilievski, Kaixin Ma +3
Intelligent Traffic Monitoring (ITMo) technologies hold the potential for improving road safety/security and for enabling smart city infrastructure. Understanding traffic situation…
What Went Wrong? Closing the Sim-to-Real Gap via Differentiable Causal Discovery
Peide Huang, Xilun Zhang, Ziang Cao +6
Training control policies in simulation is more appealing than on real robots directly, as it allows for exploring diverse states in an efficient manner. Yet, robot simulators inev…
Knowledge-enhanced Agents for Interactive Text Games
Prateek Chhikara, Jiarui Zhang, Filip Ilievski +2
Communication via natural language is a key aspect of machine intelligence, and it requires computational models to learn and reason about world concepts, with varying levels of su…
Regularizing Self-training for Unsupervised Domain Adaptation via Structural Constraints
Rajshekhar Das, Jonathan Francis, Sanket Vaibhav Mehta +3
Self-training based on pseudo-labels has emerged as a dominant approach for addressing conditional distribution shifts in unsupervised domain adaptation (UDA) for semantic segmenta…
Core Challenges in Embodied Vision-Language Planning
Jonathan Francis, Nariaki Kitamura, Felix Labelle +3
Recent advances in the areas of Multimodal Machine Learning and Artificial Intelligence (AI) have led to the development of challenging tasks at the intersection of Computer Vision…
Cross-Tool and Cross-Behavior Perceptual Knowledge Transfer for Grounded Object Recognition
Gyan Tatiya, Jonathan Francis, Jivko Sinapov
Humans learn about objects via interaction and using multiple perceptions, such as vision, sound, and touch. While vision can provide information about an object's appearance, non-…
Knowledge-driven Scene Priors for Semantic Audio-Visual Embodied Navigation
Gyan Tatiya, Jonathan Francis, Luca Bondi +4
Generalisation to unseen contexts remains a challenge for embodied navigation agents. In the context of semantic audio-visual navigation (SAVi) tasks, the notion of generalisation…
Distribution-aware Goal Prediction and Conformant Model-based Planning for Safe Autonomous Driving
Jonathan Francis, Bingqing Chen, Weiran Yao +2
The feasibility of collecting a large amount of expert demonstrations has inspired growing research interests in learning-to-drive settings, where models learn by imitating the dri…
Utilizing Background Knowledge for Robust Reasoning over Traffic Situations
Jiarui Zhang, Filip Ilievski, Aravinda Kollaa +3
Understanding novel situations in the traffic domain requires an intricate combination of domain-specific and causal commonsense knowledge. Prior work has provided sufficient perce…
Transferring Implicit Knowledge of Non-Visual Object Properties Across Heterogeneous Robot Morphologies
Gyan Tatiya, Jonathan Francis, Jivko Sinapov
Humans leverage multiple sensor modalities when interacting with objects and discovering their intrinsic properties. Using the visual modality alone is insufficient for deriving in…
Coalescing Global and Local Information for Procedural Text Understanding
Kaixin Ma, Filip Ilievski, Jonathan Francis +2
Procedural text understanding is a challenging language reasoning task that requires models to track entity states across the development of a narrative. A complete procedural unde…
An Empirical Investigation of Commonsense Self-Supervision with Knowledge Graphs
Jiarui Zhang, Filip Ilievski, Kaixin Ma +2
Self-supervision based on the information extracted from large knowledge graphs has been shown to improve the generalization of language models, in zero-shot evaluation on various…
Learn-to-Race Challenge 2022: Benchmarking Safe Learning and Cross-domain Generalisation in Autonomous Racing
Jonathan Francis, Bingqing Chen, Siddha Ganju +10
We present the results of our autonomous racing virtual challenge, based on the newly-released Learn-to-Race (L2R) simulation framework, which seeks to encourage interdisciplinary…
Generalizable Neuro-symbolic Systems for Commonsense Question Answering
Alessandro Oltramari, Jonathan Francis, Filip Ilievski +2
This chapter illustrates how suitable neuro-symbolic models for language understanding can enable domain generalizability and robustness in downstream tasks. Different methods for…
Safe Autonomous Racing via Approximate Reachability on Ego-vision
Bingqing Chen, Jonathan Francis, Jean Oh +2
Racing demands each vehicle to drive at its physical limits, when any safety infraction could lead to catastrophic failure. In this work, we study the problem of safe reinforcement…
Exploring Strategies for Generalizable Commonsense Reasoning with Pre-trained Models
Kaixin Ma, Filip Ilievski, Jonathan Francis +3
Commonsense reasoning benchmarks have been largely solved by fine-tuning language models. The downside is that fine-tuning may cause models to overfit to task-specific data and the…
Core Challenges in Embodied Vision-Language Planning
Jonathan Francis, Nariaki Kitamura, Felix Labelle +3
Recent advances in the areas of multimodal machine learning and artificial intelligence (AI) have led to the development of challenging tasks at the intersection of Computer Vision…
Learn-to-Race: A Multimodal Control Environment for Autonomous Racing
James Herman, Jonathan Francis, Siddha Ganju +7
Existing research on autonomous driving primarily focuses on urban driving, which is insufficient for characterising the complex driving behaviour underlying high-speed racing. At…
Lexically-constrained Text Generation through Commonsense Knowledge Extraction and Injection
Yikang Li, Pulkit Goel, Varsha Kuppur Rajendra +5
Conditional text generation has been a challenging task that is yet to see human-level performance from state-of-the-art models. In this work, we specifically focus on the Commonge…
Knowledge-driven Data Construction for Zero-shot Evaluation in Commonsense Question Answering
Kaixin Ma, Filip Ilievski, Jonathan Francis +3
Recent developments in pre-trained neural language modeling have led to leaps in accuracy on commonsense question-answering benchmarks. However, there is increasing concern that mo…