195 citations · 495 across the 35 of their papers we have counts for
35 papers
Future Success Prediction in Open-Vocabulary Object Manipulation Tasks Based on End-Effector Trajectories
Motonari Kambara, Komei Sugiura
This study addresses a task designed to predict the future success or failure of open-vocabulary object manipulation. In this task, the model is required to make predictions based…
DENEB: A Hallucination-Robust Automatic Evaluation Metric for Image Captioning
Kazuki Matsuda, Yuiga Wada, Komei Sugiura
In this work, we address the challenge of developing automatic evaluation metrics for image captioning, with a particular focus on robustness against hallucinations. Existing metri…
Task Success Prediction for Open-Vocabulary Manipulation Based on Multi-Level Aligned Representations
Miyu Goko, Motonari Kambara, Daichi Saito +2
In this study, we consider the problem of predicting task success for open-vocabulary manipulation by a manipulator, based on instruction sentences and egocentric images before and…
DM2RM: Dual-Mode Multimodal Ranking for Target Objects and Receptacles Based on Open-Vocabulary Instructions
Ryosuke Korekata, Kanta Kaneda, Shunya Nagashima +2
In this study, we aim to develop a domestic service robot (DSR) that, guided by open-vocabulary instructions, can carry everyday objects to the specified pieces of furniture. Few e…
Layer-Wise Relevance Propagation with Conservation Property for ResNet
Seitaro Otsuki, Tsumugi Iida, Félix Doublet +4
The transparent formulation of explanation methods is essential for elucidating the predictions of neural networks, which are typically black-box models. Layer-wise Relevance Propa…
Polos: Multimodal Metric Learning from Human Feedback for Image Captioning
Yuiga Wada, Kanta Kaneda, Daichi Saito +1
Establishing an automatic evaluation metric that closely aligns with human judgments is essential for effectively developing image captioning models. Recent data-driven metrics hav…
Learning-To-Rank Approach for Identifying Everyday Objects Using a Physical-World Search Engine
Kanta Kaneda, Shunya Nagashima, Ryosuke Korekata +2
Domestic service robots offer a solution to the increasing demand for daily care and support. A human-in-the-loop approach that combines automation and operator intervention is con…
DialMAT: Dialogue-Enabled Transformer with Moment-Based Adversarial Training
Kanta Kaneda, Ryosuke Korekata, Yuiga Wada +7
This paper focuses on the DialFRED task, which is the task of embodied instruction following in a setting where an agent can actively ask questions about the task. To address this…
Fully Automated Task Management for Generation, Execution, and Evaluation: A Framework for Fetch-and-Carry Tasks with Natural Language Instructions in Continuous Space
Motonari Kambara, Komei Sugiura
This paper aims to develop a framework that enables a robot to execute tasks based on visual information, in response to natural language instructions for Fetch-and-Carry with Obje…
JaSPICE: Automatic Evaluation Metric Using Predicate-Argument Structures for Image Captioning Models
Yuiga Wada, Kanta Kaneda, Komei Sugiura
Image captioning studies heavily rely on automatic evaluation metrics such as BLEU and METEOR. However, such n-gram-based metrics have been shown to correlate poorly with human eva…
Multimodal Diffusion Segmentation Model for Object Segmentation from Manipulation Instructions
Yui Iioka, Yu Yoshida, Yuiga Wada +2
In this study, we aim to develop a model that comprehends a natural language instruction (e.g., "Go to the living room and get the nearest pillow to the radio art on the wall") and…
Switching Head-Tail Funnel UNITER for Dual Referring Expression Comprehension with Fetch-and-Carry Tasks
Ryosuke Korekata, Motonari Kambara, Yu Yoshida +4
This paper describes a domestic service robot (DSR) that fetches everyday objects and carries them to specified destinations according to free-form natural language instructions. G…
Prototypical Contrastive Transfer Learning for Multimodal Language Understanding
Seitaro Otsuki, Shintaro Ishikawa, Komei Sugiura
Although domestic service robots are expected to assist individuals who require support, they cannot currently interact smoothly with people through natural language. For example,…
Action Q-Transformer: Visual Explanation in Deep Reinforcement Learning with Encoder-Decoder Model using Action Query
Hidenori Itaya, Tsubasa Hirakawa, Takayoshi Yamashita +2
The excellent performance of Transformer in supervised learning has led to growing interest in its potential application to deep reinforcement learning (DRL) to achieve high perfor…
Visual Explanation of Deep Q-Network for Robot Navigation by Fine-tuning Attention Branch
Yuya Maruyama, Hiroshi Fukui, Tsubasa Hirakawa +3
Robot navigation with deep reinforcement learning (RL) achieves higher performance and performs well under complex environment. Meanwhile, the interpretation of the decision-making…
Relational Future Captioning Model for Explaining Likely Collisions in Daily Tasks
Motonari Kambara, Komei Sugiura
Domestic service robots that support daily tasks are a promising solution for elderly or disabled people. It is crucial for domestic service robots to explain the collision risk be…
Moment-based Adversarial Training for Embodied Language Comprehension
Shintaro Ishikawa, Komei Sugiura
In this paper, we focus on a vision-and-language task in which a robot is instructed to execute household tasks. Given an instruction such as "Rinse off a mug and place it in the c…
Operational solar flare prediction model using Deep Flare Net
Naoto Nishizuka, Yuki Kubo, Komei Sugiura +2
We developed an operational solar flare prediction model using deep neural networks, named Deep Flare Net (DeFN). DeFN can issue probabilistic forecasts of solar flares in two cate…
LatteGAN: Visually Guided Language Attention for Multi-Turn Text-Conditioned Image Manipulation
Shoya Matsumori, Yuki Abe, Kosuke Shingyouchi +2
Text-guided image manipulation tasks have recently gained attention in the vision-and-language community. While most of the prior studies focused on single-turn manipulation, our g…
Target-dependent UNITER: A Transformer-Based Multimodal Language Comprehension Model for Domestic Service Robots
Shintaro Ishikawa, Komei Sugiura
Currently, domestic service robots have an insufficient ability to interact naturally through language. This is because understanding human instructions is complicated by various a…
Case Relation Transformer: A Crossmodal Language Generation Model for Fetching Instructions
Motonari Kambara, Komei Sugiura
There have been many studies in robotics to improve the communication skills of domestic service robots. Most studies, however, have not fully benefited from recent advances in dee…
Unified Questioner Transformer for Descriptive Question Generation in Goal-Oriented Visual Dialogue
Shoya Matsumori, Kosuke Shingyouchi, Yuki Abe +3
Building an interactive artificial intelligence that can ask questions about the real world is one of the biggest challenges for vision and language problems. In particular, goal-o…
Visual Explanation using Attention Mechanism in Actor-Critic-based Deep Reinforcement Learning
Hidenori Itaya, Tsubasa Hirakawa, Takayoshi Yamashita +2
Deep reinforcement learning (DRL) has great potential for acquiring the optimal action in complex environments such as games and robot control. However, it is difficult to analyze…
CrossMap Transformer: A Crossmodal Masked Path Transformer Using Double Back-Translation for Vision-and-Language Navigation
Aly Magassouba, Komei Sugiura, Hisashi Kawai
Navigation guided by natural language instructions is particularly suitable for Domestic Service Robots that interacts naturally with users. This task involves the prediction of a…