26 papers
STARE: Scene Text-Aware multimodal Retrieval for Everyday objects
Kento Tokura, Ryosuke Korekata, Daichi Yashima +4
In this study, we focus on service robots that retrieve and transport everyday objects based on free-form instructions that may refer to visual features and scene text. This is cha…
CANDLE: Cortical Null-Space Decomposition for Noninvasive Brain Source Imaging
Shuntaro Suzuki, Yuiga Wada, Komei Sugiura
Electrophysiological source imaging (ESI) aims to estimate cortical source activity from noninvasive electrophysiological measurements such as electroencephalogram (EEG). However,…
NarrativeFlow: Flow-Based Vision-Language-Action Model Using Robot Velocity Fields
Shota Kobayashi, Koki Seno, Daichi Yashima +1
We focus on language-conditioned flow-based manipulation, where robot flows (robot velocity fields) serve as embodiment-agnostic, motion-centric representations for leveraging data…
Rigel: Self-Distilled Score Adaptation for Image and Video Captioning Evaluation
Shuitsu Koyama, Kazuki Matsuda, Yuiga Wada +3
Automatic evaluation of image and video captioning is essential for benchmarking multimodal systems, although standard evaluation metrics show limited alignment with human judgment…
MLLM-as-a-Judge Exhibits Model Preference Bias
Shuitsu Koyama, Yuiga Wada, Daichi Yashima +1
Automatic evaluation using multimodal large language models (MLLMs), commonly referred to as MLLM-as-a-Judge, has been widely used to measure model performance. If such MLLM-as-a-J…
Stitch4D: 4D Urban Reconstruction from Sparse Multi-Location Observations via Spatio-Temporal Interpolation
Hina Kogure, Kei Katsumata, Taiki Miyanishi +1
Dynamic urban environments are often captured by cameras placed at spatially separated locations with little or no view overlap. However, most existing 4D reconstruction methods as…
HiFlow: Tokenization-Free Scale-Wise Autoregressive Policy Learning via Flow Matching
Daichi Yashima, Koki Seno, Shuhei Kurita +2
Coarse-to-fine autoregressive modeling has recently shown strong promise for visuomotor policy learning, combining the inference efficiency of autoregressive methods with the globa…
LILAC: Language-Conditioned Object-Centric Optical Flow for Open-Loop Trajectory Generation
Motonari Kambara, Koki Seno, Tomoya Kaichi +2
We address language-conditioned robotic manipulation using flow-based trajectory generation, which enables training on human and web videos of object manipulation and requires only…
AnolisVLA: Lightweight Vision-Language-Action Model with Deep State Space Models for Mobile Manipulation
Yusuke Takagi, Motonari Kambara, Daichi Yashima +3
In this study, we address the problem of language-guided robotic manipulation, where a robot is required to manipulate a wide range of objects based on visual observations and natu…
NaiLIA: Multimodal Nail Design Retrieval Based on Dense Intent Descriptions and Palette Queries
Kanon Amemiya, Daichi Yashima, Kei Katsumata +4
We focus on the task of retrieving nail design images based on dense intent descriptions, which represent multi-layered user intent for nail designs. This is challenging because su…
ReMoRa: Multimodal Large Language Model based on Refined Motion Representation for Long-Video Understanding
Daichi Yashima, Shuhei Kurita, Yusuke Oda +1
While multimodal large language models (MLLMs) have shown remarkable success across a wide range of tasks, long-form video understanding remains a significant challenge. In this st…
LLM-Free Image Captioning Evaluation in Reference-Flexible Settings
Shinnosuke Hirano, Yuiga Wada, Kazuki Matsuda +2
We focus on the automatic evaluation of image captions in both reference-based and reference-free settings. Existing metrics based on large language models (LLMs) favor their own g…
Affordance RAG: Hierarchical Multimodal Retrieval with Affordance-Aware Embodied Memory for Mobile Manipulation
Ryosuke Korekata, Quanting Xie, Yonatan Bisk +1
In this study, we address the problem of open-vocabulary mobile manipulation, where a robot is required to carry a wide range of objects to receptacles based on free-form natural l…
Cortical-SSM: A Deep State Space Model for Motor Imagery Decoding from EEG Signals
Shuntaro Suzuki, Shunya Nagashima, Komei Sugiura
Classification of electroencephalogram (EEG) signals obtained during motor imagery (MI) has substantial application potential, including communication assistance and rehabilitation…
VELA: An LLM-Hybrid-as-a-Judge Approach for Evaluating Long Image Captions
Kazuki Matsuda, Yuiga Wada, Shinnosuke Hirano +2
In this study, we focus on the automatic evaluation of long and detailed image captions generated by multimodal Large Language Models (MLLMs). Most existing automatic evaluation me…
Attention Lattice Adapter: Visual Explanation Generation for Visual Foundation Model
Shinnosuke Hirano, Yuiga Wada, Tsumugi Iida +1
In this study, we consider the problem of generating visual explanations in visual foundation models. Numerous methods have been proposed for this purpose; however, they often cann…
Pre-Manipulation Alignment Prediction with Parallel Deep State-Space and Transformer Models
Motonari Kambara, Komei Sugiura
In this work, we address the problem of predicting the future success of open-vocabulary object manipulation tasks. Conventional approaches typically determine success or failure a…
FLARE-SSM: Deep State Space Models with Influence-Balanced Loss for 72-Hour Solar Flare Prediction
Yusuke Takagi, Shunya Nagashima, Komei Sugiura
Accurate and reliable solar flare predictions are essential to mitigate potential impacts on critical infrastructure. However, the current performance of solar flare forecasting is…
GENNAV: Polygon Mask Generation for Generalized Referring Navigable Regions
Kei Katsumata, Yui Iioka, Naoki Hosomi +3
We focus on the task of identifying the location of target regions from a natural language instruction and a front camera image captured by a mobility. This task is challenging bec…
Deep Space Weather Model: Long-Range Solar Flare Prediction from Multi-Wavelength Images
Shunya Nagashima, Komei Sugiura
Accurate, reliable solar flare prediction is crucial for mitigating potential disruptions to critical infrastructure, while predicting solar flares remains a significant challenge.…
ZINA: Multimodal Fine-grained Hallucination Detection and Editing
Yuiga Wada, Kazuki Matsuda, Komei Sugiura +1
Multimodal Large Language Models (MLLMs) often generate hallucinations, where the output deviates from the visual content. Given that these hallucinations can take diverse forms, d…
Mobile Manipulation Instruction Generation from Multiple Images with Automatic Metric Enhancement
Kei Katsumata, Motonari Kambara, Daichi Yashima +2
We consider the problem of generating free-form mobile manipulation instructions based on a target object image and receptacle image. Conventional image captioning models are not a…
Open-Vocabulary Mobile Manipulation Based on Double Relaxed Contrastive Learning with Dense Labeling
Daichi Yashima, Ryosuke Korekata, Komei Sugiura
Growing labor shortages are increasing the demand for domestic service robots (DSRs) to assist in various settings. In this study, we develop a DSR that transports everyday objects…
Nearest Neighbor Future Captioning: Generating Descriptions for Possible Collisions in Object Placement Tasks
Takumi Komatsu, Motonari Kambara, Shumpei Hatanaka +5
Domestic service robots (DSRs) that support people in everyday environments have been widely investigated. However, their ability to predict and describe future risks resulting fro…