11 papers
SVoT: State-aware Visualization-of-Thought for Spatial Reasoning via Reinforcement Learning
Chao Lei, Yanbei Jiang, Markus Hiller +4
Spatial reasoning remains a challenge for Multimodal Large Language Models (MLLMs), as it requires reliable multi-hop inference over both intermediate states and state transitions.…
Mind the Perspective: Let's Reason Recursively for Theory of Mind
Chao Lei, Guang Hu, Meng Yang +2
Theory of Mind (ToM) reasoning requires inferring agents' beliefs from partial and asymmetric observations, which remains an open challenge for LLMs. Existing prompting-based appro…
Reward on Path: Learning Intermediate Supervision Signals for Knowledge Graph Question Answering
Shengxiang Gao, Chao Lei, Jey Han Lau +2
Knowledge Graph Question Answering (KGQA) aims to answer user questions by reasoning over Knowledge Graphs (KGs). Recent methods use supervision derived from answer labels or refin…
MIDI-LLaMA: An Instruction-Following Multimodal LLM for Symbolic Music Understanding
Meng Yang, Jon McCormack, Maria Teresa Llano +2
Recent advances in multimodal large language models (MLLM) for audio music have demonstrated strong capabilities in music understanding, yet symbolic music, a fundamental represent…
PROPA: Toward Process-level Optimization in Visual Reasoning via Reinforcement Learning
Yanbei Jiang, Chao Lei, Yihao Ding +2
Despite significant progress, Vision-Language Models (VLMs) still struggle with complex visual reasoning, where multi-step dependencies cause early errors to cascade through the re…
Beyond Perception: Evaluating Abstract Visual Reasoning through Multi-Stage Task
Yanbei Jiang, Yihao Ding, Chao Lei +3
Current Multimodal Large Language Models (MLLMs) excel in general visual reasoning but remain underexplored in Abstract Visual Reasoning (AVR), which demands higher-order reasoning…
From Reasoning to Generalization: Knowledge-Augmented LLMs for ARC Benchmark
Chao Lei, Nir Lipovetzky, Krista A. Ehinger +1
Recent reasoning-oriented LLMs have demonstrated strong performance on challenging tasks such as mathematics and science examinations. However, core cognitive faculties of human in…
State-Based Disassembly Planning
Chao Lei, Nir Lipovetzky, Krista A. Ehinger
It has been shown recently that physics-based simulation significantly enhances the disassembly capabilities of real-world assemblies with diverse 3D shapes and stringent motion co…
Planning-Driven Programming: A Large Language Model Programming Workflow
Chao Lei, Yanchuan Chang, Nir Lipovetzky +1
The strong performance of large language models (LLMs) raises extensive discussion on their application to code generation. Recent research suggests continuous program refinements…
Generalized Planning for the Abstraction and Reasoning Corpus
Chao Lei, Nir Lipovetzky, Krista A. Ehinger
The Abstraction and Reasoning Corpus (ARC) is a general artificial intelligence benchmark that poses difficulties for pure machine learning methods due to its requirement for fluid…
The GUA-Speech System Description for CNVSRC Challenge 2023
Shengqiang Li, Chao Lei, Baozhong Ma +2
This study describes our system for Task 1 Single-speaker Visual Speech Recognition (VSR) fixed track in the Chinese Continuous Visual Speech Recognition Challenge (CNVSRC) 2023. S…