Publications (55)
MIKO: Multimodal Intention Knowledge Distillation from Large Language Models for Social-Media Commonsense Discovery
Feihong Lu, Weiqi Wang, Yangyifei Luo +8
Social media has become a ubiquitous tool for connecting with others, staying updated with news, expressing opinions, and finding entertainment. However, understanding the intentio…
RLPF: Reinforcement Learning from Performance Feedback for Code Generation
Huihao Jing, Haozhe Cui, Wenbin Hu +9
The paper introduces RLPF, a reinforcement‑learning approach that uses staged performance feedback to train code‑generation models to produce not only correct programs but also fas…
The Curse of CoT: On the Limitations of Chain-of-Thought in In-Context Learning
Tianshi Zheng, Yixiang Chen, Chengxi Li +7
Chain-of-Thought (CoT) prompting has been widely recognized for its ability to enhance reasoning capabilities in large language models (LLMs). However, our study reveals a surprisi…
Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systems
Bang Liu, Xinfeng Li, Jiayi Zhang +45
The advent of large language models (LLMs) has catalyzed a transformative shift in artificial intelligence, paving the way for advanced intelligent agents capable of sophisticated…
BOSS: Bottom-up Cross-modal Semantic Composition with Hybrid Counterfactual Training for Robust Content-based Image Retrieval
Wenqiao Zhang, Jiannan Guo, Mengze Li +5
Content-Based Image Retrieval (CIR) aims to search for a target image by concurrently comprehending the composition of an example image and a complementary text, which potentially…
A Pre-trained Data Deduplication Model based on Active Learning
Haochen Shi, Xinyao Liu, Fengmao Lv +4
In the era of big data, the issue of data quality has become increasingly prominent. One of the main challenges is the problem of duplicate data, which can arise from repeated entr…
On the Role of Entity and Event Level Conceptualization in Generalizable Reasoning: A Survey of Tasks, Methods, Applications, and Future Directions
Weiqi Wang, Tianqing Fang, Haochen Shi +8
Conceptualization, a fundamental element of human cognition, plays a pivotal role in human generalizable reasoning. Generally speaking, it refers to the process of sequentially abs…
RoboCraft: Learning to See, Simulate, and Shape Elasto-Plastic Objects with Graph Networks
Haochen Shi, Huazhe Xu, Zhiao Huang +2
Modeling and manipulating elasto-plastic objects are essential capabilities for robots to perform complex industrial and household interaction tasks (e.g., stuffing dumplings, roll…
Empower Distantly Supervised Relation Extraction with Collaborative Adversarial Training
Tao Chen, Haochen Shi, Liyuan Liu +4
With recent advances in distantly supervised (DS) relation extraction (RE), considerable attention is attracted to leverage multi-instance learning (MIL) to distill high-quality su…
ToddlerBot: Open-Source ML-Compatible Humanoid Platform for Loco-Manipulation
Haochen Shi, Weizhuo Wang, Shuran Song +1
Learning-based robotics research driven by data demands a new approach to robot hardware design-one that serves as both a platform for policy execution and a tool for embodied data…
Enhancing Agent Learning through World Dynamics Modeling
Zhiyuan Sun, Haochen Shi, Marc-Alexandre Côté +3
Large language models (LLMs) have been increasingly applied to tasks in language understanding and interactive decision-making, with their impressive performance largely attributed…
DexCap: Scalable and Portable Mocap Data Collection System for Dexterous Manipulation
Chen Wang, Haochen Shi, Weizhuo Wang +3
Imitation learning from human hand motion data presents a promising avenue for imbuing robots with human-like dexterity in real-world manipulation tasks. Despite this potential, su…
OPEx: A Component-Wise Analysis of LLM-Centric Agents in Embodied Instruction Following
Haochen Shi, Zhiyuan Sun, Xingdi Yuan +2
Embodied Instruction Following (EIF) is a crucial task in embodied learning, requiring agents to interact with their environment through egocentric observations to fulfill natural…
Consensus Graph Representation Learning for Better Grounded Image Captioning
Wenqiao Zhang, Haochen Shi, Siliang Tang +3
The contemporary visual captioning models frequently hallucinate objects that are not actually in a scene, due to the visual misclassification or over-reliance on priors that resul…
Robot Trains Robot: Automatic Real-World Policy Adaptation and Learning for Humanoids
Kaizhe Hu, Haochen Shi, Yao He +3
Simulation-based reinforcement learning (RL) has significantly advanced humanoid locomotion tasks, yet direct real-world RL from scratch or adapting from pretrained policies remain…
History-Guided Iterative Visual Reasoning with Self-Correction
Xinglong Yang, Zhilin Peng, Zhanzhan Liu +2
Self-consistency methods are the core technique for improving the reasoning reliability of multimodal large language models (MLLMs). By generating multiple reasoning results throug…
MultivationBench: A Benchmark for Multimodal Sequential Motivation Reasoning
Kawai Chung, Chunkit Chan, Yauwai Yim +12
The paper introduces MultivationBench, a benchmark that tests multimodal large language models on their ability to reason about evolving human motivations across sequential visual…
Safety Compliance: Rethinking LLM Safety Reasoning through the Lens of Compliance
Wenbin Hu, Huihao Jing, Haochen Shi +2
The proliferation of Large Language Models (LLMs) has demonstrated remarkable capabilities, elevating the critical importance of LLM safety. However, existing safety methods rely o…
INFERENCEDYNAMICS: Efficient Routing Across LLMs through Structured Capability and Knowledge Profiling
Haochen Shi, Tianshi Zheng, Weiqi Wang +6
Large Language Model (LLM) routing is a pivotal technique for navigating a diverse landscape of LLMs, aiming to select the best-performing LLMs tailored to the domains of user quer…
LogiDynamics: Unraveling the Dynamics of Inductive, Abductive and Deductive Logical Inferences in LLM Reasoning
Tianshi Zheng, Jiayang Cheng, Chunyang Li +6
Modern large language models (LLMs) employ diverse logical inference mechanisms for reasoning, making the strategic optimization of these approaches critical for advancing their ca…
Towards Multi-Agent Reasoning Systems for Collaborative Expertise Delegation: An Exploratory Design Study
Baixuan Xu, Chunyang Li, Weiqi Wang +6
Designing effective collaboration structure for multi-agent LLM systems to enhance collective reasoning is crucial yet remains under-explored. In this paper, we systematically inve…
RoboPack: Learning Tactile-Informed Dynamics Models for Dense Packing
Bo Ai, Stephen Tian, Haochen Shi +4
Tactile feedback is critical for understanding the dynamics of both rigid and deformable objects in many manipulation tasks, such as non-prehensile manipulation and dense packing.…
Locomotion Beyond Feet
Tae Hoon Yang, Haochen Shi, Jiacheng Hu +10
Most locomotion methods for humanoid robots focus on leg-based gaits, yet natural bipeds frequently rely on hands, knees, and elbows to establish additional contacts for stability…
RoboCook: Long-Horizon Elasto-Plastic Object Manipulation with Diverse Tools
Haochen Shi, Huazhe Xu, Samuel Clarke +2
Humans excel in complex long-horizon soft body manipulation tasks via flexible tool use: bread baking requires a knife to slice the dough and a rolling pin to flatten it. Often reg…
Adaptive Hierarchical Graph Reasoning with Semantic Coherence for Video-and-Language Inference
Juncheng Li, Siliang Tang, Linchao Zhu +5
Video-and-Language Inference is a recently proposed task for joint video-and-language understanding. This new task requires a model to draw inference on whether a natural language…
OmniCompliance-100K: A Multi-Domain, Rule-Grounded, Real-World Safety Compliance Dataset
Wenbin Hu, Huihao Jing, Haochen Shi +3
Ensuring the safety and compliance of large language models (LLMs) is of paramount importance. However, existing LLM safety datasets often rely on ad-hoc taxonomies for data genera…
CANDLE: Iterative Conceptualization and Instantiation Distillation from Large Language Models for Commonsense Reasoning
Weiqi Wang, Tianqing Fang, Chunyang Li +9
The sequential process of conceptualization and instantiation is essential to generalizable commonsense reasoning as it allows the application of existing knowledge to unfamiliar s…
PerfCodeBench: Benchmarking LLMs for System-Level High-Performance Code Optimization
Huihao Jing, Wenbin Hu, Shaojin Chen +5
The paper introduces PerfCodeBench, an executable benchmark that evaluates how well large language models can generate system-level code that is not only correct but also optimized…
CollisionIK: A Per-Instant Pose Optimization Method for Generating Robot Motions with Environment Collision Avoidance
Daniel Rakita, Haochen Shi, Bilge Mutlu +1
In this work, we present a per-instant pose optimization method that can generate configurations that achieve specified pose or motion objectives as best as possible over a sequenc…
PersonalPlan: Planning Multi-Agent Systems for Personalized Programming Learning
Zhiyuan Wen, Jiannong Cao, Peng Gao +4
Effective programming education requires personalized instruction adapted to diverse learner backgrounds. However, while LLM-based multi-agent systems (MAS) excel at complex planni…
Isolation as a First-Class Principle for LLM-Agent System Safety: Concepts, Taxonomy, Challenges and Future Directions
Huihao Jing, Wenbin Hu, Shaojin Chen +10
The paper surveys how isolating components such as user inputs, tools, execution, inter‑agent communication, and environment can improve safety of LLM‑agent systems, presenting a b…
Handroid: Bridging Dexterous Hand and Humanoid
Ruogu Li, Chenyang Ma, Sikai Li +6
Dexterous hands and humanoid robots are typically developed as distinct embodiments: the former enable contact-rich manipulation at the object scale, whereas the latter provide mob…
EcomEdit: An Automated E-commerce Knowledge Editing Framework for Enhanced Product and Purchase Intention Understanding
Ching Ming Samuel Lau, Weiqi Wang, Haochen Shi +3
Knowledge Editing (KE) aims to correct and update factual information in Large Language Models (LLMs) to ensure accuracy and relevance without computationally expensive fine-tuning…
ActPlan-1K: Benchmarking the Procedural Planning Ability of Visual Language Models in Household Activities
Ying Su, Zhan Ling, Haochen Shi +3
Large language models~(LLMs) have been adopted to process textual task description and accomplish procedural planning in embodied AI tasks because of their powerful reasoning abili…
Deciphering Digital Detectives: Understanding LLM Behaviors and Capabilities in Multi-Agent Mystery Games
Dekun Wu, Haochen Shi, Zhiyuan Sun +1
In this study, we explore the application of Large Language Models (LLMs) in \textit{Jubensha}, a Chinese detective role-playing game and a novel area in Artificial Intelligence (A…
SurveyLens: A Discipline-Aware Benchmark for Automatic Survey Generation
Beichen Guo, Zhiyuan Wen, Jia Gu +6
Automatic Survey Generation (ASG) aims to produce comprehensive literature surveys by retrieving, organizing, and synthesizing academic papers. Despite rapid progress in specialize…
Minimalist Compliance Control
Haochen Shi, Songbo Hu, Yifan Hou +3
Compliance control is essential for safe physical interaction, yet its adoption is limited by hardware requirements such as force torque sensors. While recent reinforcement learnin…
CritiCal: Can Critique Help LLM Uncertainty or Confidence Calibration?
Qing Zong, Jiayu Liu, Tianshi Zheng +7
Accurate confidence calibration in Large Language Models (LLMs) is critical for safe use in high-stakes domains, where clear verbalized confidence enhances user trust. Traditional…
FürElise: Capturing and Physically Synthesizing Hand Motions of Piano Performance
Ruocheng Wang, Pei Xu, Haochen Shi +2
Piano playing requires agile, precise, and coordinated hand control that stretches the limits of dexterity. Hand motion models with the sophistication to accurately recreate piano…
DishSeg24k: A Large-Scale Benchmark for Food Segmentation with Stochastic Expert Decoding
Yilin Wang, Haochen Shi, Guanyu Chen +4
Food segmentation is essential for applications such as intelligent catering, dietary assessment, and recommendation. However, existing benchmarks fail to capture the complexity of…
Dilated Context Integrated Network with Cross-Modal Consensus for Temporal Emotion Localization in Videos
Juncheng Li, Junlin Xie, Linchao Zhu +8
Understanding human emotions is a crucial ability for intelligent robots to provide better human-robot interactions. The existing works are limited to trimmed video-level emotion c…
Robust Modality-incomplete Anomaly Detection: A Modality-instructive Framework with Benchmark
Bingchen Miao, Wenqiao Zhang, Juncheng Li +6
Multimodal Industrial Anomaly Detection (MIAD), which utilizes 3D point clouds and 2D RGB images to identify abnormal regions in products, plays a crucial role in industrial qualit…
AbsPyramid: Benchmarking the Abstraction Ability of Language Models with a Unified Entailment Graph
Zhaowei Wang, Haochen Shi, Weiqi Wang +5
Cognitive research indicates that abstraction ability is essential in human intelligence, which remains under-explored in language models. In this paper, we present AbsPyramid, a u…
QADYNAMICS: Training Dynamics-Driven Synthetic QA Diagnostic for Zero-Shot Commonsense Question Answering
Haochen Shi, Weiqi Wang, Tianqing Fang +4
Zero-shot commonsense Question-Answering (QA) requires models to reason about general situations beyond specific benchmarks. State-of-the-art approaches fine-tune language models o…
Rethinking Prospect Theory for LLMs: Revealing the Instability of Decision-Making under Epistemic Uncertainty
Rui Wang, Qihan Lin, Jiayu Liu +7
Real-world decision-making often involves uncertainty expressed in linguistic rather than numerical terms, and Prospect Theory (PT) provides a classic framework for modeling human…
Open X-Embodiment: Robotic Learning Datasets and RT-X Models
Embodiment Collaboration, Abby O'Neill, Abdul Rehman +291
Large, high-capacity models trained on diverse datasets have shown remarkable successes on efficiently tackling downstream applications. In domains from NLP to Computer Vision, thi…
TradExpert: Revolutionizing Trading with Mixture of Expert LLMs
Qianggang Ding, Haochen Shi, Jiadong Guo +1
The integration of Artificial Intelligence (AI) in the financial domain has opened new avenues for quantitative trading, particularly through the use of Large Language Models (LLMs…
TILFA: A Unified Framework for Text, Image, and Layout Fusion in Argument Mining
Qing Zong, Zhaowei Wang, Baixuan Xu +6
A main goal of Argument Mining (AM) is to analyze an author's stance. Unlike previous AM datasets focusing only on text, the shared task at the 10th Workshop on Argument Mining int…
NOVA: NOise-aware Verbal Confidence CAlibration for Robust Large Language Models in RAG Systems
Jiayu Liu, Rui Wang, Qing Zong +9
Accurately assessing model confidence is essential for deploying large language models (LLMs) in mission-critical factual domains. While retrieval-augmented generation (RAG) is wid…
TRACE: Temporal Rule-Anchored Chain-of-Evidence on Knowledge Graphs for Interpretable Stock Movement Prediction
Qianggang Ding, Haochen Shi, Luis Castejón Lozano +7
We present a Temporal Rule-Anchored Chain-of-Evidence (TRACE) on knowledge graphs for interpretable stock movement prediction that unifies symbolic relational priors, dynamic graph…
MAGIC: Multimodal relAtional Graph adversarIal inferenCe for Diverse and Unpaired Text-based Image Captioning
Wenqiao Zhang, Haochen Shi, Jiannan Guo +5
Text-based image captioning (TextCap) requires simultaneous comprehension of visual content and reading the text of images to generate a natural language description. Although a ta…
Evolving Programmatic Skill Networks
Haochen Shi, Xingdi Yuan, Bang Liu
We study continual skill acquisition in open-ended embodied environments where an agent must construct, refine, and reuse an expanding library of executable skills. We introduce th…
Phase-Specific Augmented Reality Guidance for Microscopic Cataract Surgery Using Long-Short Spatiotemporal Aggregation Transformer
Puxun Tu, Hongfei Ye, Haochen Shi +6
Phacoemulsification cataract surgery (PCS) is a routine procedure conducted using a surgical microscope, heavily reliant on the skill of the ophthalmologist. While existing PCS gui…
SING: Synthetic Intention Graph for Scalable Active Tool Discovery in LLM Agents
Qiao Xiao, Haochen Shi, Yisen Gao +9
Large language model (LLM) agents increasingly rely on agent harnesses that manage context, tools, and multi-turn execution, making tools a central interface for acting in realisti…
MIND: Multimodal Shopping Intention Distillation from Large Vision-language Models for E-commerce Purchase Understanding
Baixuan Xu, Weiqi Wang, Haochen Shi +12
Improving user experience and providing personalized search results in E-commerce platforms heavily rely on understanding purchase intention. However, existing methods for acquirin…