43 citations · 68 across the 20 of their papers we have counts for
23 papers
ComBodied Agents: a New Paradigm of Human-Centric Agentic AI
Qianggang Ding, Xingyao Wang, Rui Feng +19
After an older adult misses a medication dose, a software agent can send another reminder and an embodied agent can bring the medication. Yet neither explains whether the person fo…
Beyond Captions: Context-Grounded Reconstruction for Biomedical Multimodal Continued Pretraining
Guanghao Zhu, Zeyu Liu, Zhitian Hou +10
Biomedical figures are explained not by captions alone but by body-text passages that discuss them. Yet current multimodal corpora typically reduce figures to isolated image-captio…
CoEditor++: Instruction-based Visual Editing via Cognitive Reasoning
Minheng Ni, Yutao Fan, Zhengyuan Yang +6
Recent advances in large multimodal models (LMMs) have enabled instruction-based image editing, allowing users to modify visual content via natural language descriptions. However,…
ResponsibleRobotBench: Benchmarking Responsible Robot Manipulation using Multi-modal Large Language Models
Lei Zhang, Ju Dong, Kaixin Bai +4
Recent advances in large multimodal models have enabled new opportunities in embodied AI, particularly in robotic manipulation. These models have shown strong potential in generali…
TechImage-Bench: Rubric-Based Evaluation for Technical Image Generation
Minheng Ni, Zhengyuan Yang, Yaowen Zhang +9
We study technical image generation, where a model must synthesize information-dense, scientifically precise illustrations from detailed descriptions rather than merely produce vis…
MIRAGE: Assessing Hallucination in Multimodal Reasoning Chains of MLLM
Bowen Dong, Minheng Ni, Zitong Huang +3
Multimodal hallucination in multimodal large language models (MLLMs) restricts the correctness of MLLMs. However, multimodal hallucinations are multi-sourced and arise from diverse…
Point-RFT: Improving Multimodal Reasoning with Visually Grounded Reinforcement Finetuning
Minheng Ni, Zhengyuan Yang, Linjie Li +4
Recent advances in large language models have significantly improved textual reasoning through the effective use of Chain-of-Thought (CoT) and reinforcement learning. However, exte…
Measurement of LLM's Philosophies of Human Nature
Minheng Ni, Ennan Wu, Zidong Gong +6
The widespread application of artificial intelligence (AI) in various tasks, along with frequent reports of conflicts or violations involving AI, has sparked societal concerns abou…
Don't Let Your Robot be Harmful: Responsible Robotic Manipulation via Safety-as-Policy
Minheng Ni, Lei Zhang, Zihan Chen +4
Unthinking execution of human instructions in robotic manipulation can lead to severe safety risks, such as poisonings, fires, and even explosions. In this paper, we present respon…
Visual-O1: Understanding Ambiguous Instructions via Multi-modal Multi-turn Chain-of-thoughts Reasoning
Minheng Ni, Yutao Fan, Lei Zhang +1
As large-scale models evolve, language instructions are increasingly utilized in multi-modal tasks. Due to human language habits, these instructions often contain ambiguities in re…
AutoDirector: Online Auto-scheduling Agents for Multi-sensory Composition
Minheng Ni, Chenfei Wu, Huaying Yuan +6
With the advancement of generative models, the synthesis of different sensory elements such as music, visuals, and speech has achieved significant realism. However, the approach to…
Responsible Visual Editing
Minheng Ni, Yeli Shen, Lei Zhang +1
With recent advancements in visual synthesis, there is a growing risk of encountering images with detrimental effects, such as hate, discrimination, or privacy violations. The rese…
Ref-Diff: Zero-shot Referring Image Segmentation with Generative Models
Minheng Ni, Yabo Zhang, Kailai Feng +3
Zero-shot referring image segmentation is a challenging task because it aims to find an instance segmentation mask based on the given referring descriptions, without training on th…
ORES: Open-vocabulary Responsible Visual Synthesis
Minheng Ni, Chenfei Wu, Xiaodong Wang +4
Avoiding synthesizing specific visual concepts is an essential challenge in responsible visual synthesis. However, the visual concept that needs to be avoided for responsible visua…
NUWA-XL: Diffusion over Diffusion for eXtremely Long Video Generation
Shengming Yin, Chenfei Wu, Huan Yang +13
In this paper, we propose NUWA-XL, a novel Diffusion over Diffusion architecture for eXtremely Long video generation. Most current work generates long videos segment by segment seq…
Learning 3D Photography Videos via Self-supervised Diffusion on Single Images
Xiaodong Wang, Chenfei Wu, Shengming Yin +9
3D photography renders a static image into a video with appealing 3D visual effects. Existing approaches typically first conduct monocular depth estimation, then render the input f…
ImaginaryNet: Learning Object Detectors without Real Images and Annotations
Minheng Ni, Zitong Huang, Kailai Feng +1
Without the demand of training in reality, humans can easily detect a known concept simply based on its language description. Empowering deep learning with this ability undoubtedly…
NÜWA-LIP: Language Guided Image Inpainting with Defect-free VQGAN
Minheng Ni, Chenfei Wu, Haoyang Huang +3
Language guided image inpainting aims to fill in the defective regions of an image under the guidance of text while keeping non-defective regions unchanged. However, the encoding p…
Co-GAT: A Co-Interactive Graph Attention Network for Joint Dialog Act Recognition and Sentiment Classification
Libo Qin, Zhouyang Li, Wanxiang Che +2
In a dialog system, dialog act recognition and sentiment classification are two correlative tasks to capture speakers intentions, where dialog act and sentiment can indicate the ex…
DCR-Net: A Deep Co-Interactive Relation Network for Joint Dialog Act Recognition and Sentiment Classification
Libo Qin, Wanxiang Che, Yangming Li +2
In dialog system, dialog act recognition and sentiment classification are two correlative tasks to capture speakers intentions, where dialog act and sentiment can indicate the expl…
CoSDA-ML: Multi-Lingual Code-Switching Data Augmentation for Zero-Shot Cross-Lingual NLP
Libo Qin, Minheng Ni, Yue Zhang +1
Multi-lingual contextualized embeddings, such as multilingual-BERT (mBERT), have shown success in a variety of zero-shot cross-lingual tasks. However, these models are limited by h…
M3P: Learning Universal Representations via Multitask Multilingual Multimodal Pre-training
Minheng Ni, Haoyang Huang, Lin Su +6
We present M3P, a Multitask Multilingual Multimodal Pre-trained model that combines multilingual pre-training and multimodal pre-training into a unified framework via multitask pre…
Multi-Domain Spoken Language Understanding Using Domain- and Task-Aware Parameterization
Libo Qin, Minheng Ni, Yue Zhang +3
Spoken language understanding has been addressed as a supervised learning problem, where a set of training data is available for each domain. However, annotating data for each doma…