13 citations · 31 across the 15 of their papers we have counts for
15 papers
DeepOmni: Towards Seamless and Smart Speech Interaction with Adaptive Modality-Specific MoE
Hang Shao, Heting Gao, Yunhang Shen +5
Native multimodal large language models (MLLMs) restructure a single large language model (LLM) into a spoken language model (SLM) capable of both speech and text generation. Compa…
LLaVA-RadZ: Can Multimodal Large Language Models Effectively Tackle Zero-shot Radiology Recognition?
Bangyan Li, Wenxuan Huang, Zhenkun Gao +8
Recently, Multimodal Large Language Models (MLLMs) have demonstrated exceptional capabilities in visual understanding and reasoning across various vision-language tasks. However, w…
Probability-density-aware Semi-supervised Learning
Shuyang Liu, Ruiqiu Zheng, Yunhang Shen +4
Semi-supervised learning (SSL) assumes that neighbor points lie in the same category (neighbor assumption), and points in different clusters belong to various categories (cluster a…
VEGA: Learning Interleaved Image-Text Comprehension in Vision-Language Large Models
Chenyu Zhou, Mengdan Zhang, Peixian Chen +5
The swift progress of Multi-modal Large Models (MLLMs) has showcased their impressive ability to tackle tasks blending vision and language. Yet, most current models and benchmarks…
Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis
Chaoyou Fu, Yuhan Dai, Yongdong Luo +18
In the quest for artificial general intelligence, Multi-modal Large Language Models (MLLMs) have emerged as a focal point in recent advancements. However, the predominant focus rem…
Cantor: Inspiring Multimodal Chain-of-Thought of MLLM
Timin Gao, Peixian Chen, Mengdan Zhang +8
With the advent of large language models(LLMs) enhanced by the chain-of-thought(CoT) methodology, visual reasoning problem is usually decomposed into manageable sub-tasks and tackl…
A General and Efficient Training for Transformer via Token Expansion
Wenxuan Huang, Yunhang Shen, Jiao Xie +5
The remarkable performance of Vision Transformers (ViTs) typically requires an extremely large training cost. Existing methods have attempted to accelerate the training of ViTs, ye…
RESTORE: Towards Feature Shift for Vision-Language Prompt Learning
Yuncheng Yang, Chuyan Zhang, Zuopeng Yang +6
Prompt learning is effective for fine-tuning foundation models to improve their generalization across a variety of downstream tasks. However, the prompts that are independently opt…
Sinkhorn Distance Minimization for Knowledge Distillation
Xiao Cui, Yulei Qin, Yuting Gao +7
Knowledge distillation (KD) has been widely adopted to compress large language models (LLMs). Existing KD methods investigate various divergence measures including the Kullback-Lei…
A Challenger to GPT-4V? Early Explorations of Gemini in Visual Expertise
Chaoyou Fu, Renrui Zhang, Zihan Wang +15
The surge of interest towards Multi-modal Large Language Models (MLLMs), e.g., GPT-4V(ision) from OpenAI, has marked a significant trend in both academia and industry. They endow L…
SPD-DDPM: Denoising Diffusion Probabilistic Models in the Symmetric Positive Definite Space
Yunchen Li, Zhou Yu, Gaoqi He +4
Symmetric positive definite~(SPD) matrices have shown important value and applications in statistics and machine learning, such as FMRI analysis and traffic prediction. Previous wo…
Aligning and Prompting Everything All at Once for Universal Visual Perception
Yunhang Shen, Chaoyou Fu, Peixian Chen +6
Vision foundation models have been explored recently to build general-purpose vision systems. However, predominant paradigms, driven by casting instance-level tasks as an object-wo…
MMICT: Boosting Multi-Modal Fine-Tuning with In-Context Examples
Tao Chen, Enwei Zhang, Yuting Gao +5
Although In-Context Learning (ICL) brings remarkable performance gains to Large Language Models (LLMs), the improvements remain lower than fine-tuning on downstream tasks. This pap…
Towards Robust Text Retrieval with Progressive Learning
Tong Wu, Yulei Qin, Enwei Zhang +4
Retrieval augmentation has become an effective solution to empower large language models (LLMs) with external and verified knowledge sources from the database, which overcomes the…
CAPro: Webly Supervised Learning with Cross-Modality Aligned Prototypes
Yulei Qin, Xingyu Chen, Yunhang Shen +5
Webly supervised learning has attracted increasing attention for its effectiveness in exploring publicly accessible data at scale without manual annotation. However, most existing…