14 papers
GAMW: Generation-Aware Mel-Domain Watermarking for TTS Traceability
Yihang Zhang, Weicai Yan, Qifei Zhang +2
As text-to-speech (TTS) systems continue to advance, provenance tracking and security verification of synthesized speech have become increasingly important. Audio watermarking prov…
Text-Guided Multi-Scale Frequency Representation Adaptation
Weicai Yan, Xinhua Ma, Wang Lin +1
Parameter-efficient fine-tuning methods introduce a small number of training parameters, enabling pre-trained models to adapt rapidly to new data distributions. While these methods…
Proact-VL: A Proactive VideoLLM for Real-Time AI Companions
Weicai Yan, Yuhong Dai, Qi Ran +6
Proactive and real-time interactive experiences are essential for human-like AI companions, yet face three key challenges: (1) achieving low-latency inference under continuous stre…
Decoupling Stability and Plasticity for Multi-Modal Test-Time Adaptation
Yongbo He, Zirun Guo, Tao Jin
Adapting pretrained multi-modal models to evolving test-time distributions, known as multi-modal test-time adaptation, presents a significant challenge. Existing methods frequently…
Thinking with Programming Vision: Towards a Unified View for Thinking with Images
Zirun Guo, Minjie Hong, Feng Zhang +2
Multimodal large language models (MLLMs) that think with images can interactively use tools to reason about visual inputs, but current approaches often rely on a narrow set of tool…
Hybrid LLM and Higher-Order Quantum Approximate Optimization for CSA Collateral Management
Tao Jin, Stuart Florescu, Heyu +1
We address finance-native collateral optimization under ISDA Credit Support Annexes (CSAs), where integer lots, Schedule A haircuts, RA/MTA gating, and issuer/currency/class caps c…
LLM-I: LLMs are Naturally Interleaved Multimodal Creators
Zirun Guo, Feng Zhang, Kai Jia +1
We propose LLM-Interleaved (LLM-I), a flexible and dynamic framework that reframes interleaved image-text generation as a tool-use problem. LLM-I is designed to overcome the "one-t…
Observe-R1: Unlocking Reasoning Abilities of MLLMs with Dynamic Progressive Reinforcement Learning
Zirun Guo, Minjie Hong, Tao Jin
Reinforcement Learning (RL) has shown promise in improving the reasoning abilities of Large Language Models (LLMs). However, the specific challenges of adapting RL to multimodal da…
Diff-Prompt: Diffusion-Driven Prompt Generator with Mask Supervision
Weicai Yan, Wang Lin, Zirun Guo +5
Prompt learning has demonstrated promising results in fine-tuning pre-trained multimodal models. However, the performance improvement is limited when applied to more complex and fi…
ConceptGuard: Continual Personalized Text-to-Image Generation with Forgetting and Confusion Mitigation
Zirun Guo, Tao Jin
Diffusion customization methods have achieved impressive results with only a minimal number of user-provided images. However, existing approaches customize concepts collectively, w…
Smoothing the Shift: Towards Stable Test-Time Adaptation under Complex Multimodal Noises
Zirun Guo, Tao Jin
Test-Time Adaptation (TTA) aims to tackle distribution shifts using unlabeled test data without access to the source data. In the context of multimodal data, there are more complex…
Efficient Prompting for Continual Adaptation to Missing Modalities
Zirun Guo, Shulei Wang, Wang Lin +3
Missing modality issues are common in real-world applications, arising from factors such as equipment failures and privacy concerns. When fine-tuning pre-trained models on downstre…
A Wander Through the Multimodal Landscape: Efficient Transfer Learning via Low-rank Sequence Multimodal Adapter
Zirun Guo, Xize Cheng, Yangyang Wu +1
Efficient transfer learning methods such as adapter-based methods have shown great success in unimodal models and vision-language models. However, existing methods have two main ch…
Bridging the Gap for Test-Time Multimodal Sentiment Analysis
Zirun Guo, Tao Jin, Wenlong Xu +2
Multimodal sentiment analysis (MSA) is an emerging research topic that aims to understand and recognize human sentiment or emotions through multiple modalities. However, in real-wo…