15 papers
Cross-Modal Latent Influence: Patch-Level Training Sample Attribution for Vision-Language Models
Laura Yao, Shixing Yu, Jie Peng +3
Explaining a vision--language model (VLM) prediction requires tracing training-example influence to the multimodal evidence that mediates it, yet existing attribution methods large…
Training and Scaling Compute-Optimal Physiological Waveform Foundation Models
Pingzhi Li, Jie Peng, Shuqing Luo +2
We investigate the scaling laws and compute-optimal training of physiological waveform foundation models (FMs). We train Aether, a family of over one hundred FMs ranging from 20M t…
Graph-of-Agents: A Graph-based Framework for Multi-Agent LLM Collaboration
Sukwon Yun, Jie Peng, Pingzhi Li +5
With an ever-growing zoo of LLMs and benchmarks, the need to orchestrate multiple models for improved task performance has never been more pressing. While frameworks like Mixture-o…
Mozart: Modularized and Efficient MoE Training on 3.5D Wafer-Scale Chiplet Architectures
Shuqing Luo, Ye Han, Pingzhi Li +7
Mixture-of-Experts (MoE) architecture offers enhanced efficiency for Large Language Models (LLMs) with modularized computation, yet its inherent sparsity poses significant hardware…
Vulnerability-Aware Robust Multimodal Adversarial Training
Junrui Zhang, Xinyu Zhao, Jie Peng +3
Multimodal learning has shown significant superiority on various tasks by integrating multiple modalities. However, the interdependencies among modalities increase the susceptibili…
RTGS: Real-Time 3D Gaussian Splatting SLAM via Multi-Level Redundancy Reduction
Leshu Li, Jiayin Qin, Jie Peng +8
3D Gaussian Splatting (3DGS) based Simultaneous Localization and Mapping (SLAM) systems can largely benefit from 3DGS's state-of-the-art rendering efficiency and accuracy, but have…
Unlearning Sensitive Information in Multimodal LLMs: Benchmark and Attack-Defense Evaluation
Vaidehi Patil, Yi-Lin Sung, Peter Hase +3
LLMs trained on massive datasets may inadvertently acquire sensitive information such as personal details and potentially harmful content. This risk is further heightened in multim…
Advancing MoE Efficiency: A Collaboration-Constrained Routing (C2R) Strategy for Better Expert Parallelism Design
Mohan Zhang, Pingzhi Li, Jie Peng +2
Mixture-of-Experts (MoE) has successfully scaled up models while maintaining nearly constant computing costs. By employing a gating network to route input tokens, it selectively ac…
LightDefense: A Lightweight Uncertainty-Driven Defense against Jailbreaks via Shifted Token Distribution
Zhuoran Yang, Yanyong Zhang
Large Language Models (LLMs) face threats from jailbreak prompts. Existing methods for defending against jailbreak attacks are primarily based on auxiliary models. These strategies…
Symbiotic Cooperation for Web Agents: Harnessing Complementary Strengths of Large and Small LLMs
Ruichen Zhang, Mufan Qiu, Zhen Tan +7
Web browsing agents powered by large language models (LLMs) have shown tremendous potential in automating complex web-based tasks. Existing approaches typically rely on large LLMs…
GRNFormer: A Biologically-Guided Framework for Integrating Gene Regulatory Networks into RNA Foundation Models
Mufan Qiu, Xinyu Hu, Fengwei Zhan +6
Foundation models for single-cell RNA sequencing (scRNA-seq) have shown promising capabilities in capturing gene expression patterns. However, current approaches face critical limi…
Continually Evolved Multimodal Foundation Models for Cancer Prognosis
Jie Peng, Shuang Zhou, Longwei Yang +7
Cancer prognosis is a critical task that involves predicting patient outcomes and survival rates. To enhance prediction accuracy, previous studies have integrated diverse data moda…
Dialogue is Better Than Monologue: Instructing Medical LLMs via Strategical Conversations
Zijie Liu, Xinyu Zhao, Jie Peng +5
Current medical AI systems often fail to replicate real-world clinical reasoning, as they are predominantly trained and evaluated on static text and question-answer tasks. These tu…
Hexa-MoE: Efficient and Heterogeneous-aware Training for Mixture-of-Experts
Shuqing Luo, Jie Peng, Pingzhi Li +2
Mixture-of-Experts (MoE) has emerged as a practical approach to scale up parameters for the Transformer model to achieve better generalization while maintaining a sub-linear increa…
Harnessing Your DRAM and SSD for Sustainable and Accessible LLM Inference with Mixed-Precision and Multi-level Caching
Jie Peng, Zhang Cao, Huaizhi Qu +5
Although Large Language Models (LLMs) have demonstrated remarkable capabilities, their massive parameter counts and associated extensive computing make LLMs' deployment the main pa…