8 papers
From Patching to Pruning Visual Computation in Vision Language Models
Rahul Chowdhury, Timothy A Rupprecht, Xuan Shen +3
Vision language models (VLMs) incur substantial inference cost because every visual token is processed by the attention and MLP projections of every decoder layer, even when token-…
SpectralCache: Accelerating Diffusion-Based World Models via Spectral Feature Caching
Zhendong Mi, Pu Zhao, Ziyu Hu +4
Diffusion-based world models enable high-quality interactive environment generation but suffer from substantial inference overhead due to repeated Transformer evaluations during de…
Imagine the Future, Internalize the Gist: Efficient VLA Reasoning via Internalized Spatiotemporal Imagination
Shenglan Li, Zhendong Mi, Hengyi Zhu +6
Vision-language-action (VLA) models increasingly incorporate intermediate reasoning to improve robotic manipulation, yet existing approaches primarily reason about observed states…
Roots Beneath the Cut: Uncovering the Risk of Concept Revival in Pruning-Based Unlearning for Diffusion Models
Ci Zhang, Zhaojun Ding, Chence Yang +7
Pruning-based unlearning has recently emerged as a fast, training-free, and data-independent approach to remove undesired concepts from diffusion models. It promises high efficienc…
Towards Fast LLM Fine-tuning through Zeroth-Order Optimization with Projected Gradient-Aligned Perturbations
Zhendong Mi, Qitao Tan, Grace Li Zhang +3
Fine-tuning large language models (LLMs) using zeroth-order (ZO) optimization has emerged as a promising alternative to traditional gradient-based methods due to its reduced memory…
End-to-End On-Device Quantization-Aware Training for LLMs at Inference Cost
Qitao Tan, Xiaoying Song, Jin Lu +9
Quantization is an effective technique to reduce the deployment cost of large language models (LLMs), and post-training quantization (PTQ) has been widely studied due to its effici…
ACE: Exploring Activation Cosine Similarity and Variance for Accurate and Calibration-Efficient LLM Pruning
Zhendong Mi, Zhenglun Kong, Geng Yuan +1
With the rapid expansion of large language models (LLMs), the demand for memory and computational resources has grown significantly. Recent advances in LLM pruning aim to reduce th…
KerZOO: Kernel Function Informed Zeroth-Order Optimization for Accurate and Accelerated LLM Fine-Tuning
Zhendong Mi, Qitao Tan, Xiaodong Yu +3
Large language models (LLMs) have demonstrated impressive capabilities across numerous NLP tasks. Nevertheless, conventional first-order fine-tuning techniques impose heavy memory…