4 citations · 8 across the 9 of their papers we have counts for
10 papers
The Devil Is in the Reconstruction Loss Scale: Rethinking Optimization in LLM Quantization
Chao Li, Shigeng Wang, Anbang Yao
Post-training quantization (PTQ) methods typically use sequential quantization that partitions a pre-trained LLM into a series of units (e.g., transformer blocks), with one unit qu…
Attend to Your Own Thoughts: Breaking the Barrier for Post-Training Quantization of Reasoning LLMs through the Lens of 1.58-Bit Quantization
Shigeng Wang, Chao Li, Yangyuxuan Kang +2
We propose ScaleQ-1.58, a scalable ternary post-training quantization (PTQ) framework for reasoning LLMs. Its core insight stems from an empirical finding: although modern LLMs are…
CAT-Q: Cost-efficient and Accurate Ternary Quantization for LLMs
Shigeng Wang, Chao Li, Yangyuxuan Kang +2
In this paper, we present CAT-Q, Cost-efficient and Accurate Ternary Quantization, for compressing and accelerating LLMs. Unlike existing state-of-the-art ternary quantization meth…
Chain-of-Models Pre-Training: Rethinking Training Acceleration of Vision Foundation Models
Jiawei Fan, Shigeng Wang, Chao Li +2
In this paper, we present Chain-of-Models Pre-Training (CoM-PT), a novel performance-lossless training acceleration method for vision foundation models (VFMs). This approach fundam…
SliderQuant: Accurate Post-Training Quantization for LLMs
Shigeng Wang, Chao Li, Yangyuxuan Kang +3
In this paper, we address post-training quantization (PTQ) for large language models (LLMs) from an overlooked perspective: given a pre-trained high-precision LLM, the predominant…
ScaleKD: Strong Vision Transformers Could Be Excellent Teachers
Jiawei Fan, Chao Li, Xiaolong Liu +1
In this paper, we question if well pre-trained vision transformer (ViT) models could be used as teachers that exhibit scalable properties to advance cross architecture knowledge di…
JoyType: A Robust Design for Multilingual Visual Text Creation
Chao Li, Chen Jiang, Xiaolong Liu +2
Generating images with accurately represented text, especially in non-Latin languages, poses a significant challenge for diffusion models. Existing approaches, such as the integrat…
KernelWarehouse: Rethinking the Design of Dynamic Convolution
Chao Li, Anbang Yao
Dynamic convolution learns a linear mixture of n static kernels weighted with their input-dependent attentions, demonstrating superior performance than normal convolution. However,…
NOAH: Learning Pairwise Object Category Attentions for Image Classification
Chao Li, Aojun Zhou, Anbang Yao
A modern deep neural network (DNN) for image classification tasks typically consists of two parts: a backbone for feature extraction, and a head for feature encoding and class pred…
Augmentation-Free Dense Contrastive Knowledge Distillation for Efficient Semantic Segmentation
Jiawei Fan, Chao Li, Xiaolong Liu +2
In recent years, knowledge distillation methods based on contrastive learning have achieved promising results on image classification and object detection tasks. However, in this l…