4.4k citations · 5.3k across the 63 of their papers we have counts for
11 papers · 1 filter
PocketLLM: Ultimate Compression of Large Language Models via Meta Networks
Ye Tian, Chengcheng Wang, Jing Han +2
As Large Language Models (LLMs) continue to grow in size, storing and transmitting them on edge devices becomes increasingly challenging. Traditional methods like quantization and…
LLM Data Selection and Utilization via Dynamic Bi-level Optimization
Yang Yu, Kai Han, Hang Zhou +4
While large-scale training data is fundamental for developing capable large language models (LLMs), strategically selecting high-quality data has emerged as a critical approach to…
SlimLLM: Accurate Structured Pruning for Large Language Models
Jialong Guo, Xinghao Chen, Yehui Tang +1
Large language models(LLMs) have garnered significant attention and demonstrated impressive capabilities in a wide range of applications. However, due to their enormous computation…
Mixture of Lookup Experts
Shibo Jie, Yehui Tang, Kai Han +4
Mixture-of-Experts (MoE) activates only a subset of experts during inference, allowing the model to maintain low inference FLOPs and latency even as the parameter count scales up.…
ExCP: Extreme LLM Checkpoint Compression via Weight-Momentum Joint Shrinking
Wenshuo Li, Xinghao Chen, Han Shu +2
Large language models (LLM) have recently attracted significant attention in the field of artificial intelligence. However, the training process of these models poses significant c…
A Survey on Transformer Compression
Yehui Tang, Yunhe Wang, Jianyuan Guo +4
Transformer plays a vital role in the realms of natural language processing (NLP) and computer vision (CV), specially for constructing large language models (LLM) and large vision…
CBQ: Cross-Block Quantization for Large Language Models
Xin Ding, Xiaoyu Liu, Zhijun Tu +8
Post-training quantization (PTQ) has played a key role in compressing large language models (LLMs) with ultra-low costs. However, existing PTQ methods only focus on handling the ou…
Homogeneous Architecture Augmentation for Neural Predictor
Yuqiao Liu, Yehui Tang, Yanan Sun
Neural Architecture Search (NAS) can automatically design well-performed architectures of Deep Neural Networks (DNNs) for the tasks at hand. However, one bottleneck of NAS is the p…
Beyond Dropout: Feature Map Distortion to Regularize Deep Neural Networks
Yehui Tang, Yunhe Wang, Yixing Xu +4
Deep neural networks often consist of a great number of trainable parameters for extracting powerful features from given datasets. On one hand, massive trainable parameters signifi…
ReNAS:Relativistic Evaluation of Neural Architecture Search
Yixing Xu, Yunhe Wang, Kai Han +4
An effective and efficient architecture performance evaluation scheme is essential for the success of Neural Architecture Search (NAS). To save computational cost, most of existing…
Bringing Giant Neural Networks Down to Earth with Unlabeled Data
Yehui Tang, Shan You, Chang Xu +2
Compressing giant neural networks has gained much attention for their extensive applications on edge devices such as cellphones. During the compressing process, one of the most imp…