150 citations · 274 across the 69 of their papers we have counts for
7 papers · 1 filter
Nesting Forward Automatic Differentiation for Memory-Efficient Deep Neural Network Training
Cong Guo, Yuxian Qiu, Jingwen Leng +6
An activation function is an element-wise mathematical function and plays a crucial role in deep neural networks (DNN). Many novel and sophisticated activation functions have been…
ANT: Exploiting Adaptive Numerical Data Type for Low-bit Deep Neural Network Quantization
Cong Guo, Chen Zhang, Jingwen Leng +5
Quantization is a technique to reduce the computation and memory cost of DNN models, which are getting increasingly large. Existing quantization solutions use fixed-point integer o…
Efficient Adaptive Activation Rounding for Post-Training Quantization
Zhengyi Li, Cong Guo, Zhanda Zhu +5
Post-training quantization attracts increasing attention due to its convenience in deploying quantized neural networks. Although rounding-to-nearest remains the prevailing method f…
SALO: An Efficient Spatial Accelerator Enabling Hybrid Sparse Attention Mechanisms for Long Sequences
Guan Shen, Jieru Zhao, Quan Chen +3
The attention mechanisms of transformers effectively extract pertinent information from the input sequence. However, the quadratic complexity of self-attention w.r.t the sequence l…
Transkimmer: Transformer Learns to Layer-wise Skim
Yue Guan, Zhengyi Li, Jingwen Leng +2
Transformer architecture has become the de-facto model for many machine learning tasks from natural language processing and computer vision. As such, improving its computational ef…
SQuant: On-the-Fly Data-Free Quantization via Diagonal Hessian Approximation
Cong Guo, Yuxian Qiu, Jingwen Leng +6
Quantization of deep neural networks (DNN) has been proven effective for compressing and accelerating DNN models. Data-free quantization (DFQ) is a promising approach without the o…