45 citations · 75 across the 10 of their papers we have counts for
6 papers · 1 filter
JTok: On Token Embedding as another Axis of Scaling Law via Joint Token Self-modulation
Yebin Yang, Huaijin Wu, Fu Guo +5
LLMs have traditionally scaled along dense dimensions, where performance is coupled with near-linear increases in computational cost. While MoE decouples capacity from compute, it…
AdamHD: Decoupled Huber Decay Regularization for Language Model Pre-Training
Fu-Ming Guo, Yingfang Fan
Adaptive optimizers with decoupled weight decay, such as AdamW, are the de facto standard for pre-training large transformer-based generative models. Yet the quadratic nature of th…
SparseOptimizer: Sparsify Language Models through Moreau-Yosida Regularization and Accelerate via Compiler Co-design
Fu-Ming Guo
This paper introduces SparseOptimizer, a novel deep learning optimizer that exploits Moreau-Yosida regularization to naturally induce sparsity in large language models such as BERT…
Algorithm to Compilation Co-design: An Integrated View of Neural Network Sparsity
Fu-Ming Guo, Austin Huang
Reducing computation cost, inference latency, and memory footprint of neural networks are frequently cited as research motivations for pruning and sparsity. However, operationalizi…
Reweighted Proximal Pruning for Large-Scale Language Representation
Fu-Ming Guo, Sijia Liu, Finlay S. Mungall +2
Recently, pre-trained language representation flourishes as the mainstay of the natural language understanding community, e.g., BERT. These pre-trained language representations can…
PCONV: The Missing but Desirable Sparsity in DNN Weight Pruning for Real-time Execution on Mobile Devices
Xiaolong Ma, Fu-Ming Guo, Wei Niu +5
Model compression techniques on Deep Neural Network (DNN) have been widely acknowledged as an effective way to achieve acceleration on a variety of platforms, and DNN weight prunin…