45 citations · 75 across the 10 of their papers we have counts for
10 papers
Smooth Scaling Laws Hide Stepwise Token Learning
Pingjie Wang, Zechen Hu, Peiru Yang +2
Language model loss follows remarkably regular scaling laws over model and data size, yet it remains unclear why the aggregate loss should exhibit a power-law form. Existing explan…
JTok: On Token Embedding as another Axis of Scaling Law via Joint Token Self-modulation
Yebin Yang, Huaijin Wu, Fu Guo +5
LLMs have traditionally scaled along dense dimensions, where performance is coupled with near-linear increases in computational cost. While MoE decouples capacity from compute, it…
AdamHD: Decoupled Huber Decay Regularization for Language Model Pre-Training
Fu-Ming Guo, Yingfang Fan
Adaptive optimizers with decoupled weight decay, such as AdamW, are the de facto standard for pre-training large transformer-based generative models. Yet the quadratic nature of th…
dots.llm1 Technical Report
Bi Huo, Bin Tu, Cheng Qin +24
Mixture of Experts (MoE) models have emerged as a promising paradigm for scaling language models efficiently by activating only a subset of parameters for each input token. In this…
SparseOptimizer: Sparsify Language Models through Moreau-Yosida Regularization and Accelerate via Compiler Co-design
Fu-Ming Guo
This paper introduces SparseOptimizer, a novel deep learning optimizer that exploits Moreau-Yosida regularization to naturally induce sparsity in large language models such as BERT…
Sim2Real Docs: Domain Randomization for Documents in Natural Scenes using Ray-traced Rendering
Nikhil Maddikunta, Huijun Zhao, Sumit Keswani +5
In the past, computer vision systems for digitized documents could rely on systematically captured, high-quality scans. Today, transactions involving digital documents are more lik…
Algorithm to Compilation Co-design: An Integrated View of Neural Network Sparsity
Fu-Ming Guo, Austin Huang
Reducing computation cost, inference latency, and memory footprint of neural networks are frequently cited as research motivations for pruning and sparsity. However, operationalizi…
An Image Enhancing Pattern-based Sparsity for Real-time Inference on Mobile Devices
Xiaolong Ma, Wei Niu, Tianyun Zhang +8
Weight pruning has been widely acknowledged as a straightforward and effective method to eliminate redundancy in Deep Neural Networks (DNN), thereby achieving acceleration on vario…
Reweighted Proximal Pruning for Large-Scale Language Representation
Fu-Ming Guo, Sijia Liu, Finlay S. Mungall +2
Recently, pre-trained language representation flourishes as the mainstay of the natural language understanding community, e.g., BERT. These pre-trained language representations can…
PCONV: The Missing but Desirable Sparsity in DNN Weight Pruning for Real-time Execution on Mobile Devices
Xiaolong Ma, Fu-Ming Guo, Wei Niu +5
Model compression techniques on Deep Neural Network (DNN) have been widely acknowledged as an effective way to achieve acceleration on a variety of platforms, and DNN weight prunin…