2 citations · 4 across the 6 of their papers we have counts for
4 papers · 1 filter
Fine-Tuning Low-Bit Models with Gradient in Quantized Code Space
Shiguang Wu, Zhouchen Lin, Quanming Yao
Fine-tuning Low-bit models aims to adapt a quantized model while keeping the final deployed checkpoint in the same low-bit form. This setting is practically important as it reduces…
Attention Sinks Induce Gradient Sinks: Massive Activations as Gradient Regulators in Transformers
Yihong Chen, Zhouchen Lin, Quanming Yao
Attention sinks and massive activations are recurring and closely related phenomena in Transformer models. Existing explanations have largely focused on the forward pass, yet in pr…
Learning to Learn with Contrastive Meta-Objective
Shiguang Wu, Yaqing Wang, Yatao Bian +1
Meta-learning enables learning systems to adapt quickly to new tasks, similar to humans. Different meta-learning approaches all work under/with the mini-batch episodic training fra…
Graph Unitary Message Passing
Haiquan Qiu, Quanming Yao
Unitarity is a useful principle for stabilizing deep neural networks, but in graph neural networks (GNNs) instability is induced not only by learnable parameters but also by the gr…