1.5k citations · 5.2k across the 127 of their papers we have counts for
6 papers · 1 filter
AdaLomo: Low-memory Optimization with Adaptive Learning Rate
Kai Lv, Hang Yan, Qipeng Guo +2
Large language models have achieved remarkable success, but their extensive parameter size necessitates substantial memory for training, thereby setting a high threshold. While the…
From Hypergraph Energy Functions to Hypergraph Neural Networks
Yuxin Wang, Quan Gan, Xipeng Qiu +2
Hypergraphs are a powerful abstraction for representing higher-order interactions between entities of interest. To exploit these relationships in making downstream predictions, a v…
What Dense Graph Do You Need for Self-Attention?
Yuxin Wang, Chu-Tak Lee, Qipeng Guo +4
Transformers have made progress in miscellaneous tasks, but suffer from quadratic computational and memory complexities. Recent works propose sparse Transformers with attention on…
Learning to Teach with Student Feedback
Yitao Liu, Tianxiang Sun, Xipeng Qiu +1
Knowledge distillation (KD) has gained much attention due to its effectiveness in compressing large-scale pre-trained models. In typical KD methods, the small student model is trai…
A Survey of Transformers
Tianyang Lin, Yuxin Wang, Xiangyang Liu +1
Transformers have achieved great success in many artificial intelligence fields, such as natural language processing, computer vision, and audio processing. Therefore, it is natura…
Fork or Fail: Cycle-Consistent Training with Many-to-One Mappings
Qipeng Guo, Zhijing Jin, Ziyu Wang +5
Cycle-consistent training is widely used for jointly learning a forward and inverse mapping between two domains of interest without the cumbersome requirement of collecting matched…