4 papers
Why Post-Norm Transformers Collapse: Attention Amplification and Gradient Repair Failure
Xingjian Wang, Qingyu Han, Xiaodong Luo +1
Deep decoder-only Transformers often replace the original Post-Norm architecture with Pre-Norm variants because Post-Norm training is highly sensitive to warmup and learning rate u…
Revisiting Weight Regularization for Low-Rank Continual Learning
Yaoyue Zheng, Yin Zhang, Joost van de Weijer +4
Continual Learning (CL) with large-scale pre-trained models (PTMs) has recently gained wide attention, shifting the focus from training from scratch to continually adapting PTMs. T…
QuadEnhancer: Leveraging Quadratic Transformations to Enhance Deep Neural Networks
Qian Chen, Linxin Yang, Akang Wang +2
The combination of linear transformations and non-linear activation functions forms the foundation of most modern deep neural networks, enabling them to approximate highly complex…
Physical formula enhanced multi-task learning for pharmacokinetics prediction
Ruifeng Li, Dongzhan Zhou, Ancheng Shen +9
Artificial intelligence (AI) technology has demonstrated remarkable potential in drug dis-covery, where pharmacokinetics plays a crucial role in determining the dosage, safety, and…