2 citations · 5 across the 8 of their papers we have counts for
8 papers · 1 filter
Why Post-Norm Transformers Collapse: Attention Amplification and Gradient Repair Failure
Xingjian Wang, Qingyu Han, Xiaodong Luo +1
Deep decoder-only Transformers often replace the original Post-Norm architecture with Pre-Norm variants because Post-Norm training is highly sensitive to warmup and learning rate u…
Revisiting Weight Regularization for Low-Rank Continual Learning
Yaoyue Zheng, Yin Zhang, Joost van de Weijer +4
Continual Learning (CL) with large-scale pre-trained models (PTMs) has recently gained wide attention, shifting the focus from training from scratch to continually adapting PTMs. T…
QuadEnhancer: Leveraging Quadratic Transformations to Enhance Deep Neural Networks
Qian Chen, Linxin Yang, Akang Wang +2
The combination of linear transformations and non-linear activation functions forms the foundation of most modern deep neural networks, enabling them to approximate highly complex…
Why "classic" Transformers are shallow and how to make them go deep
Yueyao Yu, Yin Zhang
Since its introduction in 2017, Transformer has emerged as the leading neural network architecture, catalyzing revolutionary advancements in many AI disciplines. The key innovation…
POViT: Vision Transformer for Multi-objective Design and Characterization of Nanophotonic Devices
Xinyu Chen, Renjie Li, Yueyao Yu +4
We solve a fundamental challenge in semiconductor IC design: the fast and accurate characterization of nanoscale photonic devices. Much like the fusion between AI and EDA, many eff…
A Lightweight and Gradient-Stable Neural Layer
Yueyao Yu, Yin Zhang
To enhance resource efficiency and model deployability of neural networks, we propose a neural-layer architecture based on Householder weighting and absolute-value activating, call…
Multi-layer Perceptron Trainability Explained via Variability
Yueyao Yu, Yin Zhang
Despite the tremendous successes of deep neural networks (DNNs) in various applications, many fundamental aspects of deep learning remain incompletely understood, including DNN tra…
Big-Data Clustering: K-Means or K-Indicators?
Feiyu Chen, Yuchen Yang, Liwei Xu +2
The K-means algorithm is arguably the most popular data clustering method, commonly applied to processed datasets in some "feature spaces", as is in spectral clustering. Highly sen…