Improving Parameter Utilization by Sharing Neural Experts Across Layers in Transformers

Dian Jiao, Jiaxin Duan, Shuai Zhao +3

Transformer-based large language models often suffer from inter-layer parameter redundancy, where functional transformations are redundantly learned across network depths. We propo…