A Self-Attentive Meta-Optimizer with Group-Adaptive Learning Rates and Weight Decay

JiangBo Zhao, ZhaoXin Liu

Adaptive optimizers like AdamW apply uniform hyperparameters across all parameter groups, ignoring heterogeneous optimization dynamics across layers and modules. We address this li…