Evaluating Trajectory Features for Routing Final-Layer Attention

Yupeng Yao

Attention routing requires a signal that predicts the value of attention on the current prefix. We evaluate whether hidden-state extrapolation error, curvature and error change imp…