Showing cs.LGShow all
2 papers · 1 filter
cs.LG2026
Ablate and Rescue: A Causal Analysis of Residual Stream Hyper-Connections
William Peng, Josheev Rai, Kevin Tseng +2
Multi-stream transformer architectures have recently been proposed as a promising direction for managing representation collapse and the vanishing gradient problem for residual con…
cs.LG2024
Interpolated-MLPs: Controllable Inductive Bias
Sean Wu, Jordan Hong, Keyu Bai +1
Due to their weak inductive bias, Multi-Layer Perceptrons (MLPs) have subpar performance at low-compute levels compared to standard architectures such as convolution-based networks…