5 citations · 5 across the 10 of their papers we have counts for
3 papers · 2 filters
The Effect of Layer Ordering in Hybrid Architectures
John Cooper, Mingchen Ma, Jiaqi Mo +4
Hybrid attention is rapidly becoming the dominant architectural choice for frontier models. Despite their successes, such architectures complicate model design by introducing addit…
Robust Regression of General ReLUs with Queries
Ilias Diakonikolas, Daniel M. Kane, Mingchen Ma
We study the task of agnostically learning general (as opposed to homogeneous) ReLUs under the Gaussian distribution with respect to the squared loss. In the passive learning setti…
Expressivity-Efficiency Tradeoffs for Hybrid Sequence Models
John Cooper, Ilias Diakonikolas, Mingchen Ma +1
Hybrid sequence models--combining Transformer and state-space model layers--seek to gain the expressive versatility of attention as well as the computational efficiency of state-sp…