2 papers
cs.LG2026
A Defense of the Quadratic Model
Alexandru Meterez, Pranav Ajit Nair, Depen Morwani +3
Due to the complexity of neural network loss landscapes, optimization theory is forced to rely on idealized models, and there is generally a tradeoff between how theoretically trac…
cs.LG2026
The Recurrent Transformer: Greater Effective Depth and Efficient Decoding
Costin-Andrei Oncescu, Depen Morwani, Samy Jelassi +3
Transformers process tokens in parallel but are temporally shallow: at position , each layer attends to key-value pairs computed based on the previous layer, yielding a depth ca…