1 citations · 2 across the 8 of their papers we have counts for
9 papers
Looped Transformers as Optimizers
Yulong Huang, Chen Jiang, Zhanpeng Zhou +5
Looped Transformers provide a parameter-efficient approach to depth scaling by repeatedly applying shared Transformer blocks. Recent reasoning models have likewise highlighted the…
UEPS: Robust and Efficient MRI Reconstruction
Xiang Zhou, Hong Shang, Zijian Zhan +3
Deep unrolled models (DUM) have become the state-of-the-art for accelerated MRI reconstruction, yet their robustness under domain shift remains a critical barrier to clinical adopt…
Towards Distributed Neural Architectures
Aditya Cowsik, Tianyu He, Andrey Gromov
We introduce and train distributed neural architectures (DNA) in vision and language domains. DNAs are initialized with a proto-architecture that consists of (transformer, MLP, att…
(How) Can Transformers Predict Pseudo-Random Numbers?
Tao Tao, Darshil Doshi, Dayal Singh Kalra +2
Transformers excel at discovering patterns in sequential data, yet their fundamental limitations and learning mechanisms remain crucial topics of investigation. In this paper, we s…
3D-LLaVA: Towards Generalist 3D LMMs with Omni Superpoint Transformer
Jiajun Deng, Tianyu He, Li Jiang +3
Current 3D Large Multimodal Models (3D LMMs) have shown tremendous potential in 3D-vision-based dialogue and reasoning. However, how to further enhance 3D LMMs to achieve fine-grai…
Grokking Modular Polynomials
Darshil Doshi, Tianyu He, Aritra Das +1
Neural networks readily learn a subset of the modular arithmetic tasks, while failing to generalize on the rest. This limitation remains unmoved by the choice of architecture and t…
Learning to grok: Emergence of in-context learning and skill composition in modular arithmetic tasks
Tianyu He, Darshil Doshi, Aritra Das +1
Large language models can solve tasks that were not present in the training set. This capability is believed to be due to in-context learning and skill composition. In this work, w…
Universal Sharpness Dynamics in Neural Network Training: Fixed Point Analysis, Edge of Stability, and Route to Chaos
Dayal Singh Kalra, Tianyu He, Maissam Barkeshli
In gradient descent dynamics of neural networks, the top eigenvalue of the loss Hessian (sharpness) displays a variety of robust phenomena throughout training. This includes early…
To grok or not to grok: Disentangling generalization and memorization on corrupted algorithmic datasets
Darshil Doshi, Aritra Das, Tianyu He +1
Robust generalization is a major challenge in deep learning, particularly when the number of trainable parameters is very large. In general, it is very difficult to know if the net…