8 papers
The Quantization Benefits of Residual-Free Transformers
Yiping Ji, Mahalakshmi Sabanayagam, Peyman Moghadam +2
Large-scale transformer training and deployment are increasingly constrained by the transfer of activations, gradients, and optimizer states across accelerators. Low-bit quantizati…
From Failure to Feedback: Group Revision Unlocks Hard Cases in Object-Level Grounding
Yuyuan Liu, Yiping Ji, Anjie Le +6
Finetuning Large Vision-Language Models with reinforcement learning has emerged as a promising approach to enhance their capability in object-level grounding. However, existing met…
Rethinking Attention: Polynomial Alternatives to Softmax in Transformers
Hemanth Saratchandran, Jianqiao Zheng, Yiping Ji +2
This paper questions whether the strong performance of softmax attention in transformers stems from producing a probability distribution over inputs. Instead, we argue that softmax…
From Tables to Signals: Revealing Spectral Adaptivity in TabPFN
Jianqiao Zheng, Cameron Gordon, Yiping Ji +2
Task-agnostic tabular foundation models such as TabPFN have achieved impressive performance on tabular learning tasks, yet the origins of their inductive biases remain poorly under…
SineLoRA: Sine-Activated Delta Compression
Cameron Gordon, Yiping Ji, Hemanth Saratchandran +2
Resource-constrained weight deployment is a task of immense practical importance. Recently, there has been interest in the specific task of \textit{Delta Compression}, where partie…
Cutting the Skip: Training Residual-Free Transformers
Yiping Ji, James Martens, Jianqiao Zheng +5
Transformers have achieved remarkable success across a wide range of applications, a feat often attributed to their scalability. Yet training them without skip (residual) connectio…