4 papers
Complex KDA: Understanding and Enhancing the Expressivity of Kimi Delta Attention
Julien Siems, Riccardo Grazzi, Korbinian Pöppel +8
Linear RNNs based on the delta-rule enable efficient sequence modeling, but their linear updates with a low-rank correction constrain their expressivity. Prior work has shown that…
Explaining Grokking in Transformers through the Lens of Inductive Bias
Jaisidh Singh, Diganta Misra, Antonio Orvieto
We investigate grokking in transformers through the lens of inductive bias: dispositions arising from architecture or optimization that let the network prefer one solution over ano…
(Almost) Free Modality Stitching of Foundation Models
Jaisidh Singh, Diganta Misra, Boris Knyazev +1
Foundation multi-modal models are often designed by stitching of multiple existing pretrained uni-modal models: for example, an image classifier with an text model. This stitching…
Automatic Discovery and Assessment of Interpretable Systematic Errors in Semantic Segmentation
Jaisidh Singh, Sonam Singh, Amit Arvind Kale +1
This paper presents a novel method for discovering systematic errors in segmentation models. For instance, a systematic error in the segmentation model can be a sufficiently large…