1 citations · 1 across the 18 of their papers we have counts for
11 papers · 1 filter
Should We Skip Diffusion?
Yiping Ji, James Martens, Simon Lucey
Diffusion models learn semantic representations while generating images. In the Decoupled Diffusion Transformer (DDT), a condition encoder provides features that guide a velocity d…
Optimizer Geometry Sets the Pace: Spectral Learning Dynamics in Matrix Factorization
Mahalakshmi Sabanayagam, Simon Lucey
Recent successes of matrix- and curvature-based optimizers have renewed interest in how update geometry shapes learning. These methods normalize or precondition updates, changing h…
Conditioned Initialization for Attention
Hemanth Saratchandran, Simon Lucey
Transformers are a dominant architecture in modern machine learning, powering applications across vision, language, and beyond. At the core of their success lies the attention laye…
Making Transformers Logical
Mingze Ma, Hemanth Saratchandran, Cameron Gordon +1
Transformers do not merely lack data on some Boolean extrapolation tasks; they generalize in a systematically wrong way. Recent work on generalization on the unseen has shown that,…
Memory Efficient Tabular Foundation Models
Shuting Luo, Monika Mikhail Kanaan, Cameron Gordon +2
Tabular Foundation Models, such as TabPFN, have received a large amount of recent attention due to their performance on in-context tabular machine learning tasks, which often excee…
Can Transformers Really Do It All? On the Compatibility of Inductive Biases Across Tasks
Damien Teney, Liangze Jiang, Hemanth Saratchandran +1
Transformers are remarkably versatile and their design is largely consistent across a variety of applications. But are they optimal for any given task or dataset? The answer may be…
Parameter-Efficient Fine-Tuning with Learnable Rank
Arpit Garg, Simon Lucey, Hemanth Saratchandran
Low-Rank Adaptation (LoRA) is a popular parameter-efficient fine-tuning (PEFT) method that restricts weight updates to low-rank adapters, introducing a fixed low-rank inductive bia…
Mask the Target: A Plug-and-Play Regularizer Against LoRA Forgetting
Runze Xu, Arpit Garg, Hemanth Saratchandran +1
Low-Rank Adaptation (LoRA) has become one of the most widely used fine-tuning mechanisms for adapting large language models to new domains, tasks, and users. Yet adaptation perform…
The Quantization Benefits of Residual-Free Transformers
Yiping Ji, Mahalakshmi Sabanayagam, Peyman Moghadam +2
Large-scale transformer training and deployment are increasingly constrained by the transfer of activations, gradients, and optimizer states across accelerators. Low-bit quantizati…
Spectral Conditioning of Attention Improves Transformer Performance
Hemanth Saratchandran, Simon Lucey
We present a theoretical analysis of the Jacobian of an attention block within a transformer, showing that it is governed by the query, key, and value projections that define the a…
The Inlet Rank Collapse in Implicit Neural Representations: Diagnosis and Unified Remedy
Jianqiao Zheng, Hemanth Saratchandran, Simon Lucey
Implicit Neural Representations (INRs) have revolutionized continuous signal modeling, yet they struggle to recover fine-grained details within finite training budgets. While empir…