Simon Lucey

Should We Skip Diffusion?

Yiping Ji, James Martens, Simon Lucey

Diffusion models learn semantic representations while generating images. In the Decoupled Diffusion Transformer (DDT), a condition encoder provides features that guide a velocity d…

Optimizer Geometry Sets the Pace: Spectral Learning Dynamics in Matrix Factorization

Mahalakshmi Sabanayagam, Simon Lucey

Recent successes of matrix- and curvature-based optimizers have renewed interest in how update geometry shapes learning. These methods normalize or precondition updates, changing h…

Conditioned Initialization for Attention

Hemanth Saratchandran, Simon Lucey

Transformers are a dominant architecture in modern machine learning, powering applications across vision, language, and beyond. At the core of their success lies the attention laye…

cs.AI2026

Making Transformers Logical

Mingze Ma, Hemanth Saratchandran, Cameron Gordon +1

Transformers do not merely lack data on some Boolean extrapolation tasks; they generalize in a systematically wrong way. Recent work on generalization on the unseen has shown that,…

Memory Efficient Tabular Foundation Models

Shuting Luo, Monika Mikhail Kanaan, Cameron Gordon +2

Tabular Foundation Models, such as TabPFN, have received a large amount of recent attention due to their performance on in-context tabular machine learning tasks, which often excee…

Can Transformers Really Do It All? On the Compatibility of Inductive Biases Across Tasks

Damien Teney, Liangze Jiang, Hemanth Saratchandran +1

Transformers are remarkably versatile and their design is largely consistent across a variety of applications. But are they optimal for any given task or dataset? The answer may be…

cs.CL2026

Parameter-Efficient Fine-Tuning with Learnable Rank

Arpit Garg, Simon Lucey, Hemanth Saratchandran

Low-Rank Adaptation (LoRA) is a popular parameter-efficient fine-tuning (PEFT) method that restricts weight updates to low-rank adapters, introducing a fixed low-rank inductive bia…

cs.CL2026

Mask the Target: A Plug-and-Play Regularizer Against LoRA Forgetting

Runze Xu, Arpit Garg, Hemanth Saratchandran +1

Low-Rank Adaptation (LoRA) has become one of the most widely used fine-tuning mechanisms for adapting large language models to new domains, tasks, and users. Yet adaptation perform…

The Quantization Benefits of Residual-Free Transformers

Yiping Ji, Mahalakshmi Sabanayagam, Peyman Moghadam +2

Large-scale transformer training and deployment are increasingly constrained by the transfer of activations, gradients, and optimizer states across accelerators. Low-bit quantizati…

Spectral Conditioning of Attention Improves Transformer Performance

Hemanth Saratchandran, Simon Lucey

We present a theoretical analysis of the Jacobian of an attention block within a transformer, showing that it is governed by the query, key, and value projections that define the a…

The Inlet Rank Collapse in Implicit Neural Representations: Diagnosis and Unified Remedy

Jianqiao Zheng, Hemanth Saratchandran, Simon Lucey

Implicit Neural Representations (INRs) have revolutionized continuous signal modeling, yet they struggle to recover fine-grained details within finite training budgets. While empir…

Robust Physical Adversarial Patches Using Dynamically Optimized Clusters

Harrison Bagley, Will Meakin, Simon Lucey +2

Physical adversarial attacks on deep learning systems is concerning due to the ease of deploying such attacks, usually by placing an adversarial patch in a scene to manipulate the…

From Tables to Signals: Revealing Spectral Adaptivity in TabPFN

Jianqiao Zheng, Cameron Gordon, Yiping Ji +2

Task-agnostic tabular foundation models such as TabPFN have achieved impressive performance on tabular learning tasks, yet the origins of their inductive biases remain poorly under…

SineProject: Machine Unlearning for Stable Vision Language Alignment

Arpit Garg, Hemanth Saratchandran, Simon Lucey

Multimodal Large Language Models (MLLMs) increasingly need to forget specific knowledge such as unsafe or private information without requiring full retraining. However, existing u…

Cutting the Skip: Training Residual-Free Transformers

Yiping Ji, James Martens, Jianqiao Zheng +5

Transformers have achieved remarkable success across a wide range of applications, a feat often attributed to their scalability. Yet training them without skip (residual) connectio…

Stable Forgetting: Bounded Parameter-Efficient Unlearning in Foundation Models

Arpit Garg, Hemanth Saratchandran, Ravi Garg +1

Machine unlearning in foundation models (e.g., language and vision transformers) is essential for privacy and safety; however, existing approaches are unstable and unreliable. A wi…

Leaner Transformers: More Heads, Less Depth

Hemanth Saratchandran, Damien Teney, Simon Lucey

Transformers have reshaped machine learning by utilizing attention mechanisms to capture complex patterns in large datasets, leading to significant improvements in performance. Thi…

SineLoRA $Δ$ : Sine-Activated Delta Compression

Cameron Gordon, Yiping Ji, Hemanth Saratchandran +2

Resource-constrained weight deployment is a task of immense practical importance. Recently, there has been interest in the specific task of \textit{Delta Compression}, where partie…

Structured Initialization for Vision Transformers

Jianqiao Zheng, Xueqian Li, Hemanth Saratchandran +1

Convolutional Neural Networks (CNNs) inherently encode strong inductive biases, enabling effective generalization on small-scale datasets. In this paper, we propose integrating thi…

Enhancing Transformers Through Conditioned Embedded Tokens

Hemanth Saratchandran, Simon Lucey

Transformers have transformed modern machine learning, driving breakthroughs in computer vision, natural language processing, and robotics. At the core of their success lies the at…