5 papers
Annotation Entropy Predicts Per-Example Learning Dynamics in LoRA Fine-Tuning
Brady Steele
We find that LoRA fine-tuning exhibits un-learning on contested examples: items with high annotator disagreement show increasing loss during training, a qualitatively distinct patt…
Subspace Geometry Governs Catastrophic Forgetting in Low-Rank Adaptation
Brady Steele
Low-Rank Adaptation (LoRA) has emerged as a parameter-efficient approach for adapting large pre-trained models, yet its behavior under continual learning remains poorly understood.…
Why LoRA Resists Label Noise: A Theoretical Framework for Noise-Robust Parameter-Efficient Fine-Tuning
Brady Steele
Parameter-efficient fine-tuning methods like Low-Rank Adaptation (LoRA) have become the dominant paradigm for adapting large pretrained models. We present a theoretical framework e…
On the Limits of Learned Importance Scoring for KV Cache Compression
Brady Steele
We investigate learned KV cache compression through Speculative Importance Prediction (SIP), a 1.7M parameter non-query-aware scorer that predicts token importance from KV represen…
Scaling Trends for Multi-Hop Contextual Reasoning in Mid-Scale Language Models
Brady Steele, Micah Katz
We present a controlled study of multi-hop contextual reasoning in large language models, providing a clean demonstration of the task-method dissociation: rule-based pattern matchi…