69 citations · 79 across the 11 of their papers we have counts for
3 papers · 1 filter
A Pre-Training Analogue of Grokking in Language Models: Tracing Delayed Grammatical Generalization
Sherin Muckatira, Namrata Shivagunde, Vijeta Deshpande +1
Grokking, the phenomenon in which neural networks generalize long after fitting their training data, has been studied in supervised settings on many epochs. LLM pre-training instea…
Activation Steering for Synthetic Data Generation: The Role of Diversity in Downstream Safety Detection
Vijeta Deshpande, Tootiya Giyahchi, Veena Padmanabhan +2
Safety detection models require examples of HHH (Helpful, Harmless, Honest)-violating outputs for robust generalization, however such examples are scarce. Activation Steering (AS)…
Beyond Perplexity: A Geometric and Spectral Study of Low-Rank Pre-Training
Namrata Shivagunde, Vijeta Deshpande, Sherin Muckatira +1
Pre-training large language models is dominated by the memory cost of storing full-rank weights, gradients, and optimizer states. Low-rank pre-training has emerged to address this,…