69 citations · 79 across the 11 of their papers we have counts for
12 papers
AGC-Bench: Measuring Artificial General Creativity
Roger Beaty, Vijeta Deshpande, Clin K. Y. Lai +9
Creativity research has debated whether creativity is domain-specific (e.g., visual, writing, science), and if it is psychometrically separable from general intelligence. Both ques…
A Pre-Training Analogue of Grokking in Language Models: Tracing Delayed Grammatical Generalization
Sherin Muckatira, Namrata Shivagunde, Vijeta Deshpande +1
Grokking, the phenomenon in which neural networks generalize long after fitting their training data, has been studied in supervised settings on many epochs. LLM pre-training instea…
Activation Steering for Synthetic Data Generation: The Role of Diversity in Downstream Safety Detection
Vijeta Deshpande, Tootiya Giyahchi, Veena Padmanabhan +2
Safety detection models require examples of HHH (Helpful, Harmless, Honest)-violating outputs for robust generalization, however such examples are scarce. Activation Steering (AS)…
Playing with Words, Improving with Rewards: Training Language Models for Creative Association
Vijeta Deshpande, Namrata Shivagunde, Sherin Muckatira +5
Large Language Models (LLMs) are being applied to increasingly difficult problems and use cases. To navigate their vast solution spaces effectively, LLMs need to be creative. Yet t…
Beyond Perplexity: A Geometric and Spectral Study of Low-Rank Pre-Training
Namrata Shivagunde, Vijeta Deshpande, Sherin Muckatira +1
Pre-training large language models is dominated by the memory cost of storing full-rank weights, gradients, and optimizer states. Low-rank pre-training has emerged to address this,…
A Penalty Goes a Long Way: Measuring Lexical Diversity in Synthetic Texts Under Prompt-Influenced Length Variations
Vijeta Deshpande, Ishita Dasgupta, Uttaran Bhattacharya +3
Synthetic text generated by Large Language Models (LLMs) is increasingly used for further training and improvement of LLMs. Diversity is crucial for the effectiveness of synthetic…
Diverse, not Short: A Length-Controlled Data Selection Strategy for Improving Response Diversity of Language Models
Vijeta Deshpande, Debasmita Ghose, John D. Patterson +2
Diverse language model responses are crucial for creative generation, open-ended tasks, and self-improvement training. We show that common diversity metrics, and even reward models…
Emergent Abilities in Reduced-Scale Generative Language Models
Sherin Muckatira, Vijeta Deshpande, Vladislav Lialin +1
Large language models can solve new tasks without task-specific fine-tuning. This ability, also known as in-context learning (ICL), is considered an emergent ability and is primari…
LocalTweets to LocalHealth: A Mental Health Surveillance Framework Based on Twitter Data
Vijeta Deshpande, Minhwa Lee, Zonghai Yao +3
Prior research on Twitter (now X) data has provided positive evidence of its utility in developing supplementary health surveillance systems. In this study, we present a new framew…
Honey, I Shrunk the Language: Language Model Behavior at Reduced Scale
Vijeta Deshpande, Dan Pechi, Shree Thatte +2
In recent years, language models have drastically grown in size, and the abilities of these models have been shown to improve with scale. The majority of recent scaling laws studie…
Scaling Down to Scale Up: A Guide to Parameter-Efficient Fine-Tuning
Vladislav Lialin, Vijeta Deshpande, Xiaowei Yao +1
This paper presents a systematic overview of parameter-efficient fine-tuning methods, covering over 50 papers published between early 2019 and mid-2024. These methods aim to addres…
Extracting Biomedical Factual Knowledge Using Pretrained Language Model and Electronic Health Record Context
Zonghai Yao, Yi Cao, Zhichao Yang +2
Language Models (LMs) have performed well on biomedical natural language processing applications. In this study, we conducted some experiments to use prompt methods to extract know…