6 papers
AGC-Bench: Measuring Artificial General Creativity
Roger Beaty, Vijeta Deshpande, Clin K. Y. Lai +9
Creativity research has debated whether creativity is domain-specific (e.g., visual, writing, science), and if it is psychometrically separable from general intelligence. Both ques…
CrowdMath: A Dataset of Crowdsourced Mathematical Research Discussions
Sherin Muckatira, Jesse Geneson, Slava Gerovitch +3
Large language models have made substantial progress on mathematical reasoning, but existing benchmarks typically evaluate well-specified problems with final answers, step-by-step…
A Pre-Training Analogue of Grokking in Language Models: Tracing Delayed Grammatical Generalization
Sherin Muckatira, Namrata Shivagunde, Vijeta Deshpande +1
Grokking, the phenomenon in which neural networks generalize long after fitting their training data, has been studied in supervised settings on many epochs. LLM pre-training instea…
Playing with Words, Improving with Rewards: Training Language Models for Creative Association
Vijeta Deshpande, Namrata Shivagunde, Sherin Muckatira +5
Large Language Models (LLMs) are being applied to increasingly difficult problems and use cases. To navigate their vast solution spaces effectively, LLMs need to be creative. Yet t…
Beyond Perplexity: A Geometric and Spectral Study of Low-Rank Pre-Training
Namrata Shivagunde, Vijeta Deshpande, Sherin Muckatira +1
Pre-training large language models is dominated by the memory cost of storing full-rank weights, gradients, and optimizer states. Low-rank pre-training has emerged to address this,…
Deconstructing In-Context Learning: Understanding Prompts via Corruption
Namrata Shivagunde, Vladislav Lialin, Sherin Muckatira +1
The ability of large language models (LLMs) to learn in context based on the provided prompt has led to an explosive growth in their use, culminating in the proliferation of…