6 papers
A Faster Generalized Two-Stage Approximate Top-K
Yashas Samaga, Varun Yerram, Spandana Raj Babbula +2
We consider the Top- selection problem, which aims to identify the largest elements in an array. Top- selection arises in many machine learning algorithms and often becom…
Period spacings and global seismic parameters for K2 red giants using deep learning
Nipun Ghanghas, Siddharth Dhanpal, Shravan Hanasoge +2
Gravity-mode period spacings (DPi_1) of red giants probe the stellar core directly, constraining its structure, mass and evolutionary state. Their measurement requires resolving na…
A model of errors in transformers
Suvrat Raju, Praneeth Netrapalli
We study the error rate of LLMs on tasks like arithmetic that require a deterministic output, and repetitive processing of tokens drawn from a small set of alternatives. We argue t…
Spark Transformer: Reactivating Sparsity in FFN and Attention
Chong You, Kan Wu, Zhipeng Jia +16
The discovery of the lazy neuron phenomenon in trained Transformers, where the vast majority of neurons in their feed-forward networks (FFN) are inactive for each token, has spurre…
Compressing Many-Shots in In-Context Learning
Devvrit Khatri, Pranamya Kulkarni, Nilesh Gupta +9
Large Language Models (LLMs) have been shown to be able to learn different tasks without explicit finetuning when given many input-output examples / demonstrations through In-Conte…
The Feature Speed Formula: a flexible approach to scale hyper-parameters of deep neural networks
Lénaïc Chizat, Praneeth Netrapalli
Deep learning succeeds by doing hierarchical feature learning, yet tuning hyper-parameters (HP) such as initialization scales, learning rates etc., only give indirect control over…