297 citations · 1.6k across the 36 of their papers we have counts for
36 papers
The Bias is in the Details: An Assessment of Cognitive Bias in LLMs
R. Alexander Knipper, Charles S. Knipper, Kaiqi Zhang +3
As Large Language Models (LLMs) are increasingly embedded in real-world decision-making processes, it becomes crucial to examine the extent to which they exhibit cognitive biases.…
Fast and Simplex: 2-Simplicial Attention in Triton
Aurko Roy, Timothy Chou, Sai Surya Duvvuri +5
Recent work has shown that training loss scales as a power law with both model size and the number of tokens, and that achieving compute-optimal models requires scaling model size…
Gemma 3 Technical Report
Gemma Team, Aishwarya Kamath, Johan Ferret +209
We introduce Gemma 3, a multimodal addition to the Gemma family of lightweight open models, ranging in scale from 1 to 27 billion parameters. This version introduces vision underst…
Seed-Free Synthetic Data Generation Framework for Instruction-Tuning LLMs: A Case Study in Thai
Parinthapat Pengpun, Can Udomcharoenchaikit, Weerayut Buaphet +1
We present a synthetic data approach for instruction-tuning large language models (LLMs) for low-resource languages in a data-efficient manner, specifically focusing on Thai. We id…
Mitigating Extrinsic Gender Bias for Bangla Classification Tasks
Sajib Kumar Saha Joy, Arman Hassan Mahy, Meherin Sultana +4
In this study, we investigate extrinsic gender bias in Bangla pretrained language models, a largely underexplored area in low-resource languages. To assess this bias, we construct…
A Little Help Goes a Long Way: Efficient LLM Training by Leveraging Small LMs
Ankit Singh Rawat, Veeranjaneyulu Sadhanala, Afshin Rostamizadeh +12
A primary challenge in large language model (LLM) development is their onerous pre-training cost. Typically, such pre-training involves optimizing a self-supervised objective (such…
Michelangelo: Long Context Evaluations Beyond Haystacks via Latent Structure Queries
Kiran Vodrahalli, Santiago Ontanon, Nilesh Tripuraneni +21
We introduce Michelangelo: a minimal, synthetic, and unleaked long-context reasoning evaluation for large language models which is also easy to automatically score. This evaluation…
Gemma: Open Models Based on Gemini Research and Technology
Gemma Team, Thomas Mesnard, Cassidy Hardin +105
This work introduces Gemma, a family of lightweight, state-of-the art open models built from the research and technology used to create Gemini models. Gemma models demonstrate stro…
Learning from straggler clients in federated learning
Andrew Hard, Antonious M. Girgis, Ehsan Amid +4
How well do existing federated learning algorithms learn from client devices that return model updates with a significant time delay? Is it even possible to learn effectively from…
Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context
Gemini Team, Petko Georgiev, Ving Ian Lei +1132
In this report, we introduce the Gemini 1.5 family of models, representing the next generation of highly compute-efficient multimodal models capable of recalling and reasoning over…
Assessing biomedical knowledge robustness in large language models by query-efficient sampling attacks
R. Patrick Xian, Alex J. Lee, Satvik Lolla +4
The increasing depth of parametric domain knowledge in large language models (LLMs) is fueling their rapid deployment in real-world applications. Understanding model vulnerabilitie…
A Computationally Efficient Sparsified Online Newton Method
Fnu Devvrit, Sai Surya Duvvuri, Rohan Anil +3
Second-order methods hold significant promise for enhancing the convergence of deep neural network training; however, their large memory and computational demands have limited thei…
Heterogeneous Federated Learning Using Knowledge Codistillation
Jared Lichtarge, Ehsan Amid, Shankar Kumar +3
Federated Averaging, and many federated learning algorithm variants which build upon it, have a limitation: all clients must share the same model architecture. This results in unus…
Benchmarking Neural Network Training Algorithms
George E. Dahl, Frank Schneider, Zachary Nado +22
Training algorithms, broadly construed, are an essential part of every deep learning pipeline. Training algorithm improvements that speed up training across a wide variety of workl…
PaLM 2 Technical Report
Rohan Anil, Andrew M. Dai, Orhan Firat +125
We introduce PaLM 2, a new state-of-the-art language model that has better multilingual and reasoning capabilities and is more compute-efficient than its predecessor PaLM. PaLM 2 i…
Sketchy: Memory-efficient Adaptive Regularization with Frequent Directions
Vladimir Feinberg, Xinyi Chen, Y. Jennifer Sun +2
Adaptive regularization methods that exploit more than the diagonal entries exhibit state of the art performance for many tasks, but can be prohibitive in terms of memory and runni…
Layerwise Bregman Representation Learning with Applications to Knowledge Distillation
Ehsan Amid, Rohan Anil, Christopher Fifty +1
In this work, we propose a novel approach for layerwise representation learning of a trained neural network. In particular, we form a Bregman divergence based on the layer's transf…
On the Factory Floor: ML Engineering for Industrial-Scale Ads Recommendation Models
Rohan Anil, Sandra Gadanho, Da Huang +9
For industrial-scale advertising systems, prediction of ad click-through rate (CTR) is a central problem. Ad clicks constitute a significant class of user engagements and are often…
N-Grammer: Augmenting Transformers with latent n-grams
Aurko Roy, Rohan Anil, Guangda Lai +13
Transformer models have recently emerged as one of the foundational models in natural language processing, and as a byproduct, there is significant recent interest and investment i…
Learning from Randomly Initialized Neural Network Features
Ehsan Amid, Rohan Anil, Wojciech Kotłowski +1
We present the surprising result that randomly initialized neural networks are good feature extractors in expectation. These random features correspond to finite-sample realization…
Step-size Adaptation Using Exponentiated Gradient Updates
Ehsan Amid, Rohan Anil, Christopher Fifty +1
Optimizers like Adam and AdaGrad have been very successful in training large-scale neural networks. Yet, the performance of these methods is heavily dependent on a carefully tuned…
Efficiently Identifying Task Groupings for Multi-Task Learning
Christopher Fifty, Ehsan Amid, Zhe Zhao +3
Multi-task learning can leverage information learned by one task to benefit the training of other tasks. Despite this capacity, naively training all tasks together in one model oft…
Large-Scale Differentially Private BERT
Rohan Anil, Badih Ghazi, Vineet Gupta +2
In this work, we study the large-scale pretraining of BERT-Large with differentially private SGD (DP-SGD). We show that combined with a careful implementation, scaling up the batch…
LocoProp: Enhancing BackProp via Local Loss Optimization
Ehsan Amid, Rohan Anil, Manfred K. Warmuth
Second-order methods have shown state-of-the-art performance for optimizing deep neural networks. Nonetheless, their large memory requirement and high computational complexity, com…