424 citations · 505 across the 29 of their papers we have counts for
5 papers · 1 filter
Crystal: Illuminating LLM Abilities on Language and Code
Tianhua Tao, Junbo Li, Bowen Tan +8
Large Language Models (LLMs) specializing in code generation (which are also often referred to as code LLMs), e.g., StarCoder and Code Llama, play increasingly critical roles in va…
Normalization Layer Per-Example Gradients are Sufficient to Predict Gradient Noise Scale in Transformers
Gavia Gray, Aman Tiwari, Shane Bergsma +1
Per-example gradient norms are a vital ingredient for estimating gradient noise scale (GNS) with minimal variance. Observing the tensor contractions required to compute them, we pr…
Bilingual Adaptation of Monolingual Foundation Models
Gurpreet Gosal, Yishi Xu, Gokul Ramakrishnan +19
We present an efficient method for adapting a monolingual Large Language Model (LLM) to another language, addressing challenges of catastrophic forgetting and tokenizer limitations…
Sparse maximal update parameterization: A holistic approach to sparse training dynamics
Nolan Dey, Shane Bergsma, Joel Hestness
Several challenges make it difficult for sparse neural networks to compete with dense models. First, setting a large fraction of weights to zero impairs forward and gradient signal…
MediSwift: Efficient Sparse Pre-trained Biomedical Language Models
Vithursan Thangarasa, Mahmoud Salem, Shreyas Saxena +3
Large language models (LLMs) are typically trained on general source data for various domains, but a recent surge in domain-specific LLMs has shown their potential to outperform ge…