26 citations · 50 across the 9 of their papers we have counts for
3 papers · 1 filter
Efficient Pre-Training with Token Superposition
Bowen Peng, Théo Gigant, Jeffrey Quesnelle
Pre-training of Large Language Models is often prohibitively expensive and inefficient at scale, requiring complex and invasive modifications in order to achieve high data throughp…
Decoupling the Benefits of Subword Tokenization for Language Model Training via Byte-level Simulation
Théo Gigant, Bowen Peng, Jeffrey Quesnelle
Subword tokenization is an essential part of modern large language models (LLMs), yet its specific contributions to training efficiency and model performance remain poorly understo…
Long Context Pre-Training with Lighthouse Attention
Bowen Peng, Subho Ghosh, Jeffrey Quesnelle
Training causal transformers at extreme sequence lengths is bottlenecked by the quadratic time and memory of scaled dot-product attention (SDPA). In this work, we propose Lighthous…