26 citations · 50 across the 9 of their papers we have counts for
9 papers
Efficient Pre-Training with Token Superposition
Bowen Peng, Théo Gigant, Jeffrey Quesnelle
Pre-training of Large Language Models is often prohibitively expensive and inefficient at scale, requiring complex and invasive modifications in order to achieve high data throughp…
Decoupling the Benefits of Subword Tokenization for Language Model Training via Byte-level Simulation
Théo Gigant, Bowen Peng, Jeffrey Quesnelle
Subword tokenization is an essential part of modern large language models (LLMs), yet its specific contributions to training efficiency and model performance remain poorly understo…
Long Context Pre-Training with Lighthouse Attention
Bowen Peng, Subho Ghosh, Jeffrey Quesnelle
Training causal transformers at extreme sequence lengths is bottlenecked by the quadratic time and memory of scaled dot-product attention (SDPA). In this work, we propose Lighthous…
CurvaDion: Curvature-Adaptive Distributed Orthonormalization
Bhavesh Kumar, Roger Jin, Jeffrey Quesnelle
As language models scale to trillions of parameters, distributed training across many GPUs becomes essential, yet gradient synchronization over high-bandwidth, low-latency networks…
Hermes 4 Technical Report
Ryan Teknium, Roger Jin, Jai Suphavadeeprasit +6
We present Hermes 4, a family of hybrid reasoning models that combine structured, multi-turn reasoning with broad instruction-following ability. We describe the challenges encounte…
DeMo: Decoupled Momentum Optimization
Bowen Peng, Lizhang Chen, Baiyu Su +3
Scaling neural network training increasingly depends on synchronous data-parallelism, yet full-precision gradient all-reduce imposes a severe communication bottleneck. We propose D…
Hermes 3 Technical Report
Ryan Teknium, Jeffrey Quesnelle, Chen Guang
Instruct (or "chat") tuned models have become the primary way in which most people interact with large language models. As opposed to "base" or "foundation" models, instruct-tuned…
YaRN: Efficient Context Window Extension of Large Language Models
Bowen Peng, Jeffrey Quesnelle, Honglu Fan +1
Rotary Position Embeddings (RoPE) have been shown to effectively encode positional information in transformer-based language models. However, these models fail to generalize past t…
On the linkability of Zcash transactions
Jeffrey Quesnelle
Zcash is a fork of Bitcoin with optional anonymity features. While transparent transactions are fully linkable, shielded transactions use zero-knowledge proofs to obscure the parti…