45 citations · 53 across the 16 of their papers we have counts for
13 papers · 1 filter
Fixed State, Long Reach: What a Constant-Size Cache Buys Block Diffusion at Scale
Vaibhav Singh, Pierre-André Noël, Torsten Scholak +2
Diffusion language models decode tokens in parallel, but their bidirectional denoiser rules out the naive key--value (KV) cache behind fast autoregressive inference. Block diffusio…
Heterogeneous Low-Bandwidth Pre-Training of LLMs
Yazan Obeidi, Amir Sarfi, Joel Lidin +2
Pre-training large language models (LLMs) increasingly requires distributed compute, yet bandwidth constraints make it difficult to scale beyond well-provisioned datacenters-especi…
When Data Falls Short: Grokking Below the Critical Threshold
Vaibhav Singh, Eugene Belilovsky, Rahaf Aljundi
In this paper, we investigate the phenomenon of grokking, where models exhibit delayed generalization following overfitting on training data. We focus on data-scarce regimes where…
DiffuMamba: High-Throughput Diffusion LMs with Mamba Backbone
Vaibhav Singh, Oleksiy Ostapenko, Pierre-André Noël +2
Diffusion language models (DLMs) have emerged as a promising alternative to autoregressive (AR) generation, yet their reliance on Transformer backbones limits inference efficiency…
Incentivizing Permissionless Distributed Learning of LLMs
Joel Lidin, Amir Sarfi, Evangelos Pappas +3
We describe an incentive system for distributed deep learning of foundational models where peers are rewarded for contributions. The incentive system, \textit{Gauntlet}, has been d…
Beyond Cosine Decay: On the effectiveness of Infinite Learning Rate Schedule for Continual Pre-training
Vaibhav Singh, Paul Janson, Paria Mehrbod +4
The ever-growing availability of unlabeled data presents both opportunities and challenges for training artificial intelligence systems. While self-supervised learning (SSL) has em…
Dual-Phase Continual Learning: Supervised Adaptation Meets Unsupervised Retention
Vaibhav Singh, Rahaf Aljundi, Eugene Belilovsky
Foundational Vision-Language Models (VLMs) excel across diverse tasks, but adapting them to new domains without forgetting prior knowledge remains a critical challenge. Continual L…
Simulated Annealing in Early Layers Leads to Better Generalization
Amirmohammad Sarfi, Zahra Karimpour, Muawiz Chaudhary +4
Recently, a number of iterative learning methods have been introduced to improve generalization. These typically rely on training for longer periods of time in exchange for improve…
Prototype-Sample Relation Distillation: Towards Replay-Free Continual Learning
Nader Asadi, MohammadReza Davari, Sudhir Mudur +2
In Continual learning (CL) balancing effective adaptation while combating catastrophic forgetting is a central challenge. Many of the recent best-performing methods utilize various…
New Insights on Reducing Abrupt Representation Change in Online Continual Learning
Lucas Caccia, Rahaf Aljundi, Nader Asadi +3
In the online continual learning paradigm, agents must learn from a changing distribution while respecting memory and compute constraints. Experience Replay (ER), where a small sub…
Probing Representation Forgetting in Supervised and Unsupervised Continual Learning
MohammadReza Davari, Nader Asadi, Sudhir Mudur +2
Continual Learning research typically focuses on tackling the phenomenon of catastrophic forgetting in neural networks. Catastrophic forgetting is associated with an abrupt loss of…
Tackling Online One-Class Incremental Learning by Removing Negative Contrasts
Nader Asadi, Sudhir Mudur, Eugene Belilovsky
Recent work studies the supervised online continual learning setting where a learner receives a stream of data whose class distribution changes over time. Distinct from other conti…
New Insights on Reducing Abrupt Representation Change in Online Continual Learning
Lucas Caccia, Rahaf Aljundi, Nader Asadi +3
In the online continual learning paradigm, agents must learn from a changing distribution while respecting memory and compute constraints. Experience Replay (ER), where a small sub…