2 citations · 5 across the 14 of their papers we have counts for
4 papers · 1 filter
Proteus: Incremental Memory Activation for Long-Context Sequence Modeling
Reza Bayat, Ali Behrouz, Vahab Mirrokni +1
The quadratic cost of attention-based sequence models for long contexts has motivated a growing line of research on memory-based models that can compress context into a compact sta…
On-Policy Self-Distillation with Sampled Demonstrations Reduces Output Diversity
Andrei Liviu Nicolicioiu, Mohammad Pezeshki, Aaron Courville
On-policy self-distillation achieves strong pass@1 accuracy by using a single model as both teacher and student, with the teacher conditioned on a correct demonstration to provide…
Tapered Language Models
Reza Bayat, Ali Behrouz, Aaron Courville
Modern language models, including transformer, recurrent, and memory-based variants, share a common chassis: a stack of identical layers in which parameters are allocated uniformly…
Data Selection Through Iterative Self-Filtering for Vision-Language Settings
Andrei Liviu Nicolicioiu, Sarvjeet Singh Ghotra, Morgane M. Moss +1
The availability of large amounts of clean data is paramount to training neural networks. However, at large scales, manual oversight is impractical, resulting in sizeable datasets…