4 papers
What Does Layer-Importance Reveal About Transformers and State-Space Models?
Istabrak Abbes, Nizar Islah, Irina Rish +1
Transformers and state-space models (SSMs) are the two dominant families of sequence models, and a central open question is how far the analytical knowledge built for transformers…
Failed Reasoning Traces Tell You What Is Fixable (But Not by Reading Them)
Nizar Islah, Istabrak Abbes, Irina Rish +2
When post-trained language models fail on reasoning problems, the common test-time-scaling response is to spend more compute on additional attempts, and the failed traces play no f…
Revisiting Replay and Gradient Alignment for Continual Pre-Training of Large Language Models
Istabrak Abbes, Gopeshh Subbaraj, Matthew Riemer +6
Training large language models (LLMs) typically involves pre-training on massive corpora, only to restart the process entirely when new data becomes available. A more efficient and…
Small Encoders Can Rival Large Decoders in Detecting Groundedness
Istabrak Abbes, Gabriele Prato, Quentin Fournier +4
Augmenting large language models (LLMs) with external context significantly improves their performance in natural language processing (NLP) tasks. However, LLMs struggle to answer…