7 papers
Nectar: Neural Estimation of Cached-Token Attention via Regression
João Monteiro, Michal Klein, Pierre Ablin +1
Evaluating softmax attention over a fixed long context requires reading every cached key-value pair for each new query token. For a given context (a book, a manual, a legal corpus)…
LaCy: What Small Language Models Can and Should Learn is Not Just a Question of Loss
Szilvia Ujváry, Louis Béthune, Pierre Ablin +3
Language models have consistently grown to compress more world knowledge into their parameters, but the knowledge that can be pretrained into them is upper-bounded by their paramet…
DynaMiCS: Fine-tuning LLMs with Performance Constraints using Dynamic Mixtures
Eleonora Gualdoni, Sonia Laguna, Louis Bethune +3
Multi-domain fine-tuning of large language models requires improving performance on target domains while preserving performance on constrained domains, such as general knowledge, i…
GraphOmni: A Comprehensive and Extensible Benchmark Framework for Large Language Models on Graph-theoretic Tasks
Hao Xu, Xiangru Jian, Xinjian Zhao +9
This paper introduces GraphOmni, a comprehensive benchmark designed to evaluate the reasoning capabilities of LLMs on graph-theoretic tasks articulated in natural language. GraphOm…
MoSE: Hierarchical Self-Distillation Enhances Early Layer Embeddings
Andrea Gurioli, Federico Pennino, João Monteiro +1
Deploying language models often requires navigating accuracy vs. performance trade-offs to meet latency constraints while preserving utility. Traditional model distillation reduces…
Explain-Query-Test: Self-Evaluating LLMs Via Explanation and Comprehension Discrepancy
Saeid Asgari Taghanaki, Joao Monteiro
Large language models (LLMs) have demonstrated remarkable proficiency in generating detailed and coherent explanations of complex concepts. However, the extent to which these model…