9 papers
Cartridges++: KV Cache Compression without Off-Context Derailment
Sonia Laguna, Joao Monteiro, Marco Cuturi +2
Serving long documents to a Large Language Model (LLM) repeatedly is expensive: computations grow with context length, and the memory footprint of the key-value (KV) cache balloons…
KV-Lingo: Learning KV-Cache Translators with Distillation
Valérie Castin, Keitaro Sakamoto, Anastasiia Filippova +3
Large language models represent context with a key-value (KV) cache. Caches are model-specific: for the same text, models with different architectures or weights produce incompatib…
DynaMiCS: Fine-tuning LLMs with Performance Constraints using Dynamic Mixtures
Eleonora Gualdoni, Sonia Laguna, Louis Bethune +3
Multi-domain fine-tuning of large language models requires improving performance on target domains while preserving previous capabilities, such as general knowledge, instruction fo…
Nectar: Neural Estimation of Cached-Token Attention via Regression
João Monteiro, Michal Klein, Pierre Ablin +1
Evaluating softmax attention over a fixed long context requires reading every cached key-value pair for each new query token. For a given context (a book, a manual, a legal corpus)…
LaCy: What Small Language Models Can and Should Learn is Not Just a Question of Loss
Szilvia Ujváry, Louis Béthune, Pierre Ablin +3
Language models have consistently grown to compress more world knowledge into their parameters, but the knowledge that can be pretrained into them is upper-bounded by their paramet…
GraphOmni: A Comprehensive and Extensible Benchmark Framework for Large Language Models on Graph-theoretic Tasks
Hao Xu, Xiangru Jian, Xinjian Zhao +9
This paper introduces GraphOmni, a comprehensive benchmark designed to evaluate the reasoning capabilities of LLMs on graph-theoretic tasks articulated in natural language. GraphOm…
MoSE: Hierarchical Self-Distillation Enhances Early Layer Embeddings
Andrea Gurioli, Federico Pennino, João Monteiro +1
Deploying language models often requires navigating accuracy vs. performance trade-offs to meet latency constraints while preserving utility. Traditional model distillation reduces…
Explain-Query-Test: Self-Evaluating LLMs Via Explanation and Comprehension Discrepancy
Saeid Asgari Taghanaki, Joao Monteiro
Large language models (LLMs) have demonstrated remarkable proficiency in generating detailed and coherent explanations of complex concepts. However, the extent to which these model…
XC-Cache: Cross-Attending to Cached Context for Efficient LLM Inference
João Monteiro, Étienne Marcotte, Pierre-André Noël +5
In-context learning (ICL) approaches typically leverage prompting to condition decoder-only language model generation on reference information. Just-in-time processing of a context…