1 citations · 1 across the 5 of their papers we have counts for
6 papers
KV-Lingo: Learning KV-Cache Translators with Distillation
Valérie Castin, Keitaro Sakamoto, Anastasiia Filippova +3
Large language models represent context with a key-value (KV) cache. Caches are model-specific: for the same text, models with different architectures or weights produce incompatib…
Locking Pretrained Weights via Deep Low-Rank Residual Distillation
Keitaro Sakamoto, Pierre Ablin, Federico Danieli +1
The quality of open-weight language models has dramatically improved in recent years. Sharing weights greatly facilitates model adoption by enabling their use across diverse hardwa…
Explaining Grokking and Information Bottleneck through Neural Collapse Emergence
Keitaro Sakamoto, Issei Sato
The training dynamics of deep neural networks often defy expectations, even as these models form the foundation of modern machine learning. Two prominent examples are grokking, whe…
Benign Overfitting in Token Selection of Attention Mechanism
Keitaro Sakamoto, Issei Sato
Attention mechanism is a fundamental component of the transformer model and plays a significant role in its success. However, the theoretical understanding of how attention learns…
End-to-End Training Induces Information Bottleneck through Layer-Role Differentiation: A Comparative Analysis with Layer-wise Training
Keitaro Sakamoto, Issei Sato
End-to-end (E2E) training, optimizing the entire model through error backpropagation, fundamentally supports the advancements of deep learning. Despite its high performance, E2E tr…
Analyzing Lottery Ticket Hypothesis from PAC-Bayesian Theory Perspective
Keitaro Sakamoto, Issei Sato
The lottery ticket hypothesis (LTH) has attracted attention because it can explain why over-parameterized models often show high generalization ability. It is known that when we us…