3 papers
cs.LG2026
Revisiting the Capacity Gap in Chain-of-Thought Distillation from a Practical Perspective
Tokio Kajitsuka, Ukyo Honda, Sho Takase
Chain-of-thought (CoT) distillation transfers reasoning behaviors from a strong teacher to a smaller student, but prior work reports a capacity gap: distillation may fail when the…
cs.LG2024
On the Optimal Memorization Capacity of Transformers
Tokio Kajitsuka, Issei Sato
Recent research in the field of machine learning has increasingly focused on the memorization capacity of Transformers, but how efficient they are is not yet well understood. We de…
cs.LG2023
Are Transformers with One Layer Self-Attention Using Low-Rank Weight Matrices Universal Approximators?
Tokio Kajitsuka, Issei Sato
Existing analyses of the expressive capacity of Transformer models have required excessively deep layers for data memorization, leading to a discrepancy with the Transformers actua…