3 citations · 3 across the 4 of their papers we have counts for
4 papers
Blurry Window Attention
Axel Laborieux, Christos Sourmpis, Juan Gabriel Kostelec +1
The Softmax Attention operation in Transformer language models has a quadratic complexity in the sequence length and a growing state size in the form of KV cache, which becomes a b…
When Perplexity Lies: Generation-Focused Distillation of Hybrid Sequence Models
Juan Gabriel Kostelec, Qinghai Guo
Converting a pretrained Transformer into a more efficient hybrid model through distillation offers a promising approach to reducing inference costs. However, achieving high-quality…
AllMem: A Memory-centric Recipe for Efficient Long-context Modeling
Ziming Wang, Xiang Wang, Kailong Peng +5
Large Language Models (LLMs) encounter significant performance bottlenecks in long-sequence tasks due to the computational complexity and memory overhead inherent in the self-atten…
Trial matching: capturing variability with data-constrained spiking neural networks
Christos Sourmpis, Carl Petersen, Wulfram Gerstner +1
Simultaneous behavioral and electrophysiological recordings call for new methods to reveal the interactions between neural activity and behavior. A milestone would be an interpreta…