2 papers
cs.LG2025
Neural Attention: A Novel Mechanism for Enhanced Expressive Power in Transformer Models
Andrew DiGiugno, Ausif Mahmood
Transformer models typically calculate attention matrices using dot products, which have limitations when capturing nonlinear relationships between embedding vectors. We propose Ne…
cs.LG2025
CacheFormer: High Attention-Based Segment Caching
Sushant Singh, Ausif Mahmood
Efficiently handling long contexts in transformer-based language models with low perplexity is an active area of research. Numerous recent approaches like Linformer, Longformer, Pe…