3 papers
cs.CL2025
Deconstructing Attention: Investigating Design Principles for Effective Language Modeling
Huiyin Xue, Nafise Sadat Moosavi, Nikolaos Aletras
The success of Transformer language models is widely credited to their dot-product attention mechanism, which interweaves a set of key design principles: mixing information across…
cs.CL2025
DReSD: Dense Retrieval for Speculative Decoding
Milan Gritta, Huiyin Xue, Gerasimos Lampouras
Speculative decoding (SD) accelerates Large Language Model (LLM) generation by using an efficient draft model to propose the next few tokens, which are verified by the LLM in a sin…
cs.CL2022
HashFormers: Towards Vocabulary-independent Pre-trained Transformers
Huiyin Xue, Nikolaos Aletras
Transformer-based pre-trained language models are vocabulary-dependent, mapping by default each token to its corresponding embedding. This one-to-one mapping results into embedding…