5 papers
Selective Rotary Position Embedding
Sajad Movahedi, Timur Carstensen, Arshia Afzal +3
Position information is essential for language modeling. In softmax transformers, Rotary Position Embeddings (\textit{RoPE}) encode positions through \textit{fixed-angle} rotations…
Fixed-Point Reasoners: Stable and Adaptive Deep Looped Transformers
Sajad Movahedi, Vera MilovanoviÄ, Shlomo Libo Feigin +5
Looped architectures provide an inductive bias toward learning step-by-step procedures for tasks that require compositional reasoning. The number of effective layers reached by loo…
Fixed-Point RNNs: Interpolating from Diagonal to Dense
Sajad Movahedi, Felix Sarnthein, Nicola Muca Cirone +1
Linear recurrent neural networks (RNNs) and state-space models (SSMs) such as Mamba have become promising alternatives to softmax-attention as sequence mixing layers in Transformer…
Geometric Inductive Biases of Deep Networks: The Role of Data and Architecture
Sajad Movahedi, Antonio Orvieto, Seyed-Mohsen Moosavi-Dezfooli
In this paper, we propose the , which argues that the input space curvature of a neural network remains invariant under transformati…
QEQR: An Exploration of Query Expansion Methods for Question Retrieval in CQA Services
Yasin Ghafourian, Sajad Movahedi, Azadeh Shakery
CQA services are valuable sources of knowledge that can be used to find answers to users' information needs. In these services, question retrieval aims to help users with their inf…