5 papers
SO(3)-RoPE for Spherical Transformers
Christian Libner, Chase van de Geijn, Alexander S. Ecker +1
Spherical data arise in many scientific applications. Often spherical transformers disregard the geometry of the underlying spherical domain, causing distortions and coordinate sin…
TrAction: Action Recognition with Sparse Trajectories
Jan F. Meier, Felix B. Mueller, Alexander Ecker +1
Modern action recognition models operate on memory- and compute-intensive dense RGB video volumes and frequently exploit appearance and background shortcuts, for example, predictin…
Do traveling waves make good positional encodings?
Chase van de Geijn, Ayush Paliwal, Timo Lüddecke +1
Transformers rely on positional encoding to compensate for the inherent permutation invariance of self-attention. Traditional approaches use absolute sinusoidal embeddings or learn…
Clifford Algebraic Rotor Embeddings : Maybe embeddings should start to CARE
Sameeksha Sriram, Ayush Paliwal, Alexander S. Ecker +1
Rotary Positional Embeddings (RoPE) have demonstrated exceptional performance as a positional encoding method, consistently outperforming their baselines. While recent work has sou…
A Circular Argument : Does RoPE need to be Equivariant for Vision?
Chase van de Geijn, Timo Lüddecke, Polina Turishcheva +1
Rotary Positional Encodings (RoPE) have emerged as a highly effective technique for one-dimensional sequences in Natural Language Processing spurring recent progress towards genera…