From the 1 of 11 linked papers with an AI index.
11 papers
ClockRoPE: Random Fourier Rotations for Temporal Routine Modeling
Yiwen Chen, Joshua Ainslie, Krzysztof Choromanski +4
The paper introduces ClockRoPE, a method that uses random Fourier rotations to create periodic position embeddings for transformers, improving temporal routine modeling in sequenti…
Near-Linear Time Generalized Sinkhorn Algorithms for Bounded Genus Graphs
Krzysztof Choromanski, Derek Long, Ananya Parashar +1
We present GenusSink, a new class of approximate generalized Sinkhorn algorithms with shortest-path-distance costs for bounded genus (e.g. planar) graphs, providing near-linear tim…
Computationally-efficient Graph Modeling with Refined Graph Random Features
Krzysztof Choromanski, Avinava Dubey, Arijit Sehanobish +1
We propose refined GRFs (GRFs++), a new class of Graph Random Features (GRFs) for efficient and accurate computations involving kernels defined on the nodes of a graph. GRFs++ reso…
Rotary Position Encodings for Graphs
Isaac Reid, Arijit Sehanobish, Cederik Höfs +7
We study the extent to which rotary position encodings (RoPE), a recent transformer position encoding algorithm broadly adopted in large language models (LLMs) and vision transform…
Manifold Random Features
Ananya Parashar, Derek Long, Dwaipayan Saha +1
We present a new paradigm for creating random features to approximate bi-variate functions (in particular, kernels) defined on general manifolds. This new mechanism of Manifold Ran…
RelFlexformer: Efficient Attention 3D-Transformers for Integrable Relative Positional Encodings
Byeongchan Kim, Arijit Sehanobish, Avinava Dubey +2
We present a new class of efficient attention mechanisms applying universal 3D Relative Positional Encoding (RPE) methods given by arbitrary integrable modulation functions . Th…