120 citations · 192 across the 23 of their papers we have counts for
24 papers
Scalable Neural Network Kernels
Arijit Sehanobish, Krzysztof Choromanski, Yunfan Zhao +2
We introduce the concept of scalable neural network kernels (SNNKs), the replacements of regular feedforward layers (FFLs), capable of approximating the latter, but with favorable…
Learning a Fourier Transform for Linear Relative Positional Encodings in Transformers
Krzysztof Marcin Choromanski, Shanda Li, Valerii Likhosherstov +7
We propose a new class of linear Transformers called FourierLearner-Transformers (FLTs), which incorporate a wide range of relative positional encoding mechanisms (RPEs). These inc…
Efficient Graph Field Integrators Meet Point Clouds
Krzysztof Choromanski, Arijit Sehanobish, Han Lin +13
We present two new classes of algorithms for efficient field integration on graphs encoding point clouds. The first class, SeparatorFactorization(SF), leverages the bounded genus o…
FAVOR#: Sharp Attention Kernel Approximations via New Classes of Positive Random Features
Valerii Likhosherstov, Krzysztof Choromanski, Avinava Dubey +3
The problem of efficient approximation of a linear operator induced by the Gaussian or softmax kernel is often addressed using random features (RFs) which yield an unbiased approxi…
Simplex Random Features
Isaac Reid, Krzysztof Choromanski, Valerii Likhosherstov +1
We present Simplex Random Features (SimRFs), a new random feature (RF) mechanism for unbiased approximation of the softmax and Gaussian kernels by geometrical correlation of random…
Adaptive Computation with Elastic Input Sequence
Fuzhao Xue, Valerii Likhosherstov, Anurag Arnab +3
Humans have the ability to adapt the type of information they use, the procedure they employ, and the amount of time they spend when solving problems. However, most standard neural…
Karyotype AI for Precision Oncology
Zahra Shamsi, Isaac Reid, Drew Bryant +13
We present a machine learning method capable of accurately detecting chromosome abnormalities that cause blood cancers directly from microscope images of the metaphase stage of cel…
Chefs' Random Tables: Non-Trigonometric Random Features
Valerii Likhosherstov, Krzysztof Choromanski, Avinava Dubey +3
We introduce chefs' random tables (CRTs), a new class of non-trigonometric random features (RFs) to approximate Gaussian and softmax kernels. CRTs are an alternative to standard ra…
PolyViT: Co-training Vision Transformers on Images, Videos and Audio
Valerii Likhosherstov, Anurag Arnab, Krzysztof Choromanski +4
Can we train a single transformer model capable of processing multiple modalities and datasets, whilst sharing almost all of its learnable parameters? We present PolyViT, a model t…
Hybrid Random Features
Krzysztof Choromanski, Haoxian Chen, Han Lin +10
We propose a new class of random feature methods for linearizing softmax and Gaussian kernels called hybrid random features (HRFs) that automatically adapt the quality of kernel es…
From block-Toeplitz matrices to differential equations on graphs: towards a general theory for scalable masked Transformers
Krzysztof Choromanski, Han Lin, Haoxian Chen +7
In this paper we provide, to the best of our knowledge, the first comprehensive approach for incorporating various masking mechanisms into Transformers architectures in a scalable…
On the Expressive Power of Self-Attention Matrices
Valerii Likhosherstov, Krzysztof Choromanski, Adrian Weller
Transformer networks are able to capture patterns in data coming from many domains (text, images, videos, proteins, etc.) with little or no change to architecture components. We pe…
Debiasing a First-order Heuristic for Approximate Bi-level Optimization
Valerii Likhosherstov, Xingyou Song, Krzysztof Choromanski +2
Approximate bi-level optimization (ABLO) consists of (outer-level) optimization problems, involving numerical (inner-level) optimization loops. While ABLO has many applications acr…
Unlocking Pixels for Reinforcement Learning via Implicit Attention
Krzysztof Marcin Choromanski, Deepali Jain, Wenhao Yu +9
There has recently been significant interest in training reinforcement learning (RL) agents in vision-based environments. This poses many challenges, such as high dimensionality an…
Sub-Linear Memory: How to Make Performers SLiM
Valerii Likhosherstov, Krzysztof Choromanski, Jared Davis +2
The Transformer architecture has revolutionized deep learning on sequential data, becoming ubiquitous in state-of-the-art solutions for a wide variety of applications. Yet vanilla…
Rethinking Attention with Performers
Krzysztof Choromanski, Valerii Likhosherstov, David Dohan +10
We introduce Performers, Transformer architectures which can estimate regular (softmax) full-rank-attention Transformers with provable accuracy, but using only linear (as opposed t…
An Ode to an ODE
Krzysztof Choromanski, Jared Quincy Davis, Valerii Likhosherstov +6
We present a new paradigm for Neural ODE algorithms, called ODEtoODE, where time-dependent parameters of the main flow evolve according to a matrix flow on the orthogonal group O(d…
UFO-BLO: Unbiased First-Order Bilevel Optimization
Valerii Likhosherstov, Xingyou Song, Krzysztof Choromanski +2
Bilevel optimization (BLO) is a popular approach with many applications including hyperparameter optimization, neural architecture search, adversarial robustness and model-agnostic…
Masked Language Modeling for Proteins via Linearly Scalable Long-Context Transformers
Krzysztof Choromanski, Valerii Likhosherstov, David Dohan +8
Transformer models have achieved state-of-the-art results across a diverse range of domains. However, concern over the cost of training the attention mechanism to learn complex dep…
CWY Parametrization: a Solution for Parallelized Optimization of Orthogonal and Stiefel Matrices
Valerii Likhosherstov, Jared Davis, Krzysztof Choromanski +1
We introduce an efficient approach for optimization over orthogonal groups on highly parallel computation units such as GPUs or TPUs. As in earlier work, we parametrize an orthogon…
Stochastic Flows and Geometric Optimization on the Orthogonal Group
Krzysztof Choromanski, David Cheikhi, Jared Davis +12
We present a new class of stochastic, geometrically-driven optimization algorithms on the orthogonal group and naturally reductive homogeneous manifolds obtained from the ac…
Tractable Minor-free Generalization of Planar Zero-field Ising Models
Valerii Likhosherstov, Yury Maximov, Michael Chertkov
We present a new family of zero-field Ising models over binary variables/spins obtained by consecutive "gluing" of planar and -sized components and subsets of at most thr…
A New Family of Tractable Ising Models
Valerii Likhosherstov, Yury Maximov, Michael Chertkov
We present a new family of zero-field Ising models over N binary variables/spins obtained by consecutive "gluing" of planar and -sized components along with subsets of at mos…
Inference and Sampling of -free Ising Models
Valerii Likhosherstov, Yury Maximov, Michael Chertkov
We call an Ising model tractable when it is possible to compute its partition function value (statistical inference) in polynomial time. The tractability also implies an ability to…