221 citations · 746 across the 6 of their papers we have counts for
10 papers
ST-MoE: Designing Stable and Transferable Sparse Expert Models
Barret Zoph, Irwan Bello, Sameer Kumar +5
Scale has opened new frontiers in natural language processing -- but at a high cost. In response, Mixture-of-Experts (MoE) and Switch Transformers have been proposed as an energy e…
Revisiting 3D ResNets for Video Recognition
Xianzhi Du, Yeqing Li, Yin Cui +3
A recent work from Bello shows that training and scaling strategies may be more significant than model architectures for visual recognition. This short note studies effective train…
Revisiting ResNets: Improved Training and Scaling Strategies
Irwan Bello, William Fedus, Xianzhi Du +5
Novel computer vision architectures monopolize the spotlight, but the impact of the model architecture is often conflated with simultaneous changes to training methodology and scal…
LambdaNetworks: Modeling Long-Range Interactions Without Attention
Irwan Bello
We present lambda layers -- an alternative framework to self-attention -- for capturing long-range interactions between an input and structured contextual information (e.g. a pixel…
Global Self-Attention Networks for Image Recognition
Zhuoran Shen, Irwan Bello, Raviteja Vemulapalli +2
Recently, a series of works in computer vision have shown promising results on various image and video understanding tasks using self-attention. However, due to the quadratic compu…
Stand-Alone Self-Attention in Vision Models
Prajit Ramachandran, Niki Parmar, Ashish Vaswani +3
Convolutions are a fundamental building block of modern computer vision systems. Recent approaches have argued for going beyond convolutions in order to capture long-range dependen…