77 citations · 146 across the 10 of their papers we have counts for
9 papers · 1 filter
LiDAR: Sensing Linear Probing Performance in Joint Embedding SSL Architectures
Vimal Thilak, Chen Huang, Omid Saremi +5
Joint embedding (JE) architectures have emerged as a promising avenue for acquiring transferable data representations. A key obstacle to using JE methods, however, is the inherent…
Frequency-Aware Masked Autoencoders for Multimodal Pretraining on Biosignals
Ran Liu, Ellen L. Zippi, Hadi Pouransari +5
Leveraging multimodal information from biosignals is vital for building a comprehensive representation of people's physical and mental states. However, multimodal biosignals often…
MAST: Masked Augmentation Subspace Training for Generalizable Self-Supervised Priors
Chen Huang, Hanlin Goh, Jiatao Gu +1
Recent Self-Supervised Learning (SSL) methods are able to learn feature representations that are invariant to different data augmentations, which can then be transferred to downstr…
Implicit Acceleration and Feature Learning in Infinitely Wide Neural Networks with Bottlenecks
Etai Littwin, Omid Saremi, Shuangfei Zhai +4
We analyze the learning dynamics of infinitely wide neural networks with a finite sized bottle-neck. Unlike the neural tangent kernel limit, a bottleneck in an otherwise infinite w…
Uncertainty Weighted Actor-Critic for Offline Reinforcement Learning
Yue Wu, Shuangfei Zhai, Nitish Srivastava +4
Offline Reinforcement Learning promises to learn effective policies from previously-collected, static datasets without the need for exploration. However, existing Q-learning and ac…
An Attention Free Transformer
Shuangfei Zhai, Walter Talbott, Nitish Srivastava +4
We introduce Attention Free Transformer (AFT), an efficient variant of Transformers that eliminates the need for dot product self attention. In an AFT layer, the key and value are…