EquiformerV2: Improved Equivariant Transformer for Scaling to Higher-Degree Representations

Abstract

Equivariant Transformers such as Equiformer have demonstrated the efficacy of applying Transformers to the domain of 3D atomistic systems. However, they are limited to small degrees of equivariant representations due to their computational complexity. In this paper, we investigate whether these architectures can scale well to higher degrees. Starting from Equiformer, we first replace $SO(3)$ convolutions with eSCN convolutions to efficiently incorporate higher-degree tensors. Then, to better leverage the power of higher degrees, we propose three architectural improvements -- attention re-normalization, separable $S^2$ activation and separable layer normalization. Putting this all together, we propose EquiformerV2, which outperforms previous state-of-the-art methods on large-scale OC20 dataset by up to $9\%$ on forces, $4\%$ on energies, offers better speed-accuracy trade-offs, and $2\times$ reduction in DFT calculations needed for computing adsorption energies. Additionally, EquiformerV2 trained on only OC22 dataset outperforms GemNet-OC trained on both OC20 and OC22 datasets, achieving much better data efficiency. Finally, we compare EquiformerV2 with Equiformer on QM9 and OC20 S2EF-2M datasets to better understand the performance gain brought by higher degrees.

Published as a conference paper at ICLR 2024

Cited by in corpus (9)

• A practical guide to machine learning interatomic potentials -- Status and future
• DPA-2: a large atomic model as a multi-task learner
• Generative Language Model for Catalyst Discovery
• An equivariant graph neural network for the elasticity tensors of all seven crystal systems
• Accelerating the prediction of inorganic surfaces with machine learning interatomic potentials
• Improved decision making with similarity based machine learning: Applications in chemistry
• Rotation-equivariant Graph Neural Networks for Learning Glassy Liquids Representations
• Self-Supervised Generative Models for Crystal Structures
• Application-specific machine-learned interatomic potentials: exploring the trade-off between DFT convergence, MLIP expressivity, and computational cost