4 papers
Correcting the Dropout-LayerNorm Expectation Gap Improves Protein Structure Models
Isaac Ellmen, David Errington, Matthew I. J. Raybould +1
Although , here we show that is not equal to . Accordingly, the pa…
Mechanistic Interpretability of Antibody Language Models Using SAEs
Rebonto Haque, Oliver M. Turnbull, Anisha Parsan +4
Sparse autoencoders (SAEs) are a mechanistic interpretability technique that have been used to provide insight into learned concepts within large protein language models. Here, we…
Kernel-Based Evaluation of Conditional Biological Sequence Models
Pierre Glaser, Steffanie Paul, Alissa M. Hummer +3
We propose a set of kernel-based tools to evaluate the designs and tune the hyperparameters of conditional sequence models, with a focus on problems in computational biology. The b…
Transformers trained on proteins can learn to attend to Euclidean distance
Isaac Ellmen, Constantin Schneider, Matthew I. J. Raybould +1
While conventional Transformers generally operate on sequence data, they can be used in conjunction with structure models, typically SE(3)-invariant or equivariant graph neural net…