4 citations · 7 across the 11 of their papers we have counts for
7 papers · 1 filter
From Words to Amino Acids: Does the Curse of Depth Persist?
Aleena Siji, Amir Mohammad Karimi Mamaghan, Ferdinand Kapl +9
Protein language models (PLMs) have become widely adopted as general-purpose models, demonstrating strong performance in protein engineering and de novo design. Like large language…
Emergent temporal abstractions in autoregressive models enable hierarchical reinforcement learning
Seijin Kobayashi, Yanick Schimpf, Maximilian Schlegel +12
Large-scale autoregressive models pretrained on next-token prediction and finetuned with reinforcement learning (RL) have achieved unprecedented success on many problem domains. Du…
MesaNet: Sequence Modeling by Locally Optimal Test-Time Training
Johannes von Oswald, Nino Scherrer, Seijin Kobayashi +14
Sequence modeling is currently dominated by causal transformer architectures that use softmax self-attention. Although widely adopted, transformers require scaling memory and compu…
Composable Function-preserving Expansions for Transformer Architectures
Andrea Gesmundo, Kaitlin Maile
Training state-of-the-art neural networks requires a high cost in terms of compute and time. Model scale is recognized to be a critical factor to achieve and improve the state-of-t…
Equivariance-aware Architectural Optimization of Neural Networks
Kaitlin Maile, Dennis G. Wilson, Patrick Forré
Incorporating equivariance to symmetry groups as a constraint during neural network training can improve performance and generalization for tasks exhibiting those symmetries, but s…
When, where, and how to add new neurons to ANNs
Kaitlin Maile, Emmanuel Rachelson, Hervé Luga +1
Neurogenesis in ANNs is an understudied and difficult problem, even compared to other forms of structural learning like pruning. By decomposing it into triggers and initializations…
DARTS-PRIME: Regularization and Scheduling Improve Constrained Optimization in Differentiable NAS
Kaitlin Maile, Erwan Lecarpentier, Hervé Luga +1
Differentiable Architecture Search (DARTS) is a recent neural architecture search (NAS) method based on a differentiable relaxation. Due to its success, numerous variants analyzing…