Composable Function-preserving Expansions for Transformer Architectures

Andrea Gesmundo, Kaitlin Maile

Training state-of-the-art neural networks requires a high cost in terms of compute and time. Model scale is recognized to be a critical factor to achieve and improve the state-of-t…