3 citations · 8 across the 23 of their papers we have counts for
4 papers · 1 filter
When Can Attention Heads Be Statically Defined?
Weixian Waylon Li, Yintao Tai, Marcio Fonseca +1
Some attention heads learn similar patterns across inputs. Reusing these patterns could reduce training cost by avoiding repeated query-key score computation and softmax. Through c…
ONNX-Net: Towards Universal Representations and Instant Performance Prediction for Neural Architectures
Shiwen Qin, Alexander Auras, Shay B. Cohen +4
Neural architecture search (NAS) automates the design process of high-performing architectures, but remains bottlenecked by expensive performance evaluation. Most existing studies…
Transferrable Surrogates in Expressive Neural Architecture Search Spaces
Shiwen Qin, Gabriela Kadlecová, Martin Pilát +5
Neural architecture search (NAS) faces a challenge in balancing the exploration of expressive, broad search spaces that enable architectural innovation with the need for efficient…
einspace: Searching for Neural Architectures from Fundamental Operations
Linus Ericsson, Miguel Espinosa, Chenhongyi Yang +5
Neural architecture search (NAS) finds high performing networks for a given task. Yet the results of NAS are fairly prosaic; they did not e.g. create a shift from convolutional str…