75 citations · 110 across the 7 of their papers we have counts for
8 papers
SVD-Surgeon: Optimal Singular-Value Surgery for Large Language Model Compression
Mahmoud Safari, Frank Hutter
Large language models (LLMs) achieve remarkable performance across a wide range of tasks, but their deployment is constrained by substantial memory and compute requirements. Low-ra…
Gompertz Linear Units: Leveraging Asymmetry for Enhanced Learning Dynamics
Indrashis Das, Mahmoud Safari, Steven Adriaensen +1
Activation functions are fundamental elements of deep learning architectures as they significantly influence training dynamics. ReLU, while widely used, is prone to the dying neuro…
Efficient Search for Customized Activation Functions with Gradient Descent
Lukas Strack, Mahmoud Safari, Frank Hutter
Different activation functions work best for different deep learning models. To exploit this, we leverage recent advancements in gradient-based search techniques for neural archite…
Surprisingly Strong Performance Prediction with Neural Graph Features
Gabriela Kadlecová, Jovita Lukasik, Martin Pilát +4
Performance prediction has been a key part of the neural architecture search (NAS) process, allowing to speed up NAS algorithms by avoiding resource-consuming network training. Alt…
Weight-Entanglement Meets Gradient-Based Neural Architecture Search
Rhea Sanjay Sukthanker, Arjun Krishnakumar, Mahmoud Safari +1
Weight sharing is a fundamental concept in neural architecture search (NAS), enabling gradient-based methods to explore cell-based architectural spaces significantly faster than tr…
Neural Architecture Search: Insights from 1000 Papers
Colin White, Mahmoud Safari, Rhea Sukthanker +5
In the past decade, advances in deep learning have resulted in breakthroughs in a variety of areas, including computer vision, natural language understanding, speech recognition, a…
NAS-Bench-Suite-Zero: Accelerating Research on Zero Cost Proxies
Arjun Krishnakumar, Colin White, Arber Zela +3
Zero-cost proxies (ZC proxies) are a recent architecture performance prediction technique aiming to significantly speed up algorithms for neural architecture search (NAS). Recent w…
NAS-Bench-Suite: NAS Evaluation is (Now) Surprisingly Easy
Yash Mehta, Colin White, Arber Zela +6
The release of tabular benchmarks, such as NAS-Bench-101 and NAS-Bench-201, has significantly lowered the computational overhead for conducting scientific research in neural archit…