75 citations · 110 across the 7 of their papers we have counts for
8 papers · 1 filter
SVD-Surgeon: Optimal Singular-Value Surgery for Large Language Model Compression
Mahmoud Safari, Frank Hutter
Large language models (LLMs) achieve remarkable performance across a wide range of tasks, but their deployment is constrained by substantial memory and compute requirements. Low-ra…
Gompertz Linear Units: Leveraging Asymmetry for Enhanced Learning Dynamics
Indrashis Das, Mahmoud Safari, Steven Adriaensen +1
Activation functions are fundamental elements of deep learning architectures as they significantly influence training dynamics. ReLU, while widely used, is prone to the dying neuro…
Efficient Search for Customized Activation Functions with Gradient Descent
Lukas Strack, Mahmoud Safari, Frank Hutter
Different activation functions work best for different deep learning models. To exploit this, we leverage recent advancements in gradient-based search techniques for neural archite…
Surprisingly Strong Performance Prediction with Neural Graph Features
Gabriela Kadlecová, Jovita Lukasik, Martin Pilát +4
Performance prediction has been a key part of the neural architecture search (NAS) process, allowing to speed up NAS algorithms by avoiding resource-consuming network training. Alt…
Weight-Entanglement Meets Gradient-Based Neural Architecture Search
Rhea Sanjay Sukthanker, Arjun Krishnakumar, Mahmoud Safari +1
Weight sharing is a fundamental concept in neural architecture search (NAS), enabling gradient-based methods to explore cell-based architectural spaces significantly faster than tr…
Neural Architecture Search: Insights from 1000 Papers
Colin White, Mahmoud Safari, Rhea Sukthanker +5
In the past decade, advances in deep learning have resulted in breakthroughs in a variety of areas, including computer vision, natural language understanding, speech recognition, a…