3 papers
cs.CL2025
Where to Begin: Efficient Pretraining via Subnetwork Selection and Distillation
Arjun Krishnakumar, Rhea Sanjay Sukthanker, Hannan Javed Mahadik +5
Small Language models (SLMs) offer an efficient and accessible alternative to Large Language Models (LLMs), delivering strong performance while using far fewer resources. We introd…
cs.LG2025
confopt: A Library for Implementation and Evaluation of Gradient-based One-Shot NAS Methods
Abhash Kumar Jha, Shakiba Moradian, Arjun Krishnakumar +2
Gradient-based one-shot neural architecture search (NAS) has significantly reduced the cost of exploring architectural spaces with discrete design choices, such as selecting operat…
cs.LG2025
Gompertz Linear Units: Leveraging Asymmetry for Enhanced Learning Dynamics
Indrashis Das, Mahmoud Safari, Steven Adriaensen +1
Activation functions are fundamental elements of deep learning architectures as they significantly influence training dynamics. ReLU, while widely used, is prone to the dying neuro…