5 papers
Where to Begin: Efficient Pretraining via Subnetwork Selection and Distillation
Arjun Krishnakumar, Rhea Sanjay Sukthanker, Hannan Javed Mahadik +5
Small Language models (SLMs) offer an efficient and accessible alternative to Large Language Models (LLMs), delivering strong performance while using far fewer resources. We introd…
Weight-Entanglement Meets Gradient-Based Neural Architecture Search
Rhea Sanjay Sukthanker, Arjun Krishnakumar, Mahmoud Safari +1
Weight sharing is a fundamental concept in neural architecture search (NAS), enabling gradient-based methods to explore cell-based architectural spaces significantly faster than tr…
Compressing Large Language Models with Automated Sub-Network Search
Rhea Sanjay Sukthanker, Benedikt Staffler, Frank Hutter +1
Large Language Models (LLMs) demonstrate exceptional reasoning abilities, enabling strong generalization across diverse tasks such as commonsense reasoning and instruction followin…
Multi-objective Differentiable Neural Architecture Search
Rhea Sanjay Sukthanker, Arber Zela, Benedikt Staffler +3
Pareto front profiling in multi-objective optimization (MOO), i.e., finding a diverse set of Pareto optimal solutions, is challenging, especially with expensive objectives that req…
HW-GPT-Bench: Hardware-Aware Architecture Benchmark for Language Models
Rhea Sanjay Sukthanker, Arber Zela, Benedikt Staffler +4
The increasing size of language models necessitates a thorough analysis across multiple dimensions to assess trade-offs among crucial hardware metrics such as latency, energy consu…