4 papers
TetraJet-v2: Accurate NVFP4 Training for Large Language Models with Oscillation Suppression and Outlier Control
Yuxiang Chen, Yifan Liu, Xiaoming Xu +5
Large Language Models (LLMs) training is prohibitively expensive, driving interest in low-precision fully-quantized training (FQT). While novel 4-bit formats like NVFP4 offer subst…
confopt: A Library for Implementation and Evaluation of Gradient-based One-Shot NAS Methods
Abhash Kumar Jha, Shakiba Moradian, Arjun Krishnakumar +2
Gradient-based one-shot neural architecture search (NAS) has significantly reduced the cost of exploring architectural spaces with discrete design choices, such as selecting operat…
Frozen Layers: Memory-efficient Many-fidelity Hyperparameter Optimization
Timur Carstensen, Neeratyoy Mallik, Frank Hutter +1
As model sizes grow, finding efficient and cost-effective hyperparameter optimization (HPO) methods becomes increasingly crucial for deep learning pipelines. While multi-fidelity H…
Attention Is All You Need For Mixture-of-Depths Routing
Advait Gadhikar, Souptik Kumar Majumdar, Niclas Popp +3
Advancements in deep learning are driven by training models with increasingly larger numbers of parameters, which in turn heightens the computational demands. To address this issue…