170 citations · 210 across the 11 of their papers we have counts for
5 papers · 1 filter
FLUX: Geometry-Aware Longitudinal Flow Matching with Mixture of Experts
Josue Ortega Caro, Yongxu Zhang, Hannah M Batchelor +3
Many biological systems evolve through continuous local dynamics while switching between latent regimes defined by learning, stimulus context, internal state, or developmental stag…
Sparse-IFT: Sparse Iso-FLOP Transformations for Maximizing Training Efficiency
Vithursan Thangarasa, Shreyas Saxena, Abhay Gupta +1
Recent research has focused on weight sparsity in deep neural network training to reduce FLOPs, aiming for improved efficiency (test accuracy w.r.t training FLOPs). However, sparse…
SPDF: Sparse Pre-training and Dense Fine-tuning for Large Language Models
Vithursan Thangarasa, Abhay Gupta, William Marshall +5
The pre-training and fine-tuning paradigm has contributed to a number of breakthroughs in Natural Language Processing (NLP). Instead of directly training on a downstream task, lang…
Training With Data Dependent Dynamic Learning Rates
Shreyas Saxena, Nidhi Vyas, Dennis DeCoste
Recently many first and second order variants of SGD have been proposed to facilitate training of Deep Neural Networks (DNNs). A common limitation of these works stem from the fact…
Learning Soft Labels via Meta Learning
Nidhi Vyas, Shreyas Saxena, Thomas Voice
One-hot labels do not represent soft decision boundaries among concepts, and hence, models trained on them are prone to overfitting. Using soft labels as targets provide regulariza…