60 citations · 150 across the 9 of their papers we have counts for
9 papers
SPDF: Sparse Pre-training and Dense Fine-tuning for Large Language Models
Vithursan Thangarasa, Abhay Gupta, William Marshall +5
The pre-training and fine-tuning paradigm has contributed to a number of breakthroughs in Natural Language Processing (NLP). Instead of directly training on a downstream task, lang…
RevBiFPN: The Fully Reversible Bidirectional Feature Pyramid Network
Vitaliy Chiley, Vithursan Thangarasa, Abhay Gupta +3
This work introduces RevSilo, the first reversible bidirectional multi-scale feature fusion module. Like other reversible methods, RevSilo eliminates the need to store hidden activ…
Training With Data Dependent Dynamic Learning Rates
Shreyas Saxena, Nidhi Vyas, Dennis DeCoste
Recently many first and second order variants of SGD have been proposed to facilitate training of Deep Neural Networks (DNNs). A common limitation of these works stem from the fact…
Memory Efficient 3D U-Net with Reversible Mobile Inverted Bottlenecks for Brain Tumor Segmentation
Mihir Pendse, Vithursan Thangarasa, Vitaliy Chiley +3
We propose combining memory saving techniques with traditional U-Net architectures to increase the complexity of the models on the Brain Tumor Segmentation (BraTS) challenge. The B…
Stochastic Weight Averaging in Parallel: Large-Batch Training that Generalizes Well
Vipul Gupta, Santiago Akle Serrano, Dennis DeCoste
We propose Stochastic Weight Averaging in Parallel (SWAP), an algorithm to accelerate DNN training. Our algorithm uses large mini-batches to compute an approximate solution quickly…
Hot Swapping for Online Adaptation of Optimization Hyperparameters
Kevin Bache, Dennis DeCoste, Padhraic Smyth
We describe a general framework for online adaptation of optimization hyperparameters by `hot swapping' their values during learning. We investigate this approach in the context of…