34 citations · 82 across the 8 of their papers we have counts for
8 papers
Sparse Upcycling: Inference Inefficient Finetuning
Sasha Doubov, Nikhil Sardana, Vitaliy Chiley
Small, highly trained, open-source large language models are widely used due to their inference efficiency, but further improving their quality remains a challenge. Sparse upcyclin…
Inference-Friendly Models With MixAttention
Shashank Rajput, Ying Sheng, Sean Owen +1
The size of the key-value (KV) cache plays a critical role in determining both the maximum context length and the number of concurrent requests supported during inference in modern…
LoRA Learns Less and Forgets Less
Dan Biderman, Jacob Portes, Jose Javier Gonzalez Ortiz +9
Low-Rank Adaptation (LoRA) is a widely-used parameter-efficient finetuning method for large language models. LoRA saves memory by training only low rank perturbations to selected w…
RevBiFPN: The Fully Reversible Bidirectional Feature Pyramid Network
Vitaliy Chiley, Vithursan Thangarasa, Abhay Gupta +3
This work introduces RevSilo, the first reversible bidirectional multi-scale feature fusion module. Like other reversible methods, RevSilo eliminates the need to store hidden activ…
Memory Efficient 3D U-Net with Reversible Mobile Inverted Bottlenecks for Brain Tumor Segmentation
Mihir Pendse, Vithursan Thangarasa, Vitaliy Chiley +3
We propose combining memory saving techniques with traditional U-Net architectures to increase the complexity of the models on the Brain Tumor Segmentation (BraTS) challenge. The B…
Adaptive Braking for Mitigating Gradient Delay
Abhinav Venigalla, Atli Kosson, Vitaliy Chiley +1
Neural network training is commonly accelerated by using multiple synchronized workers to compute gradient updates in parallel. Asynchronous methods remove synchronization overhead…
Pipelined Backpropagation at Scale: Training Large Models without Batches
Atli Kosson, Vitaliy Chiley, Abhinav Venigalla +2
New hardware can substantially increase the speed and efficiency of deep neural network training. To guide the development of future hardware architectures, it is pertinent to expl…
Online Normalization for Training Neural Networks
Vitaliy Chiley, Ilya Sharapov, Atli Kosson +5
Online Normalization is a new technique for normalizing the hidden activations of a neural network. Like Batch Normalization, it normalizes the sample dimension. While Online Norma…