30 citations · 39 across the 6 of their papers we have counts for
7 papers
Minimal Ranks, Maximum Confidence: Parameter-efficient Uncertainty Quantification for LoRA
Patryk Marszałek, Klaudia Bałazy, Jacek Tabor +1
Low-Rank Adaptation (LoRA) enables parameter-efficient fine-tuning of large language models by decomposing weight updates into low-rank matrices, significantly reducing storage and…
LoRA-XS: Low-Rank Adaptation with Extremely Small Number of Parameters
Klaudia Bałazy, Mohammadreza Banaei, Karl Aberer +1
The growth of large language models underscores the need for parameter-efficient fine-tuning. Despite its popularity, LoRA encounters storage and computational challenges when depl…
r-softmax: Generalized Softmax with Controllable Sparsity Rate
Klaudia Bałazy, Łukasz Struski, Marek Śmieja +1
Nowadays artificial neural network models achieve remarkable results in many disciplines. Functions mapping the representation provided by the model to the probability distribution…
Revisiting Offline Compression: Going Beyond Factorization-based Methods for Transformer Language Models
Mohammadreza Banaei, Klaudia Bałazy, Artur Kasymov +3
Recent transformer language models achieve outstanding results in many natural language processing (NLP) tasks. However, their enormous size often makes them impractical on memory-…
Direction is what you need: Improving Word Embedding Compression in Large Language Models
Klaudia Bałazy, Mohammadreza Banaei, Rémi Lebret +2
The adoption of Transformer-based models in natural language processing (NLP) has led to great success using a massive number of parameters. However, due to deployment constraints…
Zero Time Waste: Recycling Predictions in Early Exit Neural Networks
Maciej Wołczyk, Bartosz Wójcik, Klaudia Bałazy +4
The problem of reducing processing time of large deep learning models is a fundamental challenge in many real-world applications. Early exit methods strive towards this goal by att…
Finding the Optimal Network Depth in Classification Tasks
Bartosz Wójcik, Maciej Wołczyk, Klaudia Bałazy +1
We develop a fast end-to-end method for training lightweight neural networks using multiple classifier heads. By allowing the model to determine the importance of each head and rew…