7 citations · 10 across the 9 of their papers we have counts for
10 papers
Active Learning with Low-Rank Structure for Data Selection
Vincent Cohen-Addad, Sasidhar Kunapuli, Vahab Mirrokni +3
In the data selection problem, the objective is to choose a small, representative subset of data that can be used to efficiently train a machine learning model. Sener and Savarese…
GSQ: Highly-Accurate Low-Precision Scalar Quantization for LLMs via Gumbel-Softmax Sampling
Alireza Dadgarnia, Soroush Tabesh, Mahdi Nikdan +4
Quantization has become a standard tool for efficient LLM deployment, especially for local inference, where models are now routinely served at 2-3 bits per parameter. The state of…
ECO: Quantized Training without Full-Precision Master Weights
Mahdi Nikdan, Amir Zandieh, Dan Alistarh +1
Quantization has significantly improved the compute and memory efficiency of Large Language Model (LLM) training. However, existing approaches still rely on accumulating their upda…
Efficient Data Selection at Scale via Influence Distillation
Mahdi Nikdan, Vincent Cohen-Addad, Dan Alistarh +1
Effective data selection is critical for efficient training of modern Large Language Models (LLMs). This paper introduces Influence Distillation, a novel, mathematically-justified…
Quartet: Native FP4 Training Can Be Optimal for Large Language Models
Roberto L. Castro, Andrei Panferov, Soroush Tabesh +5
Training large language models (LLMs) models directly in low-precision offers a way to address computational costs by improving both throughput and energy efficiency. For those pur…
QuEST: Stable Training of LLMs with 1-Bit Weights and Activations
Andrei Panferov, Jiale Chen, Soroush Tabesh +3
One approach to reducing the massive costs of large language models (LLMs) is the use of quantized or sparse representations for training or deployment. While post-training compres…
HALO: Hadamard-Assisted Lower-Precision Optimization for LLMs
Saleh Ashkboos, Mahdi Nikdan, Soroush Tabesh +3
Quantized training of Large Language Models (LLMs) remains an open challenge, as maintaining accuracy while performing all matrix multiplications in low precision has proven diffic…
Panza: Design and Analysis of a Fully-Local Personalized Text Writing Assistant
Armand Nicolicioiu, Eugenia Iofinova, Andrej Jovanovic +6
The availability of powerful open-source large language models (LLMs) opens exciting use-cases, such as using personal data to fine-tune these models to imitate a user's unique wri…
RoSA: Accurate Parameter-Efficient Fine-Tuning via Robust Adaptation
Mahdi Nikdan, Soroush Tabesh, Elvir Crnčević +1
We investigate parameter-efficient fine-tuning (PEFT) methods that can provide good accuracy under limited computational and memory budgets in the context of large language models…
SparseProp: Efficient Sparse Backpropagation for Faster Training of Neural Networks
Mahdi Nikdan, Tommaso Pegolotti, Eugenia Iofinova +2
We provide a new efficient version of the backpropagation algorithm, specialized to the case where the weights of the neural network being trained are sparse. Our algorithm is gene…