552 citations · 816 across the 123 of their papers we have counts for
4 papers · 2 filters
Predicting the Order of Upcoming Tokens Improves Language Modeling
Zayd M. K. Zuhri, Erland Hilman Fuadi, Alham Fikri Aji
Multi-token prediction (MTP) has been proposed as an auxiliary objective to improve next-token prediction (NTP) in language model training but shows inconsistent improvements, unde…
Datasheets Aren't Enough: DataRubrics for Automated Quality Metrics and Accountability
Genta Indra Winata, David Anugraha, Emmy Liu +17
High-quality datasets are fundamental to training and evaluating machine learning models, yet their creation-especially with accurate human annotations-remains a significant challe…
Softpick: No Attention Sink, No Massive Activations with Rectified Softmax
Zayd M. K. Zuhri, Erland Hilman Fuadi, Alham Fikri Aji
We introduce softpick, a rectified, not sum-to-one, drop-in replacement for softmax in transformer attention mechanisms that eliminates attention sink and massive activations. Our…
QLESS: A Quantized Approach for Data Valuation and Selection in Large Language Model Fine-Tuning
Moses Ananta, Muhammad Farid Adilazuarda, Zayd Muhammad Kawakibi Zuhri +2
Fine-tuning large language models (LLMs) is often constrained by the computational costs of processing massive datasets. We propose \textbf{QLESS} (Quantized Low-rank Gradient Simi…