8 papers
Measuring and Reducing Cross-Vendor Mismatch in Language Models
Erland Hilman Fuadi, Chong Tian, Xiaosong Ma +1
Running the same language model on different graphics processing unit (GPU) vendors can produce different logits, even when the model weights and inputs are the same. We analyze cr…
Ego2Act: Evaluating Goal-Directed Manipulation in Egocentric Video Generation
Patrick Amadeus Irawan, Iskandar Muda Rizky Parlambang, Rava Maulana +10
Video generation models are increasingly being explored as world simulators for embodied planning and learning. To do so effectively, these models must not only generate visually a…
COPUS: Co-adaptive Parallelism and Batch Size Selection in Large Language Model Training
Akhmed Sakip, Erland Hilman Fuadi, Omar Sayedelahl +6
Training large language models requires jointly configuring two interdependent aspects of the system: the global batch size, which governs statistical efficiency, and the 3D parall…
LinguDistill: Recovering Linguistic Ability in Vision-Language Models via Selective Cross-Modal Distillation
Patrick Amadeus Irawan, Erland Hilman Fuadi, Shanu Kumar +2
Turning a pretrained language model (LM) into a vision-language model (VLM) through multimodal fine-tuning often erodes its native language ability, a form of catastrophic forgetti…
Predicting the Order of Upcoming Tokens Improves Language Modeling
Zayd M. K. Zuhri, Erland Hilman Fuadi, Alham Fikri Aji
Multi-token prediction (MTP) has been proposed as an auxiliary objective to improve next-token prediction (NTP) in language model training but shows inconsistent improvements, unde…
Softpick: No Attention Sink, No Massive Activations with Rectified Softmax
Zayd M. K. Zuhri, Erland Hilman Fuadi, Alham Fikri Aji
We introduce softpick, a rectified, not sum-to-one, drop-in replacement for softmax in transformer attention mechanisms that eliminates attention sink and massive activations. Our…
COPAL-ID: Indonesian Language Reasoning with Local Culture and Nuances
Haryo Akbarianto Wibowo, Erland Hilman Fuadi, Made Nindyatama Nityasya +2
We present COPAL-ID, a novel, public Indonesian language common sense reasoning dataset. Unlike the previous Indonesian COPA dataset (XCOPA-ID), COPAL-ID incorporates Indonesian lo…
Gated Self-supervised Learning For Improving Supervised Learning
Erland Hilman Fuadi, Aristo Renaldo Ruslim, Putu Wahyu Kusuma Wardhana +1
In past research on self-supervised learning for image classification, the use of rotation as an augmentation has been common. However, relying solely on rotation as a self-supervi…