113 citations · 1k across the 66 of their papers we have counts for
6 papers · 1 filter
Computational Bottlenecks of Training Small-scale Large Language Models
Saleh Ashkboos, Iman Mirzadeh, Keivan Alizadeh +4
While large language models (LLMs) dominate the AI landscape, Small-scale large Language Models (SLMs) are gaining attention due to cost and efficiency demands from consumers. Howe…
SALSA: Soup-based Alignment Learning for Stronger Adaptation in RLHF
Atoosa Chegini, Hamid Kazemi, Iman Mirzadeh +5
In Large Language Model (LLM) development, Reinforcement Learning from Human Feedback (RLHF) is crucial for aligning models with human values and preferences. RLHF traditionally re…
Duo-LLM: A Framework for Studying Adaptive Computation in Large Language Models
Keivan Alizadeh, Iman Mirzadeh, Hooman Shahrokhi +6
Large Language Models (LLMs) typically generate outputs token by token using a fixed compute budget, leading to inefficient resource utilization. To address this shortcoming, recen…
GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models
Iman Mirzadeh, Keivan Alizadeh, Hooman Shahrokhi +3
Recent advancements in Large Language Models (LLMs) have sparked interest in their formal reasoning capabilities, particularly in mathematics. The GSM8K benchmark is widely used to…
Scaling Smart: Accelerating Large Language Model Pre-training with Small Model Initialization
Mohammad Samragh, Iman Mirzadeh, Keivan Alizadeh Vahid +5
The pre-training phase of language models often begins with randomly initialized parameters. With the current trends in scaling models, training their large number of parameters ca…
CatLIP: CLIP-level Visual Recognition Accuracy with 2.7x Faster Pre-training on Web-scale Image-Text Data
Sachin Mehta, Maxwell Horton, Fartash Faghri +5
Contrastive learning has emerged as a transformative method for learning effective visual representations through the alignment of image and text embeddings. However, pairwise simi…