318 citations · 825 across the 71 of their papers we have counts for
14 papers · 1 filter
Structuring MoE Expert Selection for Agentic Reinforcement Learning
Bolian Li, Ting-Yao Hu, Cheng-Yu Hsieh +3
Long-horizon LLM agents are frequently implemented using sparse mixture-of-experts (MoE) models, yet the co-design of agentic behavior and MoE structures remains underexplored. In…
Proxy-FDA: Proxy-based Feature Distribution Alignment for Fine-tuning Vision Foundation Models without Forgetting
Chen Huang, Skyler Seto, Hadi Pouransari +6
Vision foundation models pre-trained on massive data encode rich representations of real-world concepts, which can be adapted to downstream tasks by fine-tuning. However, fine-tuni…
TiC-LM: A Web-Scale Benchmark for Time-Continual LLM Pretraining
Jeffrey Li, Mohammadreza Armandpour, Iman Mirzadeh +8
Large Language Models (LLMs) trained on historical web data inevitably become outdated. We investigate evaluation strategies and update methods for LLMs as new data becomes availab…
GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models
Iman Mirzadeh, Keivan Alizadeh, Hooman Shahrokhi +3
Recent advancements in Large Language Models (LLMs) have sparked interest in their formal reasoning capabilities, particularly in mathematics. The GSM8K benchmark is widely used to…
Weight subcloning: direct initialization of transformers using larger pretrained ones
Mohammad Samragh, Mehrdad Farajtabar, Sachin Mehta +5
Training large transformer models from scratch for a target task requires lots of data and is computationally demanding. The usual practice of transfer learning overcomes this chal…
CLIP meets Model Zoo Experts: Pseudo-Supervision for Visual Enhancement
Mohammadreza Salehi, Mehrdad Farajtabar, Maxwell Horton +7
Contrastive language image pretraining (CLIP) is a standard method for training vision-language models. While CLIP is scalable, promptable, and robust to distribution shifts on ima…
ReLU Strikes Back: Exploiting Activation Sparsity in Large Language Models
Iman Mirzadeh, Keivan Alizadeh, Sachin Mehta +5
Large Language Models (LLMs) with billions of parameters have drastically transformed AI applications. However, their demanding computation during inference has raised significant…
APE: Aligning Pretrained Encoders to Quickly Learn Aligned Multimodal Representations
Elan Rosenfeld, Preetum Nakkiran, Hadi Pouransari +2
Recent advances in learning aligned multimodal representations have been primarily driven by training large neural networks on massive, noisy paired-modality datasets. In this work…
Style Equalization: Unsupervised Learning of Controllable Generative Sequence Models
Jen-Hao Rick Chang, Ashish Shrivastava, Hema Swetha Koppula +2
Controllable generative sequence models with the capability to extract and replicate the style of specific examples enable many applications, including narrating audiobooks in diff…
SapAugment: Learning A Sample Adaptive Policy for Data Augmentation
Ting-Yao Hu, Ashish Shrivastava, Jen-Hao Rick Chang +5
Data augmentation methods usually apply the same augmentation (or a mix of them) to all the training samples. For example, to perturb data with noise, the noise is sampled from a N…
Extracurricular Learning: Knowledge Transfer Beyond Empirical Distribution
Hadi Pouransari, Mojan Javaheripi, Vinay Sharma +1
Knowledge distillation has been used to transfer knowledge learned by a sophisticated model (teacher) to a simpler model (student). This technique is widely used to compress model…
Subject-Aware Contrastive Learning for Biosignals
Joseph Y. Cheng, Hanlin Goh, Kaan Dogrusoz +2
Datasets for biosignals, such as electroencephalogram (EEG) and electrocardiogram (ECG), often have noisy labels and have limited number of subjects (<100). To handle these challen…
Least squares binary quantization of neural networks
Hadi Pouransari, Zhucheng Tu, Oncel Tuzel
Quantizing weights and activations of deep neural networks results in significant improvement in inference efficiency at the cost of lower accuracy. A source of the accuracy gap be…
Nonlinear Conjugate Gradients For Scaling Synchronous Distributed DNN Training
Saurabh Adya, Vinay Palakkode, Oncel Tuzel
Nonlinear conjugate gradient (NLCG) based optimizers have shown superior loss convergence properties compared to gradient descent based optimizers for traditional optimization prob…