18 citations · 56 across the 18 of their papers we have counts for
13 papers · 1 filter
Transformation-Augmented GRPO for Enhancing Exploration in Reasoning of Large Language Models
Khiem Le, Phuc Nguyen, Youssef Mroueh +4
Group Relative Policy Optimization (GRPO) has become the dominant method for reinforcement learning with verifiable rewards in large language models, but it suffers from two critic…
Your contrastive learning problem is secretly a distribution alignment problem
Zihao Chen, Chi-Heng Lin, Ran Liu +2
Despite the success of contrastive learning (CL) in vision and language, its theoretical foundations and mechanisms for building representations remain poorly understood. In this w…
ToMoE: Converting Dense Large Language Models to Mixture-of-Experts through Dynamic Structural Pruning
Shangqian Gao, Ting Hua, Reza Shirkavand +10
Large Language Models (LLMs) have demonstrated remarkable abilities in tackling a wide range of complex tasks. However, their huge computational and memory costs raise significant…
MoDeGPT: Modular Decomposition for Large Language Model Compression
Chi-Heng Lin, Shangqian Gao, James Seale Smith +5
Large Language Models (LLMs) have reshaped the landscape of artificial intelligence by demonstrating exceptional performance across various tasks. However, substantial computationa…
Balanced Data, Imbalanced Spectra: Unveiling Class Disparities with Spectral Imbalance
Chiraag Kaushik, Ran Liu, Chi-Heng Lin +5
Classification models are expected to perform equally well for different classes, yet in practice, there are often large gaps in their performance. This issue of class bias is wide…
Half-Hop: A graph upsampling approach for slowing down message passing
Mehdi Azabou, Venkataramana Ganesh, Shantanu Thakoor +6
Message passing neural networks have shown a lot of success on graph-structured data. However, there are many instances where message passing can lead to over-smoothing or fail whe…
The good, the bad and the ugly sides of data augmentation: An implicit spectral regularization perspective
Chi-Heng Lin, Chiraag Kaushik, Eva L. Dyer +1
Data augmentation (DA) is a powerful workhorse for bolstering performance in modern machine learning. Specific augmentations like translations and scaling in computer vision are tr…
Drop, Swap, and Generate: A Self-Supervised Approach for Generating Neural Activity
Ran Liu, Mehdi Azabou, Max Dabagia +5
Meaningful and simplified representations of neural activity can yield insights into how and what information is being processed within a neural circuit. However, without labels, f…
Escaping Saddle Points Faster with Stochastic Momentum
Jun-Kun Wang, Chi-Heng Lin, Jacob Abernethy
Stochastic gradient descent (SGD) with stochastic momentum is popular in nonconvex stochastic optimization and particularly for the training of deep neural networks. In standard SG…
Mine Your Own vieW: Self-Supervised Learning Through Across-Sample Prediction
Mehdi Azabou, Mohammad Gheshlaghi Azar, Ran Liu +10
State-of-the-art methods for self-supervised learning (SSL) build representations by maximizing the similarity between different transformed "views" of a sample. Without sufficient…
Making transport more robust and interpretable by moving data through a small number of anchor points
Chi-Heng Lin, Mehdi Azabou, Eva L. Dyer
Optimal transport (OT) is a widely used technique for distribution alignment, with applications throughout the machine learning, graphics, and vision communities. Without any addit…
A Modular Analysis of Provable Acceleration via Polyak's Momentum: Training a Wide ReLU Network and a Deep Linear Network
Jun-Kun Wang, Chi-Heng Lin, Jacob Abernethy
Incorporating a so-called "momentum" dynamic in gradient descent methods is widely used in neural net training as it has been broadly observed that, at least empirically, it often…
Bayesian optimization for modular black-box systems with switching costs
Chi-Heng Lin, Joseph D. Miano, Eva L. Dyer
Most existing black-box optimization methods assume that all variables in the system being optimized have equal cost and can change freely at each iteration. However, in many real…