2 citations · 3 across the 12 of their papers we have counts for
7 papers · 1 filter
Model Balancing Helps Low-data Training and Fine-tuning
Zihang Liu, Yuanzhe Hu, Tianyu Pang +3
Recent advances in foundation models have emphasized the need to align pre-trained models with specialized domains using small, curated datasets. Studies on these foundation models…
AlphaPruning: Using Heavy-Tailed Self Regularization Theory for Improved Layer-wise Pruning of Large Language Models
Haiquan Lu, Yefan Zhou, Shiwei Liu +3
Recent work on pruning large language models (LLMs) has shown that one can eliminate a large number of parameters without compromising performance, making pruning a promising strat…
AlphaLoRA: Assigning LoRA Experts Based on Layer Training Quality
Peijun Qing, Chongyang Gao, Yefan Zhou +3
Parameter-efficient fine-tuning methods, such as Low-Rank Adaptation (LoRA), are known to enhance training efficiency in Large Language Models (LLMs). Due to the limited parameters…
Mitigating Memorization In Language Models
Mansi Sakarvadia, Aswathy Ajith, Arham Khan +6
Language models (LMs) can "memorize" information, i.e., encode training data in their weights in such a way that inference-time queries can lead to verbatim regurgitation of that d…
Sharpness-diversity tradeoff: improving flat ensembles with SharpBalance
Haiquan Lu, Xiaotian Liu, Yefan Zhou +6
Recent studies on deep ensembles have identified the sharpness of the local minima of individual learners and the diversity of the ensemble members as key factors in improving test…
MD tree: a model-diagnostic tree grown on loss landscape
Yefan Zhou, Jianlong Chen, Qinxue Cao +2
This paper considers "model diagnosis", which we formulate as a classification problem. Given a pre-trained neural network (NN), the goal is to predict the source of failure from a…
From Spikes to Heavy Tails: Unveiling the Spectral Evolution of Neural Networks
Vignesh Kothapalli, Tianyu Pang, Shenyang Deng +2
Training strategies for modern deep neural networks (NNs) tend to induce a heavy-tailed (HT) empirical spectral density (ESD) in the layer weights. While previous efforts have show…