10 citations · 26 across the 20 of their papers we have counts for
16 papers · 1 filter
Compress then Merge: From Multiple LoRAs into One Low-Rank Adapter
Zhengbao He, Ruiqi Ding, Zhehao Huang +3
Low-rank adaptation (LoRA) enables parameter-efficient specialization of foundation models, but the proliferation of task-specific adapters fragments capabilities across many adapt…
VL-RouterBench: A Benchmark for Vision-Language Model Routing
Zhehao Huang, Baijiong Lin, Jingyuan Zhang +5
Multi-model routing has evolved from an engineering technique into essential infrastructure, yet existing work lacks a systematic, reproducible benchmark for evaluating vision-lang…
Bi-LoRA: Efficient Sharpness-Aware Minimization for Fine-Tuning Large-Scale Models
Yuhang Liu, Tao Li, Zhehao Huang +2
Fine-tuning large-scale pre-trained models with limited data presents significant challenges for generalization. While Sharpness-Aware Minimization (SAM) has proven effective in im…
A Unified Gradient-based Framework for Task-agnostic Continual Learning-Unlearning
Zhehao Huang, Xinwen Cheng, Jie Zhang +5
Recent advancements in deep models have highlighted the need for intelligent systems that combine continual learning (CL) for knowledge acquisition with machine unlearning (MU) for…
Unified Gradient-Based Machine Unlearning with Remain Geometry Enhancement
Zhehao Huang, Xinwen Cheng, JingHao Zheng +4
Machine unlearning (MU) has emerged to enhance the privacy and trustworthiness of deep neural networks. Approximate MU is a practical method for large-scale models. Our investigati…
Flat-LoRA: Low-Rank Adaptation over a Flat Loss Landscape
Tao Li, Zhengbao He, Yujun Li +3
Fine-tuning large-scale pre-trained models is prohibitively expensive in terms of computation and memory costs. Low-Rank Adaptation (LoRA), a popular Parameter-Efficient Fine-Tunin…
Learning Scalable Model Soup on a Single GPU: An Efficient Subspace Training Strategy
Tao Li, Weisen Jiang, Fanghui Liu +2
Pre-training followed by fine-tuning is widely adopted among practitioners. The performance can be improved by "model soups"~\cite{wortsman2022model} via exploring various hyperpar…
Towards Natural Machine Unlearning
Zhengbao He, Tao Li, Xinwen Cheng +2
Machine unlearning (MU) aims to eliminate information that has been learned from specific training data, namely forgetting data, from a pre-trained model. Currently, the mainstream…
Revisiting Random Weight Perturbation for Efficiently Improving Generalization
Tao Li, Qinghua Tao, Weihao Yan +5
Improving the generalization ability of modern deep neural networks (DNNs) is a fundamental challenge in machine learning. Two branches of methods have been proposed to seek flat m…
Friendly Sharpness-Aware Minimization
Tao Li, Pan Zhou, Zhengbao He +2
Sharpness-Aware Minimization (SAM) has been instrumental in improving deep neural network training by minimizing both training loss and loss sharpness. Despite the practical succes…
Online Continual Learning via Logit Adjusted Softmax
Zhehao Huang, Tao Li, Chenhe Yuan +2
Online continual learning is a challenging problem where models must learn from a non-stationary data stream while avoiding catastrophic forgetting. Inter-class imbalance during tr…
Investigating Catastrophic Overfitting in Fast Adversarial Training: A Self-fitting Perspective
Zhengbao He, Tao Li, Sizhe Chen +1
Although fast adversarial training provides an efficient approach for building robust networks, it may suffer from a serious problem known as catastrophic overfitting (CO), where m…
Multi-head Ensemble of Smoothed Classifiers for Certified Robustness
Kun Fang, Qinghua Tao, Yingwen Wu +3
Randomized Smoothing (RS) is a promising technique for certified robustness, and recently in RS the ensemble of multiple Deep Neural Networks (DNNs) has shown state-of-the-art perf…
Trainable Weight Averaging: Accelerating Training and Improving Generalization
Tao Li, Zhehao Huang, Yingwen Wu +4
Weight averaging is a widely used technique for accelerating training and improving the generalization of deep neural networks (DNNs). While existing approaches like stochastic wei…
Subspace Adversarial Training
Tao Li, Yingwen Wu, Sizhe Chen +2
Single-step adversarial training (AT) has received wide attention as it proved to be both efficient and robust. However, a serious problem of catastrophic overfitting exists, i.e.,…
Low Dimensional Landscape Hypothesis is True: DNNs can be Trained in Tiny Subspaces
Tao Li, Lei Tan, Qinghua Tao +2
Deep neural networks (DNNs) usually contain massive parameters, but there is redundancy such that it is guessed that the DNNs could be trained in low-dimensional subspaces. In this…