32 citations · 99 across the 23 of their papers we have counts for
24 papers
First-Order LoRA Training with Phone NPUs
Xu Zheng, Wei Cheng, Zhuomin Chen +4
On-device fine-tuning lets a large language model (LLM) learn from its user's private data without uploading it. Low-rank adaptation (LoRA) reduces the trainable parameters, but ea…
Adaptive Draft-Verification for Efficient Large Language Model Decoding
Xukun Liu, Bowen Lei, Ruqi Zhang +1
Large language model (LLM) decoding involves generating a sequence of tokens based on a given context, where each token is predicted one at a time using the model's learned probabi…
Embracing Unknown Step by Step: Towards Reliable Sparse Training in Real World
Bowen Lei, Dongkuan Xu, Ruqi Zhang +1
Sparse training has emerged as a promising method for resource-efficient deep neural networks (DNNs) in real-world applications. However, the reliability of sparse models remains a…
Gentopia: A Collaborative Platform for Tool-Augmented LLMs
Binfeng Xu, Xukun Liu, Hua Shen +7
Augmented Language Models (ALMs) empower large language models with the ability to use tools, transforming them into intelligent agents for real-world interactions. However, most e…
Towards Personalized Federated Learning via Heterogeneous Model Reassembly
Jiaqi Wang, Xingyi Yang, Suhan Cui +4
This paper focuses on addressing the practical yet challenging problem of model heterogeneity in federated learning, where clients possess models with different network structures.…
Towards Reliable Rare Category Analysis on Graphs via Individual Calibration
Longfeng Wu, Bowen Lei, Dongkuan Xu +1
Rare categories abound in a number of real-world networks and play a pivotal role in a variety of high-stakes applications, including financial fraud detection, network intrusion d…
Rethinking Data Distillation: Do Not Overlook Calibration
Dongyao Zhu, Bowen Lei, Jie Zhang +4
Neural networks trained on distilled data often produce over-confident output and require correction by calibration methods. Existing calibration methods such as temperature scalin…
AutoST: Training-free Neural Architecture Search for Spiking Transformers
Ziqing Wang, Qidong Zhao, Jinku Cui +2
Spiking Transformers have gained considerable attention because they achieve both the energy efficiency of Spiking Neural Networks (SNNs) and the high capacity of Transformers. How…
ReWOO: Decoupling Reasoning from Observations for Efficient Augmented Language Models
Binfeng Xu, Zhiyuan Peng, Bowen Lei +3
Augmented Language Models (ALMs) blend the reasoning capabilities of Large Language Models (LLMs) with tools that allow for knowledge retrieval and action execution. Existing ALM s…
Neurogenesis Dynamics-inspired Spiking Neural Network Training Acceleration
Shaoyi Huang, Haowen Fang, Kaleel Mahmood +7
Biologically inspired Spiking Neural Networks (SNNs) have attracted significant attention for their ability to provide extremely energy-efficient machine intelligence through event…
Time Series Contrastive Learning with Information-Aware Augmentations
Dongsheng Luo, Wei Cheng, Yingheng Wang +8
Various contrastive learning approaches have been proposed in recent years and achieve significant empirical success. While effective and prevalent, contrastive learning has been l…
Calibrating the Rigged Lottery: Making All Tickets Reliable
Bowen Lei, Ruqi Zhang, Dongkuan Xu +1
Although sparse training has been successfully used in various resource-limited deep learning tasks to save memory, accelerate training, and reduce inference time, the reliability…
Efficient Informed Proposals for Discrete Distributions via Newton's Series Approximation
Yue Xiang, Dongyao Zhu, Bowen Lei +2
Gradients have been exploited in proposal distributions to accelerate the convergence of Markov chain Monte Carlo algorithms on discrete distributions. However, these methods requi…
Balance is Essence: Accelerating Sparse Training via Adaptive Gradient Correction
Bowen Lei, Dongkuan Xu, Ruqi Zhang +2
Despite impressive performance, deep neural networks require significant memory and computation costs, prohibiting their application in resource-constrained scenarios. Sparse train…
Accelerating Dataset Distillation via Model Augmentation
Lei Zhang, Jie Zhang, Bowen Lei +6
Dataset Distillation (DD), a newly emerging field, aims at generating much smaller but efficient synthetic training datasets from large ones. Existing DD methods based on gradient…
A Survey for Efficient Open Domain Question Answering
Qin Zhang, Shangsi Chen, Dongkuan Xu +4
Open domain question answering (ODQA) is a longstanding task aimed at answering factual questions from a large knowledge corpus without any explicit evidence in natural language pr…
Dynamic Sparse Training via Balancing the Exploration-Exploitation Trade-off
Shaoyi Huang, Bowen Lei, Dongkuan Xu +4
Over-parameterization of deep neural networks (DNNs) has shown high prediction accuracy for many applications. Although effective, the large number of parameters hinders its popula…
You Need Multiple Exiting: Dynamic Early Exiting for Accelerating Unified Vision Language Model
Shengkun Tang, Yaqing Wang, Zhenglun Kong +6
Large-scale Transformer models bring significant improvements for various downstream vision language tasks with a unified architecture. The performance improvements come with incre…
S4: a High-sparsity, High-performance AI Accelerator
Ian En-Hsu Yen, Zhibin Xiao, Dongkuan Xu
Exploiting sparsity underlying neural networks has become one of the most potential methodologies to reduce the memory footprint, I/O cost, and computation workloads during inferen…
An Automatic and Efficient BERT Pruning for Edge AI Systems
Shaoyi Huang, Ning Liu, Yueying Liang +5
With the yearning for deep learning democratization, there are increasing demands to implement Transformer-based natural language processing (NLP) models on resource-constrained de…
AutoDistil: Few-shot Task-agnostic Neural Architecture Search for Distilling Large Language Models
Dongkuan Xu, Subhabrata Mukherjee, Xiaodong Liu +5
Knowledge distillation (KD) methods compress large models into smaller students with manually-designed student architectures given pre-specified computational cost. This requires s…
InfoGCL: Information-Aware Graph Contrastive Learning
Dongkuan Xu, Wei Cheng, Dongsheng Luo +2
Various graph contrastive learning models have been proposed to improve the performance of learning tasks on graph datasets in recent years. While effective and prevalent, these mo…
Sparse Progressive Distillation: Resolving Overfitting under Pretrain-and-Finetune Paradigm
Shaoyi Huang, Dongkuan Xu, Ian E. H. Yen +8
Conventional wisdom in pruning Transformer-based language models is that pruning reduces the model expressiveness and thus is more likely to underfit rather than overfit. However,…
Rethinking Network Pruning -- under the Pre-train and Fine-tune Paradigm
Dongkuan Xu, Ian E. H. Yen, Jinxi Zhao +1
Transformer-based pre-trained language models have significantly improved the performance of various natural language processing (NLP) tasks in the recent years. While effective an…