148 citations · 708 across the 141 of their papers we have counts for
28 papers · 1 filter
Exploring Sparsity in Graph Transformers
Chuang Liu, Yibing Zhan, Xueqi Ma +5
Graph Transformers (GTs) have achieved impressive results on various graph-related tasks. However, the huge computational cost of GTs hinders their deployment and application, espe…
Concrete Subspace Learning based Interference Elimination for Multi-task Model Fusion
Anke Tang, Xianglin Luo, Li Shen +5
Merging models fine-tuned from a common, extensively pre-trained large model but specialized for different tasks has been demonstrated as a cheap and scalable strategy to construct…
SpliceMix: A Cross-scale and Semantic Blending Augmentation Strategy for Multi-label Image Classification
Lei Wang, Yibing Zhan, Leilei Ma +3
Recently, Mix-style data augmentation methods (e.g., Mixup and CutMix) have shown promising performance in various visual tasks. However, these methods are primarily designed for s…
Merging Experts into One: Improving Computational Efficiency of Mixture of Experts
Shwai He, Run-Ze Fan, Liang Ding +3
Scaling the size of language models usually leads to remarkable advancements in NLP tasks. But it often comes with a price of growing computational cost. Although a sparse Mixture…
Zero-Shot Sharpness-Aware Quantization for Pre-trained Language Models
Miaoxi Zhu, Qihuang Zhong, Li Shen +4
Quantization is a promising approach for reducing memory overhead and accelerating inference, especially in large pre-trained language model (PLM) scenarios. While having no access…
Diversifying the Mixture-of-Experts Representation for Language Models with Orthogonal Optimizer
Boan Liu, Liang Ding, Li Shen +4
The Mixture of Experts (MoE) has emerged as a highly successful technique in deep learning, based on the principle of divide-and-conquer to maximize model capacity without signific…
Unlikelihood Tuning on Negative Samples Amazingly Improves Zero-Shot Translation
Changtong Zan, Liang Ding, Li Shen +4
Zero-shot translation (ZST), which is generally based on a multilingual neural machine translation model, aims to translate between unseen language pairs in training data. The comm…
Deep Model Fusion: A Survey
Weishi Li, Yong Peng, Miao Zhang +3
Deep model fusion/merging is an emerging technique that merges the parameters or predictions of multiple deep learning models into a single one. It combines the abilities of differ…
MerA: Merging Pretrained Adapters For Few-Shot Learning
Shwai He, Run-Ze Fan, Liang Ding +3
Adapter tuning, which updates only a few parameters, has become a mainstream method for fine-tuning pretrained language models to downstream tasks. However, it often yields subpar…
Can Linguistic Knowledge Improve Multimodal Alignment in Vision-Language Pretraining?
Fei Wang, Liang Ding, Jun Rao +3
The multimedia community has shown a significant interest in perceiving and representing the physical world with multimodal pretrained neural network models, and among them, the vi…
Recursively Summarizing Enables Long-Term Dialogue Memory in Large Language Models
Qingyue Wang, Yanhe Fu, Yanan Cao +3
Recently, large language models (LLMs), such as GPT-4, stand out remarkable conversational abilities, enabling them to engage in dynamic and contextually relevant dialogues across…
Efficient Federated Learning via Local Adaptive Amended Optimizer with Linear Speedup
Yan Sun, Li Shen, Hao Sun +2
Adaptive optimization has achieved notable success for distributed learning while extending adaptive optimizer to federated Learning (FL) suffers from severe inefficiency, includin…
Free-Form Composition Networks for Egocentric Action Recognition
Haoran Wang, Qinghua Cheng, Baosheng Yu +4
Egocentric action recognition is gaining significant attention in the field of human action recognition. In this paper, we address data scarcity issue in egocentric action recognit…
Unsupervised Dense Retrieval with Relevance-Aware Contrastive Pre-Training
Yibin Lei, Liang Ding, Yu Cao +3
Dense retrievers have achieved impressive performance, but their demand for abundant training data limits their application scenarios. Contrastive pre-training, which constructs ps…
Divide, Conquer, and Combine: Mixture of Semantic-Independent Experts for Zero-Shot Dialogue State Tracking
Qingyue Wang, Liang Ding, Yanan Cao +5
Zero-shot transfer learning for Dialogue State Tracking (DST) helps to handle a variety of task-oriented dialogue domains without the cost of collecting in-domain data. Existing wo…
Self-Evolution Learning for Discriminative Language Model Pretraining
Qihuang Zhong, Liang Ding, Juhua Liu +2
Masked language modeling, widely used in discriminative language model (e.g., BERT) pretraining, commonly adopts a random masking strategy. However, random masking does not conside…
Revisiting Token Dropping Strategy in Efficient BERT Pretraining
Qihuang Zhong, Liang Ding, Juhua Liu +4
Token dropping is a recently-proposed strategy to speed up the pretraining of masked language models, such as BERT, by skipping the computation of a subset of the input tokens at s…
Self-Evolution Learning for Mixup: Enhance Data Augmentation on Few-Shot Text Classification Tasks
Haoqi Zheng, Qihuang Zhong, Liang Ding +4
Text classification tasks often encounter few shot scenarios with limited labeled data, and addressing data scarcity is crucial. Data augmentation with mixup has shown to be effect…
Dynamic Regularized Sharpness Aware Minimization in Federated Learning: Approaching Global Consistency and Smooth Landscape
Yan Sun, Li Shen, Shixiang Chen +2
In federated learning (FL), a cluster of local clients are chaired under the coordination of the global server and cooperatively train one model with privacy protection. Due to the…
Prompt-Learning for Cross-Lingual Relation Extraction
Chiaming Hsu, Changtong Zan, Liang Ding +5
Relation Extraction (RE) is a crucial task in Information Extraction, which entails predicting relationships between entities within a given sentence. However, extending pre-traine…
On Efficient Training of Large-Scale Deep Learning Models: A Literature Review
Li Shen, Yan Sun, Zhiyuan Yu +3
The field of deep learning has witnessed significant progress, particularly in computer vision (CV), natural language processing (NLP), and speech. The use of large-scale models tr…
Can ChatGPT Understand Too? A Comparative Study on ChatGPT and Fine-tuned BERT
Qihuang Zhong, Liang Ding, Juhua Liu +2
Recently, ChatGPT has attracted great attention, as it can generate fluent and high-quality responses to human inquiries. Several prior studies have shown that ChatGPT attains rema…
Error Analysis Prompting Enables Human-Like Translation Evaluation in Large Language Models
Qingyu Lu, Baopu Qiu, Liang Ding +3
Generative large language models (LLMs), e.g., ChatGPT, have demonstrated remarkable proficiency across several NLP tasks, such as machine translation, text summarization. Recent r…
Towards Making the Most of ChatGPT for Machine Translation
Keqin Peng, Liang Ding, Qihuang Zhong +5
ChatGPT shows remarkable capabilities for machine translation (MT). Several prior studies have shown that it achieves comparable results to commercial systems for high-resource lan…