189 citations · 1.7k across the 110 of their papers we have counts for
110 papers
Gradient Knowledge Distillation for Pre-trained Language Models
Lean Wang, Lei Li, Xu Sun
Knowledge distillation (KD) is an effective framework to transfer knowledge from a large-scale teacher to a compact yet well-performing student. Previous KD practices for pre-train…
From Mimicking to Integrating: Knowledge Integration for Pre-Trained Language Models
Lei Li, Yankai Lin, Xuancheng Ren +4
Investigating better ways to reuse the released pre-trained language models (PLMs) can significantly reduce the computational cost and the potential environmental side-effects. Thi…
Prophet Attention: Predicting Attention with Future Attention for Image Captioning
Fenglin Liu, Xuancheng Ren, Xian Wu +3
Recently, attention based models have been used extensively in many sequence-to-sequence learning systems. Especially for image captioning, the attention based models are expected…
Distributional Correlation--Aware Knowledge Distillation for Stock Trading Volume Prediction
Lei Li, Zhiyuan Zhang, Ruihan Bao +2
Traditional knowledge distillation in classification problems transfers the knowledge via class correlations in the soft label produced by teacher models, which are not available i…
Delving into the Openness of CLIP
Shuhuai Ren, Lei Li, Xuancheng Ren +2
Contrastive Language-Image Pre-training (CLIP) formulates image classification as an image-to-text matching task, i.e., matching images to the corresponding natural language descri…
Hierarchical Inductive Transfer for Continual Dialogue Learning
Shaoxiong Feng, Xuancheng Ren, Kan Li +1
Pre-trained models have achieved excellent performance on the dialogue task. However, for the continual increase of online chit-chat scenarios, directly fine-tuning these models fo…
Model Uncertainty-Aware Knowledge Amalgamation for Pre-Trained Language Models
Lei Li, Yankai Lin, Xuancheng Ren +4
As many fine-tuned pre-trained language models~(PLMs) with promising performance are generously released, investigating better ways to reuse these models is vital as it can greatly…
Auto-Encoding Knowledge Graph for Unsupervised Medical Report Generation
Fenglin Liu, Chenyu You, Xian Wu +3
Medical report generation, which aims to automatically generate a long and coherent report of a given medical image, has been receiving growing research interests. Existing approac…
RAP: Robustness-Aware Perturbations for Defending against Backdoor Attacks on NLP Models
Wenkai Yang, Yankai Lin, Peng Li +2
Backdoor attacks, which maliciously control a well-trained model's outputs of the instances with specific triggers, are recently shown to be serious threats to the safety of reusin…
Topology-Imbalance Learning for Semi-Supervised Node Classification
Deli Chen, Yankai Lin, Guangxiang Zhao +4
The class imbalance problem, as an important issue in learning node representations, has drawn increasing attention from the community. Although the imbalance considered by existin…
Well-classified Examples are Underestimated in Classification with Deep Neural Networks
Guangxiang Zhao, Wenkai Yang, Xuancheng Ren +3
The conventional wisdom behind learning deep classification models is to focus on bad-classified examples and ignore well-classified examples that are far from the decision boundar…
Dynamic Knowledge Distillation for Pre-trained Language Models
Lei Li, Yankai Lin, Shuhuai Ren +3
Knowledge distillation~(KD) has been proved effective for compressing large-scale pre-trained language models. However, existing methods conduct KD statically, e.g., the student mo…
Adversarial Parameter Defense by Multi-Step Risk Minimization
Zhiyuan Zhang, Ruixuan Luo, Xuancheng Ren +3
Previous studies demonstrate DNNs' vulnerability to adversarial examples and adversarial training can establish a defense to adversarial examples. In addition, recent studies show…
How to Inject Backdoors with Better Consistency: Logit Anchoring on Clean Data
Zhiyuan Zhang, Lingjuan Lyu, Weiqiang Wang +2
Since training a large-scale backdoored model from scratch requires a large training dataset, several recent attacks have considered to inject backdoors into a trained clean model…
Text AutoAugment: Learning Compositional Augmentation Policy for Text Classification
Shuhuai Ren, Jinchao Zhang, Lei Li +2
Data augmentation aims to enrich training samples for alleviating the overfitting issue in low-resource or class-imbalanced situations. Traditional methods first devise task-specif…
Long-term, Short-term and Sudden Event: Trading Volume Movement Prediction with Graph-based Multi-view Modeling
Liang Zhao, Wei Li, Ruihan Bao +3
Trading volume movement prediction is the key in a variety of financial applications. Despite its importance, there is few research on this topic because of its requirement for com…
ASAT: Adaptively Scaled Adversarial Training in Time Series
Zhiyuan Zhang, Wei Li, Ruihan Bao +3
Adversarial training is a method for enhancing neural networks to improve the robustness against adversarial examples. Besides the security concerns of potential adversarial exampl…
O2NA: An Object-Oriented Non-Autoregressive Approach for Controllable Video Captioning
Fenglin Liu, Xuancheng Ren, Xian Wu +4
Video captioning combines video understanding and language generation. Different from image captioning that describes a static image with details of almost every object, video capt…
Contrastive Attention for Automatic Chest X-ray Report Generation
Fenglin Liu, Changchang Yin, Xian Wu +4
Recently, chest X-ray report generation, which aims to automatically generate descriptions of given chest X-ray images, has received growing research interests. The key challenge o…
Learning Relation Alignment for Calibrated Cross-modal Retrieval
Shuhuai Ren, Junyang Lin, Guangxiang Zhao +5
Despite the achievements of large-scale multimodal pre-training approaches, cross-modal retrieval, e.g., image-text retrieval, remains a challenging task. To bridge the semantic ga…
Alleviating the Knowledge-Language Inconsistency: A Study for Deep Commonsense Knowledge
Yi Zhang, Lei Li, Yunfang Wu +2
Knowledge facts are typically represented by relational triples, while we observe that some commonsense facts are represented by the triples whose forms are inconsistent with the e…
Rethinking Skip Connection with Layer Normalization in Transformers and ResNets
Fenglin Liu, Xuancheng Ren, Zhiyuan Zhang +2
Skip connection, is a widely-used technique to improve the performance and the convergence of deep neural networks, which is believed to relieve the difficulty in optimization due…
Be Careful about Poisoned Word Embeddings: Exploring the Vulnerability of the Embedding Layers in NLP Models
Wenkai Yang, Lei Li, Zhiyuan Zhang +3
Recent studies have revealed a security threat to natural language processing (NLP) models, called the Backdoor Attack. Victim models can maintain competitive performance on clean…
Stability of Multi-Microgrids: New Certificates, Distributed Control, and Braess's Paradox
Amin Gholami, Xu Andy Sun
This paper investigates the theory of resilience and stability in multi-microgrid networks. We derive new sufficient conditions to guarantee small-signal stability of multi-microgr…