148 citations · 509 across the 46 of their papers we have counts for
46 papers
MPipeMoE: Memory Efficient MoE for Pre-trained Models with Adaptive Pipeline Parallelism
Zheng Zhang, Donglin Yang, Yaqi Xia +4
Recently, Mixture-of-Experts (MoE) has become one of the most popular techniques to scale pre-trained models to extraordinarily large sizes. Dynamic activation of experts allows fo…
Unlikelihood Tuning on Negative Samples Amazingly Improves Zero-Shot Translation
Changtong Zan, Liang Ding, Li Shen +4
Zero-shot translation (ZST), which is generally based on a multilingual neural machine translation model, aims to translate between unseen language pairs in training data. The comm…
MerA: Merging Pretrained Adapters For Few-Shot Learning
Shwai He, Run-Ze Fan, Liang Ding +3
Adapter tuning, which updates only a few parameters, has become a mainstream method for fine-tuning pretrained language models to downstream tasks. However, it often yields subpar…
Neural Collapse Terminus: A Unified Solution for Class Incremental Learning and Its Variants
Yibo Yang, Haobo Yuan, Xiangtai Li +6
How to enable learnability for new classes while keeping the capability well on old classes has been a crucial challenge for class incremental learning. Beyond the normal case, lon…
Recursively Summarizing Enables Long-Term Dialogue Memory in Large Language Models
Qingyue Wang, Yanhe Fu, Yanan Cao +3
Recently, large language models (LLMs), such as GPT-4, stand out remarkable conversational abilities, enabling them to engage in dynamic and contextually relevant dialogues across…
PNT-Edge: Towards Robust Edge Detection with Noisy Labels by Learning Pixel-level Noise Transitions
Wenjie Xuan, Shanshan Zhao, Yu Yao +5
Relying on large-scale training data with pixel-level labels, previous edge detection methods have achieved high performance. However, it is hard to manually label edges accurately…
Unsupervised Dense Retrieval with Relevance-Aware Contrastive Pre-Training
Yibin Lei, Liang Ding, Yu Cao +3
Dense retrievers have achieved impressive performance, but their demand for abundant training data limits their application scenarios. Contrastive pre-training, which constructs ps…
Divide, Conquer, and Combine: Mixture of Semantic-Independent Experts for Zero-Shot Dialogue State Tracking
Qingyue Wang, Liang Ding, Yanan Cao +5
Zero-shot transfer learning for Dialogue State Tracking (DST) helps to handle a variety of task-oriented dialogue domains without the cost of collecting in-domain data. Existing wo…
Self-Evolution Learning for Discriminative Language Model Pretraining
Qihuang Zhong, Liang Ding, Juhua Liu +2
Masked language modeling, widely used in discriminative language model (e.g., BERT) pretraining, commonly adopts a random masking strategy. However, random masking does not conside…
Revisiting Token Dropping Strategy in Efficient BERT Pretraining
Qihuang Zhong, Liang Ding, Juhua Liu +4
Token dropping is a recently-proposed strategy to speed up the pretraining of masked language models, such as BERT, by skipping the computation of a subset of the input tokens at s…
VanillaNet: the Power of Minimalism in Deep Learning
Hanting Chen, Yunhe Wang, Jianyuan Guo +1
At the heart of foundation models is the philosophy of "more is different", exemplified by the astonishing success in computer vision and natural language processing. However, the…
Self-Evolution Learning for Mixup: Enhance Data Augmentation on Few-Shot Text Classification Tasks
Haoqi Zheng, Qihuang Zhong, Liang Ding +4
Text classification tasks often encounter few shot scenarios with limited labeled data, and addressing data scarcity is crucial. Data augmentation with mixup has shown to be effect…
Upcycling Models under Domain and Category Shift
Sanqing Qu, Tianpei Zou, Florian Roehrbein +4
Deep neural networks (DNNs) often perform poorly in the presence of domain shift and category shift. How to upcycle DNNs and adapt them to the target task remains an important open…
Can ChatGPT Understand Too? A Comparative Study on ChatGPT and Fine-tuned BERT
Qihuang Zhong, Liang Ding, Juhua Liu +2
Recently, ChatGPT has attracted great attention, as it can generate fluent and high-quality responses to human inquiries. Several prior studies have shown that ChatGPT attains rema…
Error Analysis Prompting Enables Human-Like Translation Evaluation in Large Language Models
Qingyu Lu, Baopu Qiu, Liang Ding +3
Generative large language models (LLMs), e.g., ChatGPT, have demonstrated remarkable proficiency across several NLP tasks, such as machine translation, text summarization. Recent r…
Bag of Tricks for Effective Language Model Pretraining and Downstream Adaptation: A Case Study on GLUE
Qihuang Zhong, Liang Ding, Keqin Peng +5
This technical report briefly describes our JDExplore d-team's submission Vega v1 on the General Language Understanding Evaluation (GLUE) leaderboard, where GLUE is a collection of…
Pseudo Contrastive Learning for Graph-based Semi-supervised Learning
Weigang Lu, Ziyu Guan, Wei Zhao +5
Pseudo Labeling is a technique used to improve the performance of semi-supervised Graph Neural Networks (GNNs) by generating additional pseudo-labels based on confident predictions…
A Comprehensive Survey of Dataset Distillation
Shiye Lei, Dacheng Tao
Deep learning technology has developed unprecedentedly in the last decade and has become the primary choice in many application domains. This progress is mainly attributed to a sys…
Original or Translated? On the Use of Parallel Data for Translation Quality Estimation
Baopu Qiu, Liang Ding, Di Wu +3
Machine Translation Quality Estimation (QE) is the task of evaluating translation output in the absence of human-written references. Due to the scarcity of human-labeled QE data, p…
Toward Human-Like Evaluation for Natural Language Generation with Error Analysis
Qingyu Lu, Liang Ding, Liping Xie +3
The state-of-the-art language model-based automatic metrics, e.g. BARTScore, benefiting from large-scale contextualized pre-training, have been successfully used in a wide range of…
ContraFeat: Contrasting Deep Features for Semantic Discovery
Xinqi Zhu, Chang Xu, Dacheng Tao
StyleGAN has shown strong potential for disentangled semantic control, thanks to its special design of multi-layer intermediate latent variables. However, existing semantic discove…
Toward Efficient Language Model Pretraining and Downstream Adaptation via Self-Evolution: A Case Study on SuperGLUE
Qihuang Zhong, Liang Ding, Yibing Zhan +11
This technical report briefly describes our JDExplore d-team's Vega v2 submission on the SuperGLUE leaderboard. SuperGLUE is more challenging than the widely used general language…
Improving Simultaneous Machine Translation with Monolingual Data
Hexuan Deng, Liang Ding, Xuebo Liu +3
Simultaneous machine translation (SiMT) is usually done via sequence-level knowledge distillation (Seq-KD) from a full-sentence neural machine translation (NMT) model. However, the…
From Images to Textual Prompts: Zero-shot VQA with Frozen Large Language Models
Jiaxian Guo, Junnan Li, Dongxu Li +4
Large language models (LLMs) have demonstrated excellent zero-shot generalization to new language tasks. However, effective utilization of LLMs for zero-shot visual question-answer…