195 citations · 382 across the 14 of their papers we have counts for
14 papers
RecycleGPT: An Autoregressive Language Model with Recyclable Module
Yufan Jiang, Qiaozhi He, Xiaomin Zhuang +4
Existing large language models have to run K times to generate a sequence of K tokens. In this paper, we present RecycleGPT, a generative language model with fast decoding speed by…
TA-MoE: Topology-Aware Large Scale Mixture-of-Expert Training
Chang Chen, Min Li, Zhihua Wu +2
Sparsely gated Mixture-of-Expert (MoE) has demonstrated its effectiveness in scaling up deep neural networks to an extreme scale. Despite that numerous efforts have been made to im…
Efficient AlphaFold2 Training using Parallel Evoformer and Branch Parallelism
Guoxia Wang, Zhihua Wu, Xiaomin Fang +4
The accuracy of AlphaFold2, a frontier end-to-end structure prediction system, is already close to that of the experimental determination techniques. Due to the complex model archi…
Boosting Distributed Training Performance of the Unpadded BERT Model
Jinle Zeng, Min Li, Zhihua Wu +4
Pre-training models are an important tool in Natural Language Processing (NLP), while the BERT model is a classic pre-training model whose structure has been widely adopted by foll…
HelixFold: An Efficient Implementation of AlphaFold2 using PaddlePaddle
Guoxia Wang, Xiaomin Fang, Zhihua Wu +6
Accurate protein structure prediction can significantly accelerate the development of life science. The accuracy of AlphaFold2, a frontier end-to-end structure prediction system, i…
ERNIE-ViLG: Unified Generative Pre-training for Bidirectional Vision-Language Generation
Han Zhang, Weichong Yin, Yewei Fang +7
Conventional methods for the image-text generation tasks mainly tackle the naturally bidirectional generation tasks separately, focusing on designing task-specific frameworks to im…
ERNIE 3.0 Titan: Exploring Larger-scale Knowledge Enhanced Pre-training for Language Understanding and Generation
Shuohuan Wang, Yu Sun, Yang Xiang +26
Pre-trained language models have achieved state-of-the-art results in various Natural Language Processing (NLP) tasks. GPT-3 has shown that scaling up pre-trained language models c…
End-to-end Adaptive Distributed Training on PaddlePaddle
Yulong Ao, Zhihua Wu, Dianhai Yu +10
Distributed training has become a pervasive and effective approach for training a large neural network (NN) model with processing massive data. However, it is very challenging to s…
HeterPS: Distributed Deep Learning With Reinforcement Learning Based Scheduling in Heterogeneous Environments
Ji Liu, Zhihua Wu, Dianhai Yu +6
Deep neural networks (DNNs) exploit many layers and a large number of parameters to achieve excellent performance. The training process of DNN models generally handles large-scale…
PLATO-XL: Exploring the Large-scale Pre-training of Dialogue Generation
Siqi Bao, Huang He, Fan Wang +11
To explore the limit of dialogue generation pre-training, we present the models of PLATO-XL with up to 11 billion parameters, trained on both Chinese and English social media conve…
ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generation
Yu Sun, Shuohuan Wang, Shikun Feng +19
Pre-trained models have achieved state-of-the-art results in various Natural Language Processing (NLP) tasks. Recent works such as T5 and GPT-3 have shown that scaling up pre-train…
ASCNet: Self-supervised Video Representation Learning with Appearance-Speed Consistency
Deng Huang, Wenhao Wu, Weiwen Hu +6
We study self-supervised video representation learning, which is a challenging task due to 1) lack of labels for explicit supervision; 2) unstructured and noisy visual information.…
Secure Federated Submodel Learning
Chaoyue Niu, Fan Wu, Shaojie Tang +5
Federated learning was proposed with an intriguing vision of achieving collaborative machine learning among numerous clients without uploading their private data to a cloud server.…
From Server-Based to Client-Based Machine Learning: A Comprehensive Survey
Renjie Gu, Chaoyue Niu, Fan Wu +4
In recent years, mobile devices have gained increasing development with stronger computation capability and larger storage space. Some of the computation-intensive machine learning…