CARLS: Cross-platform Asynchronous Representation Learning System

Abstract

In this work, we propose CARLS, a novel framework for augmenting the capacity of existing deep learning frameworks by enabling multiple components -- model trainers, knowledge makers and knowledge banks -- to concertedly work together in an asynchronous fashion across hardware platforms. The proposed CARLS is particularly suitable for learning paradigms where model training benefits from additional knowledge inferred or discovered during training, such as node embeddings for graph neural networks or reliable pseudo labels from model predictions. We also describe three learning paradigms -- semi-supervised learning, curriculum learning and multimodal learning -- as examples that can be scaled up efficiently by CARLS. One version of CARLS has been open-sourced and available for download at: https://github.com/tensorflow/neural-structured-learning/tree/master/research/carls

References in corpus (7)

• Distilling the Knowledge in a Neural Network
• Semi-Supervised Classification with Graph Convolutional Networks
• Language Models are Few-Shot Learners
• GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
• One Model To Learn Them All
• Extreme Classification in Log Memory using Count-Min Sketch: A Case Study of Amazon Search with 50M Products
• DynamicEmbedding: Extending TensorFlow for Colossal-Scale Applications