Unsupervised Representation Learning for Binary Networks by Joint Classifier Learning

Abstract

Self-supervised learning is a promising unsupervised learning framework that has achieved success with large floating point networks. But such networks are not readily deployable to edge devices. To accelerate deployment of models with the benefit of unsupervised representation learning to such resource limited devices for various downstream tasks, we propose a self-supervised learning method for binary networks that uses a moving target network. In particular, we propose to jointly train a randomly initialized classifier, attached to a pretrained floating point feature extractor, with a binary network. Additionally, we propose a feature similarity loss, a dynamic loss balancing and modified multi-stage training to further improve the accuracy, and call our method BURN. Our empirical validations over five downstream tasks using seven datasets show that BURN outperforms self-supervised baselines for binary networks and sometimes outperforms supervised pretraining. Code is availabe at https://github.com/naver-ai/burn.

to appear in CVPR2022

References in corpus (9)

• Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
• What Makes for Good Views for Contrastive Learning?
• Large Batch Training of Convolutional Networks
• Self-supervised Pretraining of Visual Features in the Wild
• Why gradient clipping accelerates training: A theoretical justification for adaptivity
• Understanding self-supervised Learning Dynamics without Contrastive Pairs
• Training Binary Neural Networks through Learning with Noisy Supervision
• BinaryDuo: Reducing Gradient Mismatch in Binary Activation Network by Coupling Binary Activations
• CO2: Consistent Contrast for Unsupervised Visual Representation Learning