Differentiable Sorting Networks for Scalable Sorting and Ranking Supervision
arXiv:2105.04019
Abstract
Sorting and ranking supervision is a method for training neural networks end-to-end based on ordering constraints. That is, the ground truth order of sets of samples is known, while their absolute values remain unsupervised. For that, we propose differentiable sorting networks by relaxing their pairwise conditional swap operations. To address the problems of vanishing gradients and extensive blurring that arise with larger numbers of layers, we propose mapping activations to regions with moderate gradients. We consider odd-even as well as bitonic sorting networks, which outperform existing relaxations of the sorting operation. We show that bitonic sorting networks can achieve stable training on large input sets of up to 1024 elements.
arXiv commentPublished at ICML 2021, Code @ https://github.com/Felix-Petersen/diffsort, Video @ https://www.youtube.com/watch?v=38dvqdYEs1o
References in corpus (6)
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- Multi-digit Number Recognition from Street View Imagery using Deep Convolutional Neural Networks
- Learning Latent Permutations with Gumbel-Sinkhorn Networks
- Ranking via Sinkhorn Propagation
- Stochastic Optimization of Sorting Networks via Continuous Relaxations