Reconciling the Discrete-Continuous Divide: Towards a Mathematical Theory of Sparse Communication
arXiv:2104.00755
Abstract
Neural networks and other machine learning models compute continuous representations, while humans communicate with discrete symbols. Reconciling these two forms of communication is desirable to generate human-readable interpretations or to learn discrete latent variable models, while maintaining end-to-end differentiability. Some existing approaches (such as the Gumbel-softmax transformation) build continuous relaxations that are discrete approximations in the zero-temperature limit, while others (such as sparsemax transformations and the hard concrete distribution) produce discrete/continuous hybrids. In this paper, we build rigorous theoretical foundations for these hybrids. Our starting point is a new "direct sum" base measure defined on the face lattice of the probability simplex. From this measure, we introduce a new entropy function that includes the discrete and differential entropies as particular cases, and has an interpretation in terms of code optimality, as well as two other information-theoretic counterparts that generalize the mutual information and Kullback-Leibler divergences. Finally, we introduce "mixed languages" as strings of hybrid symbols and a new mixed weighted finite state automaton that recognizes a class of regular mixed languages, generalizing closure properties of regular languages.
References in corpus (13)
- Categorical Reparameterization with Gumbel-Softmax
- Towards A Rigorous Science of Interpretable Machine Learning
- Spike and slab variable selection: Frequentist and Bayesian strategies
- End-To-End Memory Networks
- Neural Variational Inference and Learning in Belief Networks
- Learning to Transduce with Unbounded Memory
- Programming with a Differentiable Forth Interpreter
- Recurrent Independent Mechanisms
- Dimensions of Neural-symbolic Integration - A Structured Survey
- Learning with Differentiable Perturbed Optimizers
- Differentiable Perturb-and-Parse: Semi-Supervised Parsing with a Structured Variational Autoencoder
- Emergent Multi-Agent Communication in the Deep Learning Era
- Coordination Among Neural Modules Through a Shared Global Workspace