PatchGame: Learning to Signal Mid-level Patches in Referential Games
arXiv:2111.01785
Abstract
We study a referential game (a type of signaling game) where two agents communicate with each other via a discrete bottleneck to achieve a common goal. In our referential game, the goal of the speaker is to compose a message or a symbolic representation of "important" image patches, while the task for the listener is to match the speaker's message to a different view of the same image. We show that it is indeed possible for the two agents to develop a communication protocol without explicit or implicit supervision. We further investigate the developed protocol and show the applications in speeding up recent Vision Transformers by using only important patches, and as pre-training for downstream recognition tasks (e.g., classification). Code available at https://github.com/kampta/PatchGame.
arXiv commentTo appear at NeurIPS 2021
References in corpus (22)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- Auto-Encoding Variational Bayes
- Bootstrap your own latent: A new approach to self-supervised Learning
- Categorical Reparameterization with Gumbel-Softmax
- Neural Discrete Representation Learning
- Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
- Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
- Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
- Zero-Shot Text-to-Image Generation
- Barlow Twins: Self-Supervised Learning via Redundancy Reduction
- Exploring Simple Siamese Representation Learning
- Sharp transition towards shared vocabularies in multi-agent systems
- Emergence of Grounded Compositional Language in Multi-Agent Populations
- Axiom-based Grad-CAM: Towards Accurate Visualization and Explanation of CNNs
- A Study of Face Obfuscation in ImageNet
- Emergent Translation in Multi-Agent Communication
- Learning to Play Guess Who? and Inventing a Grounded Language as a Consequence
- Generating Images with Sparse Representations
- Compositional properties of emergent languages in deep learning
- Avoiding hashing and encouraging visual semantics in referential emergent language games
- The emergence of visual semantics through communication games