Emergent Properties of Foveated Perceptual Systems

Abstract

The goal of this work is to characterize the representational impact that foveation operations have for machine vision systems, inspired by the foveated human visual system, which has higher acuity at the center of gaze and texture-like encoding in the periphery. To do so, we introduce models consisting of a first-stage \textit{fixed} image transform followed by a second-stage \textit{learnable} convolutional neural network, and we varied the first stage component. The primary model has a foveated-textural input stage, which we compare to a model with foveated-blurred input and a model with spatially-uniform blurred input (both matched for perceptual compression), and a final reference model with minimal input-based compression. We find that: 1) the foveated-texture model shows similar scene classification accuracy as the reference model despite its compressed input, with greater i.i.d. generalization than the other models; 2) the foveated-texture model has greater sensitivity to high-spatial frequency information and greater robustness to occlusion, w.r.t the comparison models; 3) both the foveated systems, show a stronger center image-bias relative to the spatially-uniform systems even with a weight sharing constraint. Critically, these results are preserved over different classical CNN architectures throughout their learning dynamics. Altogether, this suggests that foveation with peripheral texture-based computations yields an efficient, distinct, and robust representational format of scene information, and provides symbiotic computational insight into the representational consequences that texture-based peripheral encoding may have for processing in the human visual system, while also potentially inspiring the next generation of computer vision models via spatially-adaptive computation. Code + Data available here: https://github.com/ArturoDeza/EmergentProperties

An updated pre-print. Currently under review at NeurIPS 2021. Themes: Foveation, Machine Perception & Representation Learning

References in corpus (8)

• Shortcut Learning in Deep Neural Networks
• Object Detectors Emerge in Deep Scene CNNs
• Crowding Reveals Fundamental Differences in Local vs. Global Processing in Humans and Machines
• DADA: Differentiable Automatic Data Augmentation
• Computational role of eccentricity dependent cortical magnification
• SideEye: A Generative Neural Network Based Simulator of Human Peripheral Vision
• Can Peripheral Representations Improve Clutter Metrics on Complex Scenes?
• Self-Supervised Learning of a Biologically-Inspired Visual Texture Model

Cited by in corpus (7)

• On the surprising similarities between supervised and self-supervised models
• Hierarchically Compositional Tasks and Deep Convolutional Networks
• Passive Attention in Artificial Neural Networks Predicts Human Visual Selectivity
• CUDA-Optimized real-time rendering of a Foveated Visual System
• FoveaTer: Foveated Transformer for Image Classification
• The Effects of Image Distribution and Task on Adversarial Robustness
• Unsupervised Foveal Vision Neural Networks with Top-Down Attention