Deep Polynomial Neural Networks

Abstract

Deep Convolutional Neural Networks (DCNNs) are currently the method of choice both for generative, as well as for discriminative learning in computer vision and machine learning. The success of DCNNs can be attributed to the careful selection of their building blocks (e.g., residual blocks, rectifiers, sophisticated normalization schemes, to mention but a few). In this paper, we propose $\Pi$-Nets, a new class of function approximators based on polynomial expansions. $\Pi$-Nets are polynomial neural networks, i.e., the output is a high-order polynomial of the input. The unknown parameters, which are naturally represented by high-order tensors, are estimated through a collective tensor factorization with factors sharing. We introduce three tensor decompositions that significantly reduce the number of parameters and show how they can be efficiently implemented by hierarchical neural networks. We empirically demonstrate that $\Pi$-Nets are very expressive and they even produce good results without the use of non-linear activation functions in a large battery of tasks and signals, i.e., images, graphs, and audio. When used in conjunction with activation functions, $\Pi$-Nets produce state-of-the-art results in three challenging tasks, i.e. image generation, face verification and 3D mesh representation learning. The source code is available at \url{https://github.com/grigorisg9gr/polynomial_nets}.

Published in IEEE Transactions on Pattern Analysis and Machine Intelligence (T-PAMI). Code: https://github.com/grigorisg9gr/polynomial_nets. arXiv admin note: substantial text overlap with arXiv:2003.03828

References in corpus (33)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Deep Learning in Neural Networks: An Overview
• Neural Machine Translation by Jointly Learning to Align and Translate
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Spectral Normalization for Generative Adversarial Networks
• Instance Normalization: The Missing Ingredient for Fast Stylization
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Squeeze-and-Excitation Networks
• MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems
• Wide Residual Networks
• Tensor Decomposition for Signal Processing and Machine Learning
• On the Convergence of Adam and Beyond
• The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
• Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition
• Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
• Searching for Activation Functions
• Predicting Parameters in Deep Learning
• Highway Networks
• RetinaFace: Single-stage Dense Face Localisation in the Wild
• Residual Networks of Residual Networks: Multilevel Residual Networks
• Cross-Age LFW: A Database for Studying Cross-Age Face Recognition in Unconstrained Environments
• CirCNN: Accelerating and Compressing Deep Neural Networks Using Block-CirculantWeight Matrices
• On the Expressive Power of Deep Learning: A Tensor Analysis
• A Convergence Analysis of Gradient Descent for Deep Linear Neural Networks
• Neural 3D Morphable Models: Spiral Convolutional Networks for 3D Shape Representation Learning and Generation
• On the Expressive Power of Deep Polynomial Neural Networks
• Class-Splitting Generative Adversarial Networks
• Sharing Residual Units Through Collective Tensor Factorization in Deep Neural Networks
• PolyGAN: High-Order Polynomial Generators

Cited by in corpus (7)

• Tensor Methods in Computer Vision and Deep Learning
• Interpretable Polynomial Neural Ordinary Differential Equations
• NN2Poly: A polynomial representation for deep feed-forward artificial neural networks
• flexgrid2vec: Learning Efficient Visual Representations Vectors
• Quadratic Neuron-empowered Heterogeneous Autoencoder for Unsupervised Anomaly Detection
• On Expressivity and Trainability of Quadratic Networks
• Kineto-Dynamical Planning and Accurate Execution of Minimum-Time Maneuvers on Three-Dimensional Circuits