Convolutional Kernel Networks

Abstract

An important goal in visual recognition is to devise image representations that are invariant to particular transformations. In this paper, we address this goal with a new type of convolutional neural network (CNN) whose invariance is encoded by a reproducing kernel. Unlike traditional approaches where neural networks are learned either to represent data or for solving a classification task, our network learns to approximate the kernel feature map on training data. Such an approach enjoys several benefits over classical ones. First, by teaching CNNs to be invariant, we obtain simple network architectures that achieve a similar accuracy to more complex ones, while being easy to train and robust to overfitting. Second, we bridge a gap between the neural network literature and kernels, which are natural tools to model invariance. We evaluate our methodology on visual recognition tasks where CNNs have proven to perform well, e.g., digit recognition with the MNIST dataset, and the more challenging CIFAR-10 and STL-10 datasets, where our accuracy is competitive with the state of the art.

appears in Advances in Neural Information Processing Systems (NIPS), Dec 2014, Montreal, Canada, http://nips.cc

References in corpus (2)

• DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition
• Learning Invariant Representations with Local Transformations

Cited by in corpus (58)

• Bayesian Convolutional Neural Networks with Bernoulli Approximate Variational Inference
• Deep Clustering for Unsupervised Learning of Visual Features
• On Exact Computation with an Infinitely Wide Neural Net
• ReNet: A Recurrent Neural Network Based Alternative to Convolutional Networks
• Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivity
• Winner-Take-All Autoencoders
• Deep Neural Networks with Random Gaussian Weights: A Universal Classification Strategy?
• Scattering Networks for Hybrid Representation Learning
• Self-supervising Action Recognition by Statistical Moment and Subspace Descriptors
• Convolutional Sparse Kernel Network for Unsupervised Medical Image Analysis
• Neural Kernels Without Tangents
• Why Do Deep Residual Networks Generalize Better than Deep Feedforward Networks? -- A Neural Tangent Kernel Perspective
• PCONV: The Missing but Desirable Sparsity in DNN Weight Pruning for Real-time Execution on Mobile Devices
• Structured adaptive and random spinners for fast machine learning computations
• Time-Domain Audio Source Separation Based on Wave-U-Net Combined with Discrete Wavelet Transform
• A Unified Gradient Regularization Family for Adversarial Examples
• Delving Deeper into Anti-aliasing in ConvNets
• 1-Dimensional polynomial neural networks for audio signal related problems
• WaveCycleGAN2: Time-domain Neural Post-filter for Speech Waveform Generation
• Improving adversarial robustness of deep neural networks by using semantic information
• Unsupervised Feature Learning with K-means and An Ensemble of Deep Convolutional Neural Networks for Medical Image Classification
• Linear Time Sinkhorn Divergences using Positive Features
• On the Similarity between the Laplace and Neural Tangent Kernels
• Wavelet Integrated CNNs for Noise-Robust Image Classification
• Fisher-Bures Adversary Graph Convolutional Networks
• Learning with Group Invariant Features: A Kernel Perspective
• When Kernel Methods meet Feature Learning: Log-Covariance Network for Action Recognition from Skeletal Data
• A Statistical Investigation of Long Memory in Language and Music
• Faster Convergence in Deep-Predictive-Coding Networks to Learn Deeper Representations
• Dataset Meta-Learning from Kernel Ridge-Regression
• Lesion Net -- Skin Lesion Segmentation Using Coordinate Convolution and Deep Residual Units
• Approximation and Learning with Deep Convolutional Models: a Kernel Perspective
• A Spectral Analysis of Dot-product Kernels
• Correlated Weights in Infinite Limits of Deep Convolutional Neural Networks
• CNN-based Action Recognition and Supervised Domain Adaptation on 3D Body Skeletons via Kernel Feature Maps
• Adaptive Normalized Risk-Averting Training For Deep Neural Networks
• Tensor Representations via Kernel Linearization for Action Recognition from 3D Skeletons (Extended Version)
• Learning Representation for Mixed Data Types with a Nonlinear Deep Encoder-Decoder Framework
• Kernel-based Translations of Convolutional Networks
• Deep Global-Connected Net With The Generalized Multi-Piecewise ReLU Activation in Deep Learning
• The Unreasonable Effectiveness of Patches in Deep Convolutional Kernels Methods
• Higher-order Pooling of CNN Features via Kernel Linearization for Action Recognition
• A Neural Network Approach for Online Nonlinear Neyman-Pearson Classification
• Deep Context-Aware Kernel Networks
• End-to-end training of deep kernel map networks for image classification
• Deformation Robust Roto-Scale-Translation Equivariant CNNs
• Convolutional Motif Kernel Networks
• Structured Convolutional Kernel Networks for Airline Crew Scheduling
• Conditional Deep Gaussian Processes: multi-fidelity kernel learning
• Totally Deep Support Vector Machines
• Dictionary Learning and Sparse Coding for Third-order Super-symmetric Tensors
• Deep Neural-Kernel Machines
• Parametric machines: a fresh approach to architecture search
• Scalable Kernel Learning via the Discriminant Information
• Do place cells dream of conditional probabilities? Learning Neural Nyström representations
• Rapid training of deep neural networks without skip connections or normalization layers using Deep Kernel Shaping
• Recovery of surfaces and functions in high dimensions: sampling theory and links to neural networks
• IKA: Independent Kernel Approximator