Deep Neural Networks using a Single Neuron: Folded-in-Time Architecture using Feedback-Modulated Delay Loops

Abstract

Deep neural networks are among the most widely applied machine learning tools showing outstanding performance in a broad range of tasks. We present a method for folding a deep neural network of arbitrary size into a single neuron with multiple time-delayed feedback loops. This single-neuron deep neural network comprises only a single nonlinearity and appropriately adjusted modulations of the feedback signals. The network states emerge in time as a temporal unfolding of the neuron's dynamics. By adjusting the feedback-modulation within the loops, we adapt the network's connection weights. These connection weights are determined via a back-propagation algorithm, where both the delay-induced and local network connections must be taken into account. Our approach can fully represent standard Deep Neural Networks (DNN), encompasses sparse DNNs, and extends the DNN concept toward dynamical systems implementations. The new method, which we call Folded-in-time DNN (Fit-DNN), exhibits promising performance in a set of benchmark tasks.

References in corpus (14)

• Deep Learning in Neural Networks: An Overview
• Recent Advances in Physical Reservoir Computing: A Review
• Optoelectronic Reservoir Computing
• Human-level performance in first-person multiplayer games with population-based deep reinforcement learning
• Scalable Training of Artificial Neural Networks with Adaptive Sparse Connectivity inspired by Network Science
• All-optical Reservoir Computing
• Stable Architectures for Deep Neural Networks
• High performance photonic reservoir computer based on a coherently driven passive cavity
• Reservoir computing with a single time-delay autonomous Boolean node
• Spatio-temporal phenomena in complex systems with time delays
• Performance boost of time-delay reservoir computing by non-resonant clock cycle
• Delayed Dynamical Systems: Networks, Chimeras and Reservoir Computing
• PAM-4 Transmission at 1550nm using Photonic Reservoir Computing Post-processing
• Experiments with arbitrary networks in time-multiplexed delay systems

Cited by in corpus (3)

• High Speed Photonic Neuromorphic Computing Using Recurrent Optical Spectrum Slicing Neural Networks
• Delay-induced self-oscillation excitation in the FitzHugh-Nagumo model: regular and chaotic dynamics
• Fine-Pruning: A Biologically Inspired Algorithm for Personalization of Machine Learning Models