Human action recognition with a large-scale brain-inspired photonic computer

Abstract

The recognition of human actions in video streams is a challenging task in computer vision, with cardinal applications in e.g. brain-computer interface and surveillance. Deep learning has shown remarkable results recently, but can be found hard to use in practice, as its training requires large datasets and special purpose, energy-consuming hardware. In this work, we propose a scalable photonic neuro-inspired architecture based on the reservoir computing paradigm, capable of recognising video-based human actions with state-of-the-art accuracy. Our experimental optical setup comprises off-the-shelf components, and implements a large parallel recurrent neural network that is easy to train and can be scaled up to hundreds of thousands of nodes. This work paves the way towards simply reconfigurable and energy-efficient photonic information processing systems for real-time video processing.

Authors' version before final reviews (12 pages, 4 figures)

References in corpus (7)

• Optoelectronic Reservoir Computing
• All-optical Reservoir Computing
• Reinforcement Learning in a large scale photonic Recurrent Neural Network
• High performance photonic reservoir computer based on a coherently driven passive cavity
• Online Training of an Opto-Electronic Reservoir Computer Applied to Real-Time Channel Equalisation
• Brain-inspired photonic signal processor for periodic pattern generation and chaotic system emulation
• Parallel photonic reservoir computing using frequency multiplexing of neurons

Cited by in corpus (23)

• Photonics for artificial intelligence and neuromorphic computing
• Large-scale neuromorphic optoelectronic computing with a reconfigurable diffractive processing unit
• Opportunities in Quantum Reservoir Computing and Extreme Learning Machines
• Photonic extreme learning machine by free-space optical propagation
• Soliton crystal Kerr microcombs for high-speed, scalable optical neural networks at 10 GigaOPs/s
• Photonic real time video image signal processor at 17Tb/s based on a Kerr microcomb
• High Speed Human Action Recognition using a Photonic Reservoir Computer
• Energy-efficient neural network inference with microcavity exciton-polaritons
• Ultrafast single-channel machine vision based on neuro-inspired photonic computing
• Muti-view Mouse Social Behaviour Recognition with Deep Graphical Model
• Deep Photonic Reservoir Computer for Speech Recognition
• LiNbO${}_3$ volatile memristors for reservoir computing
• Persistent Homology Meets Object Unity: Object Recognition in Clutter
• Quantum Reservoir Computing for Speckle-Disorder Potentials
• Image classification using collective optical modes of an array of nanolasers
• Neural networks based on ultrafast time-delayed effects in exciton-polaritons
• High-resolution dynamic consistency analysis of photonic time-delay reservoir computer
• Photonic neural field on a silicon chip: large-scale, high-speed neuro-inspired computing and sensing
• Photonic single perceptron at Giga-OP/s speeds with Kerr microcombs for scalable optical neural networks
• THOR2: Topological Analysis for 3D Shape and Color-Based Human-Inspired Object Recognition in Unseen Environments
• A Novel Approach for Robust Multi Human Action Recognition and Summarization based on 3D Convolutional Neural Networks
• Single photonic perceptron based on a soliton crystal Kerr microcomb for high-speed, scalable, optical neural networks
• High speed and reconfigurable optronic neural network with digital nonlinear activation