Theano: new features and speed improvements

Abstract

Theano is a linear algebra compiler that optimizes a user's symbolically-specified mathematical computations to produce efficient low-level implementations. In this paper, we present new features and efficiency improvements to Theano, and benchmarks demonstrating Theano's performance relative to Torch7, a recently introduced machine learning library, and to RNNLM, a C++ library targeted at recurrent neural networks.

arXiv commentPresented at the Deep Learning Workshop, NIPS 2012

Cited by in corpus (234)

• A Survey on Deep Learning in Medical Image Analysis
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Explaining and Harnessing Adversarial Examples
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting
• On the difficulty of training Recurrent Neural Networks
• Efficient Multi-Scale 3D CNN with Fully Connected CRF for Accurate Brain Lesion Segmentation
• Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
• FitNets: Hints for Thin Deep Nets
• MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems
• BinaryConnect: Training Deep Neural Networks with binary weights during propagations
• Attention-Based Models for Speech Recognition
• Validation, comparison, and combination of algorithms for automatic detection of pulmonary nodules in computed tomography images: the LUNA16 challenge
• Surrogate Modeling for Fluid Flows Based on Physics-Constrained Deep Learning Without Simulation Data
• Deep Learning for Short-Term Traffic Flow Prediction
• Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks
• Neural NILM: Deep Neural Networks Applied to Energy Disaggregation
• Rotation-invariant convolutional neural networks for galaxy morphology prediction
• A Recurrent Latent Variable Model for Sequential Data
• Adversarially Learned Inference
• A C-LSTM Neural Network for Text Classification
• Semi-Supervised Learning with Ladder Networks
• NiftyNet: a deep-learning platform for medical imaging
• Training Deep Nets with Sublinear Memory Cost
• On Using Monolingual Corpora in Neural Machine Translation
• Identifying and attacking the saddle point problem in high-dimensional non-convex optimization
• Gated Feedback Recurrent Neural Networks
• LSTM-based Deep Learning Models for Non-factoid Answer Selection
• Gradient-based Hyperparameter Optimization through Reversible Learning
• Variational Autoencoder for Deep Learning of Images, Labels and Captions
• Markov Chain Monte Carlo and Variational Inference: Bridging the Gap
• Action Recognition using Visual Attention
• GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
• Distributional Smoothing with Virtual Adversarial Training
• MADE: Masked Autoencoder for Distribution Estimation
• Unsupervised and Semi-supervised Learning with Categorical Generative Adversarial Networks
• Mixed Neural Network Approach for Temporal Sleep Stage Classification
• Feed-Forward Networks with Attention Can Solve Some Long-Term Memory Problems
• Auxiliary Deep Generative Models
• Doctor AI: Predicting Clinical Events via Recurrent Neural Networks
• Jet Substructure Classification in High-Energy Physics with Deep Neural Networks
• Ladder Variational Autoencoders
• ReNet: A Recurrent Neural Network Based Alternative to Convolutional Networks
• Automatic tagging using deep convolutional neural networks
• GluonCV and GluonNLP: Deep Learning in Computer Vision and Natural Language Processing
• Pylearn2: a machine learning research library
• Describing Videos by Exploiting Temporal Structure
• Learning to Optimize Tensor Programs
• A Comprehensive Study of Deep Bidirectional LSTM RNNs for Acoustic Modeling in Speech Recognition
• Discourse-Aware Rumour Stance Classification in Social Media Using Sequential Classifiers
• Database Meets Deep Learning: Challenges and Opportunities
• Equilibrated adaptive learning rates for non-convex optimization
• Delving Deeper into Convolutional Networks for Learning Video Representations
• TVM: An Automated End-to-End Optimizing Compiler for Deep Learning
• Shallow and Deep Convolutional Networks for Saliency Prediction
• Blocks and Fuel: Frameworks for deep learning
• Adversarial Feature Matching for Text Generation
• Semantically Conditioned LSTM-based Natural Language Generation for Spoken Dialogue Systems
• Towards Universal Paraphrastic Sentence Embeddings
• A Neural Knowledge Language Model
• Distilling Knowledge from Deep Networks with Applications to Healthcare Domain
• Architectural Complexity Measures of Recurrent Neural Networks
• Black box variational inference for state space models
• Protein Secondary Structure Prediction with Long Short Term Memory Networks
• Censoring Representations with an Adversary
• Traversing Knowledge Graphs in Vector Space
• Dropout as data augmentation
• Practical Black-Box Attacks against Machine Learning
• Brain4Cars: Car That Knows Before You Do via Sensory-Fusion Deep Learning Architecture
• Improved Deep Learning Baselines for Ubuntu Corpus Dialogs
• Deep learning for galaxy surface brightness profile fitting
• Variance Reduction in SGD by Distributed Importance Sampling
• Comparative Study of Deep Learning Software Frameworks
• GossipGraD: Scalable Deep Learning using Gossip Communication based Asynchronous Gradient Descent
• Feature Representation in Convolutional Neural Networks
• Stochastic Language Generation in Dialogue using Recurrent Neural Networks with Convolutional Sentence Reranking
• Syntactically Guided Neural Machine Translation
• Towards End-to-End Speech Recognition with Deep Convolutional Neural Networks
• Generating Images from Captions with Attention
• On the saddle point problem for non-convex optimization
• Automatic Building Extraction in Aerial Scenes Using Convolutional Networks
• Benchmark of Deep Learning Models on Large Healthcare MIMIC Datasets
• Deep Reconstruction-Classification Networks for Unsupervised Domain Adaptation
• Tensor-Train Recurrent Neural Networks for Video Classification
• rlpyt: A Research Code Base for Deep Reinforcement Learning in PyTorch
• Natural Language Understanding with Distributed Representation
• Constrained Bayesian Optimization for Automatic Chemical Design
• Learning from Real Users: Rating Dialogue Success with Neural Networks for Reinforcement Learning in Spoken Dialogue Systems
• Visual Causal Feature Learning
• deepMiRGene: Deep Neural Network based Precursor microRNA Prediction
• High-Resolution Shape Completion Using Deep Neural Networks for Global Structure and Local Geometry Inference
• Alzheimer's Disease Diagnostics by a Deeply Supervised Adaptable 3D Convolutional Network
• Hierarchical Recurrent Neural Encoder for Video Representation with Application to Captioning
• Linear dynamical neural population models through nonlinear embeddings
• EmoNets: Multimodal deep learning approaches for emotion recognition in video
• Embedding Word Similarity with Neural Machine Translation
• Stochastic gradient variational Bayes for gamma approximating distributions
• Recurrent Spatial Transformer Networks
• Explorations on high dimensional landscapes
• Theano-based Large-Scale Visual Recognition with Multiple GPUs
• Recurrent Neural Networks with External Memory for Language Understanding
• Neural Generative Question Answering
• Representing high throughput expression profiles via perturbation barcodes reveals compound targets
• Sales Forecast in E-commerce using Convolutional Neural Network
• From A to Z: Supervised Transfer of Style and Content Using Deep Neural Network Generators
• SentiCap: Generating Image Descriptions with Sentiments
• TheanoLM - An Extensible Toolkit for Neural Network Language Modeling
• Task Loss Estimation for Sequence Prediction
• Regularizing RNNs by Stabilizing Activations
• Maximum-Margin Structured Learning with Deep Networks for 3D Human Pose Estimation
• Max-margin Deep Generative Models
• A Unified Gradient Regularization Family for Adversarial Examples
• From phonemes to images: levels of representation in a recurrent neural model of visually-grounded language learning
• Emergence of foveal image sampling from learning to attend in visual scenes
• Iterative Neural Autoregressive Distribution Estimator (NADE-k)
• What to talk about and how? Selective Generation using LSTMs with Coarse-to-Fine Alignment
• Interpretable Conservation Law Estimation by Deriving the Symmetries of Dynamics from Trained Deep Neural Networks
• Deep Learning in Bioinformatics
• Compatible Value Gradients for Reinforcement Learning of Continuous Deep Policies
• Churn analysis using deep convolutional neural networks and autoencoders
• Escoin: Efficient Sparse Convolutional Neural Network Inference on GPUs
• Recurrent Convolutional Neural Network Regression for Continuous Pain Intensity Estimation in Video
• Differentiation of the Cholesky decomposition
• Owl: A General-Purpose Numerical Library in OCaml
• Neural Networks Designing Neural Networks: Multi-Objective Hyper-Parameter Optimization
• Deep Directed Generative Autoencoders
• Convolutional RNN: an Enhanced Model for Extracting Features from Sequential Data
• DeepRT: deep learning for peptide retention time prediction in proteomics
• Machine learning for metal additive manufacturing: Predicting temperature and melt pool fluid dynamics using physics-informed neural networks
• End-to-end Convolutional Network for Saliency Prediction
• Multi-Label Learning from Medical Plain Text with Convolutional Residual Models
• Patterns for Learning with Side Information
• Learning Generic Sentence Representations Using Convolutional Neural Networks
• Functional connectivity patterns of autism spectrum disorder identified by deep feature learning
• Large Scale Evolution of Convolutional Neural Networks Using Volunteer Computing
• Telugu OCR Framework using Deep Learning
• Variational Inference with Hamiltonian Monte Carlo
• Learning to Filter with Predictive State Inference Machines
• Neural Networks with Smooth Adaptive Activation Functions for Regression
• Single Channel Audio Source Separation using Convolutional Denoising Autoencoders
• Learning to Generate with Memory
• The Game Imitation: Deep Supervised Convolutional Networks for Quick Video Game AI
• Robobarista: Learning to Manipulate Novel Objects via Deep Multimodal Embedding
• Deconvolutional Feature Stacking for Weakly-Supervised Semantic Segmentation
• Denoising without access to clean data using a partitioned autoencoder
• Enabling My Robot To Play Pictionary : Recurrent Neural Networks For Sketch Recognition
• Grid Loss: Detecting Occluded Faces
• ADASECANT: Robust Adaptive Secant Method for Stochastic Gradient
• Compositional Hierarchical Tensor Factorization: Representing Hierarchical Intrinsic and Extrinsic Causal Factors
• Shape Completion Enabled Robotic Grasping
• Vector representations of text data in deep learning
• Exploiting long-term temporal dynamics for video captioning
• A detailed comparative study of open source deep learning frameworks
• Semi-Supervised Generation with Cluster-aware Generative Models
• Unsupervised Discovery of El Nino Using Causal Feature Learning on Microlevel Climate Data
• LightNet: A Versatile, Standalone Matlab-based Environment for Deep Learning
• QBDC: Query by dropout committee for training deep supervised architecture
• Predictive Clinical Decision Support System with RNN Encoding and Tensor Decoding
• Likelihood Estimation for Generative Adversarial Networks
• Recurrent Neural Network based Part-of-Speech Tagger for Code-Mixed Social Media Text
• Learning language through pictures
• Hybrid Dialog State Tracker
• Learning Deep Generative Models with Doubly Stochastic MCMC
• Tied Hidden Factors in Neural Networks for End-to-End Speaker Recognition
• Dynamic Capacity Networks
• Blending LSTMs into CNNs
• Why not be Versatile? Applications of the SGNMT Decoder for Machine Translation
• MNN: A Universal and Efficient Inference Engine
• DeepCut: Object Segmentation from Bounding Box Annotations using Convolutional Neural Networks
• Fast Training of Triplet-based Deep Binary Embedding Networks
• A Multimodal Approach towards Emotion Recognition of Music using Audio and Lyrical Content
• Cyber-Physical System for Energy-Efficient Stadium Operation: Methodology and Experimental Validation
• Satirical News Detection and Analysis using Attention Mechanism and Linguistic Features
• Deep Learning in the Automotive Industry: Recent Advances and Application Examples
• Action Classification and Highlighting in Videos
• DiffSharp: Automatic Differentiation Library
• Attend in groups: a weakly-supervised deep learning framework for learning from web data
• Neural Dataset Generality
• Prediction of Prokaryotic and Eukaryotic Promoters Using Convolutional Deep Learning Neural Networks
• Analytic Insights into Structure and Rank of Neural Network Hessian Maps
• Adaptive DCTNet for Audio Signal Classification
• Neural Network Regularization via Robust Weight Factorization
• Read, Tag, and Parse All at Once, or Fully-neural Dependency Parsing
• A Hierarchical Recurrent Encoder-Decoder For Generative Context-Aware Query Suggestion
• LLNet: A Deep Autoencoder Approach to Natural Low-light Image Enhancement
• Recovery of non-linear cause-effect relationships from linearly mixed neuroimaging data
• Learned-Norm Pooling for Deep Feedforward and Recurrent Neural Networks
• Adaptive Normalized Risk-Averting Training For Deep Neural Networks
• MERLiN: Mixture Effect Recovery in Linear Networks
• On the Equivalence Between Deep NADE and Generative Stochastic Networks
• Can deep learning help you find the perfect match?
• Compression of Deep Neural Networks on the Fly
• Knowledge transfer in deep block-modular neural networks
• Continual Learning: Tackling Catastrophic Forgetting in Deep Neural Networks with Replay Processes
• Integrated Inference and Learning of Neural Factors in Structural Support Vector Machines
• A Framework for Distributed Deep Learning Layer Design in Python
• Reader-Aware Multi-Document Summarization: An Enhanced Model and The First Dataset
• Optimal Gradient Checkpoint Search for Arbitrary Computation Graphs
• Towards universal neural nets: Gibbs machines and ACE
• Tree-Structured Neural Machine for Linguistics-Aware Sentence Generation
• Train Feedfoward Neural Network with Layer-wise Adaptive Rate via Approximating Back-matching Propagation
• Can Deep Learning Predict Risky Retail Investors? A Case Study in Financial Risk Behavior Forecasting
• Symmetries and control in generative neural nets
• Learning to Assign Orientations to Feature Points
• Neural Machine Translation: A Review and Survey
• Semantics, Representations and Grammars for Deep Learning
• Deep Triphone Embedding Improves Phoneme Recognition
• Neural Variational Inference and Learning in Undirected Graphical Models
• Max-Margin Deep Generative Models for (Semi-)Supervised Learning
• MLitB: Machine Learning in the Browser
• PyTorchPipe: a framework for rapid prototyping of pipelines combining language and vision
• Optimizing Recurrent Neural Networks Architectures under Time Constraints
• Benchmarking Python Tools for Automatic Differentiation
• Social Cohesion in Autonomous Driving
• Connectionist Recommendation in the Wild: On the utility and scrutability of neural networks for personalized course guidance
• Optical Character Recognition (OCR) for Telugu: Database, Algorithm and Application
• Character-level Deep Conflation for Business Data Analytics
• Structured Sparse Convolutional Autoencoder
• Can fully convolutional networks perform well for general image restoration problems?
• On Binary Classification with Single-Layer Convolutional Neural Networks
• PZnet: Efficient 3D ConvNet Inference on Manycore CPUs
• What does fault tolerant Deep Learning need from MPI?
• Additive manufacturing process design with differentiable simulations
• Computational Graph Approach for Detection of Composite Human Activities
• Stacked Denoising Autoencoders and Transfer Learning for Immunogold Particles Detection and Recognition
• Optimising The Input Window Alignment in CD-DNN Based Phoneme Recognition for Low Latency Processing
• Discriminative Enhancement for Single Channel Audio Source Separation using Deep Neural Networks
• GaDei: On Scale-up Training As A Service For Deep Learning
• Accurate computational evolution of proteins and its dependence on deep learning
• Towards Explainable Fact Checking
• Understanding Minimum Probability Flow for RBMs Under Various Kinds of Dynamics
• Learning Robust Video Synchronization without Annotations
• Language discrimination and clustering via a neural network approach
• A Benchmark of Selected Algorithmic Differentiation Tools on Some Problems in Computer Vision and Machine Learning