Intelligence, physics and information -- the tradeoff between accuracy and simplicity in machine learning

Abstract

How can we enable machines to make sense of the world, and become better at learning? To approach this goal, I believe viewing intelligence in terms of many integral aspects, and also a universal two-term tradeoff between task performance and complexity, provides two feasible perspectives. In this thesis, I address several key questions in some aspects of intelligence, and study the phase transitions in the two-term tradeoff, using strategies and tools from physics and information. Firstly, how can we make the learning models more flexible and efficient, so that agents can learn quickly with fewer examples? Inspired by how physicists model the world, we introduce a paradigm and an AI Physicist agent for simultaneously learning many small specialized models (theories) and the domain they are accurate, which can then be simplified, unified and stored, facilitating few-shot learning in a continual way. Secondly, for representation learning, when can we learn a good representation, and how does learning depend on the structure of the dataset? We approach this question by studying phase transitions when tuning the tradeoff hyperparameter. In the information bottleneck, we theoretically show that these phase transitions are predictable and reveal structure in the relationships between the data, the model, the learned representation and the loss landscape. Thirdly, how can agents discover causality from observations? We address part of this question by introducing an algorithm that combines prediction and minimizing information from the input, for exploratory causal discovery from observational time series. Fourthly, to make models more robust to label noise, we introduce Rank Pruning, a robust algorithm for classification with noisy labels. I believe that building on the work of my thesis we will be one step closer to enable more intelligent machines that can make sense of the world.

PhD Thesis, 352 pages. Reference improved

References in corpus (18)

• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Neural Architecture Search with Reinforcement Learning
• Theoretical Models of Learning to Learn
• Meta-SGD: Learning to Learn Quickly for Few-Shot Learning
• Learning phase transitions by confusion
• Interaction Networks for Learning about Objects, Relations and Physics
• Group Sparse Regularization for Deep Neural Networks
• Kernel method for nonlinear Granger causality
• A Collection of Definitions of Intelligence
• Learning to reinforcement learn
• Local information transfer as a spatiotemporal filter for complex systems
• Kernel Granger causality and the analysis of dynamical networks
• A Compositional Object-Based Approach to Learning Physical Dynamics
• RobustFill: Neural Program Learning under Noisy I/O
• Neuro-Symbolic Program Synthesis
• Permutation-equivariant neural networks applied to dynamics prediction
• Predicting accurate probabilities with a ranking loss
• Multiple Kernel Learning from Noisy Labels by Stochastic Programming