Intelligence, physics and information -- the tradeoff between accuracy and simplicity in machine learning
arXiv:2001.03780
Abstract
How can we enable machines to make sense of the world, and become better at learning? To approach this goal, I believe viewing intelligence in terms of many integral aspects, and also a universal two-term tradeoff between task performance and complexity, provides two feasible perspectives. In this thesis, I address several key questions in some aspects of intelligence, and study the phase transitions in the two-term tradeoff, using strategies and tools from physics and information. Firstly, how can we make the learning models more flexible and efficient, so that agents can learn quickly with fewer examples? Inspired by how physicists model the world, we introduce a paradigm and an AI Physicist agent for simultaneously learning many small specialized models (theories) and the domain they are accurate, which can then be simplified, unified and stored, facilitating few-shot learning in a continual way. Secondly, for representation learning, when can we learn a good representation, and how does learning depend on the structure of the dataset? We approach this question by studying phase transitions when tuning the tradeoff hyperparameter. In the information bottleneck, we theoretically show that these phase transitions are predictable and reveal structure in the relationships between the data, the model, the learned representation and the loss landscape. Thirdly, how can agents discover causality from observations? We address part of this question by introducing an algorithm that combines prediction and minimizing information from the input, for exploratory causal discovery from observational time series. Fourthly, to make models more robust to label noise, we introduce Rank Pruning, a robust algorithm for classification with noisy labels. I believe that building on the work of my thesis we will be one step closer to enable more intelligent machines that can make sense of the world.
PhD Thesis, 352 pages. Reference improved
References in corpus (60)
- Auto-Encoding Variational Bayes
- Rejoinder to "Least angle regression" by Efron et al
- Discussion of "Least angle regression" by Efron et al
- Discussion of "Least angle regression" by Efron et al
- Discussion of "Least angle regression" by Efron et al
- Discussion of "Least angle regression" by Efron et al
- Least Angle Regression
- Discussion of "Least angle regression" by Efron et al
- Discussion of "Least angle regression" by Efron et al
- Discussion of "Least angle regression" by Efron et al
- Discussion of "Least angle regression" by Efron et al
- Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
- Prototypical Networks for Few-shot Learning
- Measuring Information Transfer
- Estimating Mutual Information
- Neural Architecture Search with Reinforcement Learning
- The information bottleneck method
- Machine learning phases of matter
- Theoretical Models of Learning to Learn
- Searching for Exotic Particles in High-Energy Physics with Deep Learning
- Meta-SGD: Learning to Learn Quickly for Few-Shot Learning
- On the performance of different synchronization measures in real data: a case study on EEG signals
- Classification with Noisy Labels by Importance Reweighting
- Learning phase transitions by confusion
- AutoAugment: Learning Augmentation Policies from Data
- Discovering Phase Transitions with Unsupervised Learning
- Interaction Networks for Learning about Objects, Relations and Physics
- Large-Scale Evolution of Image Classifiers
- Group Sparse Regularization for Deep Neural Networks
- Discovering physical concepts with neural networks
- Kernel method for nonlinear Granger causality
- A Collection of Definitions of Intelligence
- Learning to reinforcement learn
- Deep Variational Information Bottleneck
- Reconstructing quantum states with generative models
- Local information transfer as a spatiotemporal filter for complex systems
- Variational Continual Learning
- Characterization and Greedy Learning of Interventional Markov Equivalence Classes of Directed Acyclic Graphs
- When Will AI Exceed Human Performance? Evidence from AI Experts
- Kernel Granger causality and the analysis of dynamical networks
- Learning Representations for Neural Network-Based Classification Using the Information Bottleneck Principle
- A Compositional Object-Based Approach to Learning Physical Dynamics
- Towards a Neural Statistician
- Shannon Information and Kolmogorov Complexity
- Relational Neural Expectation Maximization: Unsupervised Discovery of Objects and their Interactions
- A Robust Ensemble Approach to Learn From Positive and Unlabeled Data Using SVM Base Models
- RobustFill: Neural Program Learning under Noisy I/O
- Neuro-Symbolic Program Synthesis
- Dynamics-Aware Unsupervised Discovery of Skills
- Increasing the Interpretability of Recurrent Neural Networks Using Hidden Markov Models
- Learning with Confident Examples: Rank Pruning for Robust Classification with Noisy Labels
- A Theory of Universal Artificial Intelligence based on Algorithmic Complexity
- Unsupervised Learning of Latent Physical Properties Using Perception-Prediction Networks
- Statistical Criticality arises in Most Informative Representations
- Permutation-equivariant neural networks applied to dynamics prediction
- A machine learns to predict the stability of circumbinary planets
- Predicting accurate probabilities with a ranking loss
- Caveats for information bottleneck in deterministic scenarios
- Multiple Kernel Learning from Noisy Labels by Stochastic Programming
- Unsupervised Discovery of Temporal Structure in Noisy Data with Dynamical Components Analysis