A Model of Inductive Bias Learning

Abstract

A major problem in machine learning is that of inductive bias: how to choose a learner's hypothesis space so that it is large enough to contain a solution to the problem being learnt, yet small enough to ensure reliable generalization from reasonably-sized training sets. Typically such bias is supplied by hand through the skill and insights of experts. In this paper a model for automatically learning bias is investigated. The central assumption of the model is that the learner is embedded within an environment of related learning tasks. Within such an environment the learner can sample from multiple tasks, and hence it can search for a hypothesis space that contains good solutions to many of the problems in the environment. Under certain restrictions on the set of all hypothesis spaces available to the learner, we show that a hypothesis space that performs well on a sufficiently large number of training tasks will also perform well when learning novel tasks in the same environment. Explicit bounds are also derived demonstrating that learning multiple tasks within an environment of related tasks can potentially give much better generalization than learning a single task.

References in corpus (1)

• The Canonical Distortion Measure for Vector Quantization and Function Approximation

Cited by in corpus (42)

• PhysNet: A Neural Network for Predicting Energies, Forces, Dipole Moments and Partial Charges
• Machine Learning of Molecular Electronic Properties in Chemical Compound Space
• Fairness in Machine Learning
• Detection and Localization of Robotic Tools in Robot-Assisted Surgery Videos Using Deep Neural Networks for Region Proposal and Detection
• Conic Programming Reformulations of Two-Stage Distributionally Robust Linear Programs over Wasserstein Balls
• Learn on Source, Refine on Target:A Model Transfer Learning Framework with Random Forests
• Multitask diffusion adaptation over networks with common latent representations
• Synergic Adversarial Label Learning for Grading Retinal Diseases via Knowledge Distillation and Multi-task Learning
• A Tutorial on Meta-Reinforcement Learning
• DUET: A Tuning-Free Device-Cloud Collaborative Parameters Generation Framework for Efficient Device Model Generalization
• Learning Pair Potentials using Differentiable Simulations
• Machine learning for complete intersection Calabi-Yau manifolds: a methodological study
• Client-server multi-task learning from distributed datasets
• Simultaneous prediction of multiple outcomes using revised stacking algorithms
• Lightweight and Effective Tensor Sensitivity for Atomistic Neural Networks
• Vision Transformers in Precision Agriculture: A Comprehensive Survey
• Information-Theoretic Generalization Bounds for Meta-Learning and Applications
• Probing transfer learning with a model of synthetic correlated datasets
• Robust Online Multi-Task Learning with Correlative and Personalized Structures
• DeepTSF: Codeless machine learning operations for time series forecasting
• DC-Check: A Data-Centric AI checklist to guide the development of reliable machine learning systems
• Why we need biased AI -- How including cognitive and ethical machine biases can enhance AI systems
• Control of Dual-Sourcing Inventory Systems using Recurrent Neural Networks
• Multi-Task Learning for Multi-Dimensional Regression: Application to Luminescence Sensing
• Saliency-Regularized Deep Multi-Task Learning
• Constructing Time-Series Momentum Portfolios with Deep Multi-Task Learning
• Follow the Prophet: Accurate Online Conversion Rate Prediction in the Face of Delayed Feedback
• Compression-Based Regularization with an Application to Multi-Task Learning
• Quantum Supervised Learning
• PAC-Bayesian Lifelong Learning For Multi-Armed Bandits
• Study of positional encoding approaches for Audio Spectrogram Transformers
• BERP: A Blind Estimator of Room Parameters for Single-Channel Noisy Speech Signals
• Fairness in Multi-Task Learning via Wasserstein Barycenters
• CatVRNN: Generating Category Texts via Multi-task Learning
• Collaborative Intelligence Orchestration: Inconsistency-Based Fusion of Semi-Supervised Learning and Active Learning
• Low-Energy Electron-Track Imaging for a Liquid Argon Time-Projection-Chamber Telescope Concept using Probabilistic Deep Learning
• Meta Learning to Rank for Sparsely Supervised Queries
• On the Bounds of Function Approximations
• CAM2: Conformity-Aware Multi-Task Ranking Model for Large-Scale Recommender Systems
• Optimal Protocols for Continual Learning via Statistical Physics and Control Theory
• On regret bounds for continual single-index learning
• A chain rule for the expected suprema of Gaussian processes