Local Boosting for Weakly-Supervised Learning
arXiv:2306.02859 · doi:10.1145/3580305.3599417
Abstract
Boosting is a commonly used technique to enhance the performance of a set of base models by combining them into a strong ensemble model. Though widely adopted, boosting is typically used in supervised learning where the data is labeled accurately. However, in weakly supervised learning, where most of the data is labeled through weak and noisy sources, it remains nontrivial to design effective boosting approaches. In this work, we show that the standard implementation of the convex combination of base learners can hardly work due to the presence of noisy labels. Instead, we propose , a novel framework for weakly-supervised boosting. LocalBoost iteratively boosts the ensemble model from two dimensions, i.e., intra-source and inter-source. The intra-source boosting introduces locality to the base learners and enables each base learner to focus on a particular feature regime by training new base learners on granularity-varying error regions. For the inter-source boosting, we leverage a conditional function to indicate the weak source where the sample is more likely to appear. To account for the weak labels, we further design an estimate-then-modify approach to compute the model weights. Experiments on seven datasets show that our method significantly outperforms vanilla boosting methods and other weakly-supervised methods.
arXiv commentAccepted by KDD 2023 Research Track
References in corpus (26)
- XGBoost: A Scalable Tree Boosting System
- Decoupled Weight Decay Regularization
- Character-level Convolutional Networks for Text Classification
- SemEval-2010 Task 8: Multi-Way Classification of Semantic Relations Between Pairs of Nominals
- Algorithms and Theory for Multiple-Source Adaptation
- Denoising Multi-Source Weak Supervision for Neural Text Classification
- A Survey on Programmatic Weak Supervision
- WRENCH: A Comprehensive Benchmark for Weak Supervision
- Learning Deep ResNet Blocks Sequentially using Boosting Theory
- Learning from Rules Generalizing Labeled Exemplars
- Language Models in the Loop: Incorporating Prompting into Weak Supervision
- Fast and Three-rious: Speeding Up Weak Supervision with Triplet Methods
- Weakly-Supervised Scientific Document Classification via Retrieval-Augmented Multi-Stage Training
- COCO-DR: Combating Distribution Shifts in Zero-Shot Dense Retrieval with Contrastive and Distributionally Robust Learning
- Improved Regularization and Robustness for Fine-tuning in Neural Networks
- MiCE: Mixture of Contrastive Experts for Unsupervised Image Clustering
- Cold-Start Data Selection for Few-shot Language Model Fine-tuning: A Prompt-Based Uncertainty Propagation Approach
- Universalizing Weak Supervision
- Building Robust Ensembles via Margin Boosting
- A Discriminative Technique for Multiple-Source Adaptation
- Adaptive Multi-view Rule Discovery for Weakly-Supervised Compatible Products Prediction
- Do Not Blindly Imitate the Teacher: Using Perturbed Loss for Knowledge Distillation
- Functional Gradient Boosting based on Residual Network Perception
- Leveraging Instance Features for Label Aggregation in Programmatic Weak Supervision
- ReGen: Zero-Shot Text Classification via Training Data Generation with Progressive Dense Retrieval
- SepLL: Separating Latent Class Labels from Weak Supervision Noise