Optimize TSK Fuzzy Systems for Classification Problems: Mini-Batch Gradient Descent with Uniform Regularization and Batch Normalization

Abstract

Takagi-Sugeno-Kang (TSK) fuzzy systems are flexible and interpretable machine learning models; however, they may not be easily optimized when the data size is large, and/or the data dimensionality is high. This paper proposes a mini-batch gradient descent (MBGD) based algorithm to efficiently and effectively train TSK fuzzy classifiers. It integrates two novel techniques: 1) uniform regularization (UR), which forces the rules to have similar average contributions to the output, and hence to increase the generalization performance of the TSK classifier; and, 2) batch normalization (BN), which extends BN from deep neural networks to TSK fuzzy classifiers to expedite the convergence and improve the generalization performance. Experiments on 12 UCI datasets from various application domains, with varying size and dimensionality, demonstrated that UR and BN are effective individually, and integrating them can further improve the classification performance.

References in corpus (12)

• Adam: A Method for Stochastic Optimization
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• An overview of gradient descent optimization algorithms
• Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)
• Wide Residual Networks
• The Marginal Value of Adaptive Gradient Methods in Machine Learning
• Improving Generalization Performance by Switching from Adam to SGD
• Revisiting Small Batch Training for Deep Neural Networks
• Adaptive Gradient Methods with Dynamic Bound of Learning Rate
• A Survey on Soft Subspace Clustering
• Mixture Models for Diverse Machine Translation: Tricks of the Trade
• Revisit Fuzzy Neural Network: Demystifying Batch Normalization and ReLU with Generalized Hamming Network

Cited by in corpus (6)

• RSDiff: Remote Sensing Image Generation from Text Using Diffusion Model
• An Integrated Fusion Framework for Ensemble Learning Leveraging Gradient Boosting and Fuzzy Rule-Based Models
• Distilling a Deep Neural Network into a Takagi-Sugeno-Kang Fuzzy Inference System
• FCM-RDpA: TSK Fuzzy Regression Model Construction Using Fuzzy C-Means Clustering, Regularization, DropRule, and Powerball AdaBelief
• Supervised Enhanced Soft Subspace Clustering (SESSC) for TSK Fuzzy Classifiers
• MBGD-RDA Training and Rule Pruning for Concise TSK Fuzzy Regression Models