Fermi-Bose Machine achieves both generalization and adversarial robustness
arXiv:2404.13631 · doi:10.1007/s11433-024-2504-8
Abstract
Distinct from human cognitive processing, deep neural networks trained by backpropagation can be easily fooled by adversarial examples. To design a semantically meaningful representation learning, we discard backpropagation, and instead, propose a local contrastive learning, where the representation for the inputs bearing the same label shrink (akin to boson) in hidden layers, while those of different labels repel (akin to fermion). This layer-wise learning is local in nature, being biological plausible. A statistical mechanics analysis shows that the target fermion-pair-distance is a key parameter. Moreover, the application of this local contrastive learning to MNIST benchmark dataset demonstrates that the adversarial vulnerability of standard perceptron can be greatly mitigated by tuning the target distance, i.e., controlling the geometric separation of prototype manifolds.
arXiv comment32 pages, 6 figures, a physics inspired machine without backpropagation yet with enhanced adversarial robustness
References in corpus (8)
- Shortcut Learning in Deep Neural Networks
- Supervised Hebbian Learning
- The committee machine: Computational to statistical gaps in learning a two-layers neural network
- Mechanisms of dimensionality reduction and decorrelation in deep neural networks
- Eight challenges in developing theory of intelligence
- Weakly-correlated synapses promote dimension reduction in deep neural networks
- Emergence of hierarchical modes from deep learning
- Relationship between manifold smoothness and adversarial vulnerability in deep learning with local errors