Topos and Stacks of Deep Neural Networks
arXiv:2106.14587
Abstract
Every known artificial deep neural network (DNN) corresponds to an object in a canonical Grothendieck's topos; its learning dynamic corresponds to a flow of morphisms in this topos. Invariance structures in the layers (like CNNs or LSTMs) correspond to Giraud's stacks. This invariance is supposed to be responsible of the generalization property, that is extrapolation from learning data under constraints. The fibers represent pre-semantic categories (Culioli, Thom), over which artificial languages are defined, with internal logics, intuitionist, classical or linear (Girard). Semantic functioning of a network is its ability to express theories in such a language for answering questions in output about input data. Quantities and spaces of semantic information are defined by analogy with the homological interpretation of Shannon's entropy of P.Baudot and D.Bennequin in 2015). They generalize the measures found by Carnap and Bar-Hillel (1952). Amazingly, the above semantical structures are classified by geometric fibrant objects in a closed model category of Quillen, then they give rise to homotopical invariants of DNNs and of their semantic functioning. Intentional type theories (Martin-Loef) organize these objects and fibrations between them. Information contents and exchanges are analyzed by Grothendieck's derivators.
arXiv comment151 pages, 12 figures
References in corpus (25)
- Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
- Deep Learning Enabled Semantic Communication Systems
- A simple neural network module for relational reasoning
- Gauge Equivariant Convolutional Networks and the Icosahedral CNN
- Relational Deep Reinforcement Learning
- Deep Visual-Semantic Alignments for Generating Image Descriptions
- Measuring abstract reasoning in neural networks
- A Characterization of Entropy in Terms of Information Loss
- Learning to Compose Neural Networks for Question Answering
- Sheaves, Cosheaves and Applications
- Discovering objects and their relations from entangled scene representations
- N-body Networks: a Covariant Hierarchical Neural Network Architecture for Learning Atomic Potentials
- Seven Sketches in Compositionality: An Invitation to Applied Category Theory
- Exact completions and small sheaves
- Homotopy Theoretic Models of Type Theory
- Universal models and definability
- Minimal Gated Unit for Recurrent Neural Networks
- Simplified Minimal Gated Unit Variations for Recurrent Neural Networks
- Homotopy Theoretic and Categorical Models of Neural Information Networks
- The Hintons in your Neural Network: a Quantum Field Theory View of Deep Learning
- Cohomology in Grothendieck Topologies and Lower Bounds in Boolean Complexity
- Intersection property and interaction decomposition
- A Homological Approach to Belief Propagation and Bethe Approximations
- Pursuing Stacks
- Interaction decomposition for presheaves