On the Maximum Mutual Information Capacity of Neural Architectures

Brandon Foggo, Nanpeng Yu

We derive the closed-form expression of the maximum mutual information - the maximum value of $I(X;Z)$ obtainable via training - for a broad family of neural network architectures.…