Low-dimensional topology of deep neural networks

Junyu Ren, Lek-Heng Lim

We study layered models, including feedforward networks, ResNets, and transformers, by limiting each layer to a width of $d=3$, i.e., ${\mathbb{R}}^3$ as representation space. This…