4 papers
The Resurrection of the ReLU
CoÅku Can Horuz, Geoffrey Kasenbacher, Saya Higuchi +7
Modeling sophisticated activation functions within deep learning architectures has evolved into a distinct research direction. Functions such as GELU, SELU, and SiLU offer smooth g…
The Space Between: On Folding, Symmetries and Sampling
Michal Lewandowski, Bernhard Heinzl, Raphael Pisoni +1
Recent findings suggest that consecutive layers of neural networks with the ReLU activation function \emph{fold} the input space during the learning process. While many works hint…
On Space Folds of ReLU Neural Networks
Michal Lewandowski, Hamid Eghbalzadeh, Bernhard Heinzl +2
Recent findings suggest that the consecutive layers of ReLU neural networks can be understood geometrically as space folding transformations of the input space, revealing patterns…
CantorNet: A Sandbox for Testing Geometrical and Topological Complexity Measures
Michal Lewandowski, Hamid Eghbalzadeh, Bernhard A. Moser
Many natural phenomena are characterized by self-similarity, for example the symmetry of human faces, or a repetitive motif of a song. Studying of such symmetries will allow us to…