Nesting Forward Automatic Differentiation for Memory-Efficient Deep Neural Network Training

Cong Guo, Yuxian Qiu, Jingwen Leng +6

An activation function is an element-wise mathematical function and plays a crucial role in deep neural networks (DNN). Many novel and sophisticated activation functions have been…