AP: Selective Activation for De-sparsifying Pruned Neural Networks

Shiyu Liu, Rohan Ghosh, Dylan Tan +1

The rectified linear unit (ReLU) is a highly successful activation function in neural networks as it allows networks to easily obtain sparse representations, which reduces overfitt…