34 citations · 34 across the 3 of their papers we have counts for
4 papers
Temperon: Full-Time SAM Quality at a Third Less Wall-Clock
Stamatis Mastromichalakis
Sharpness-aware minimization (SAM) doubles the cost of every training step, yet its benefit concentrates where training ends. We study where an expensive training mode should be sp…
OptiRoulette Optimizer: A New Stochastic Meta-Optimizer for up to 5.3x Faster Convergence
Stamatis Mastromichalakis
This paper presents OptiRoulette, a stochastic meta-optimizer that selects update rules during training instead of fixing a single optimizer. The method combines warmup optimizer l…
Parametric Leaky Tanh: A New Hybrid Activation Function for Deep Learning
Stamatis Mastromichalakis
Activation functions (AFs) are crucial components of deep neural networks (DNNs), having a significant impact on their performance. An activation function in a DNN is typically a s…
ALReLU: A different approach on Leaky ReLU activation function to improve Neural Networks Performance
Stamatis Mastromichalakis
Despite the unresolved 'dying ReLU problem', the classical ReLU activation function (AF) has been extensively applied in Deep Neural Networks (DNN), in particular Convolutional Neu…