64 citations · 102 across the 16 of their papers we have counts for
16 papers · 1 filter
NeuroTrails: Training with Dynamic Sparse Heads as the Key to Effective Ensembling
Bram Grooten, Farid Hasanov, Chenxiang Zhang +9
Model ensembles have long been a cornerstone for improving generalization and robustness in deep learning. However, their effectiveness often comes at the cost of substantial compu…
Mind the GAP! The Challenges of Scale in Pixel-based Deep Reinforcement Learning
Ghada Sokar, Pablo Samuel Castro
Scaling deep reinforcement learning in pixel-based environments presents a significant challenge, often resulting in diminished performance. While recent works have proposed algori…
Don't flatten, tokenize! Unlocking the key to SoftMoE's efficacy in deep RL
Ghada Sokar, Johan Obando-Ceron, Aaron Courville +2
The use of deep neural networks in reinforcement learning (RL) often suffers from performance degradation as model size increases. While soft mixtures of experts (SoftMoEs) have re…
Mixtures of Experts Unlock Parameter Scaling for Deep RL
Johan Obando-Ceron, Ghada Sokar, Timon Willi +6
The recent rapid progress in (self) supervised learning models is in large part predicted by empirical scaling laws: a model's performance scales proportionally to its size. Analog…
Continual Learning with Dynamic Sparse Training: Exploring Algorithms for Effective Model Updates
Murat Onur Yildirim, Elif Ceren Gok Yildirim, Ghada Sokar +2
Continual learning (CL) refers to the ability of an intelligent system to sequentially acquire and retain knowledge from a stream of data with as little computational overhead as p…
Automatic Noise Filtering with Dynamic Sparse Training in Deep Reinforcement Learning
Bram Grooten, Ghada Sokar, Shibhansh Dohare +4
Tomorrow's robots will need to distinguish useful information from noise when performing different tasks. A household robot for instance may continuously receive a plethora of info…
The Dormant Neuron Phenomenon in Deep Reinforcement Learning
Ghada Sokar, Rishabh Agarwal, Pablo Samuel Castro +1
In this work we identify the dormant neuron phenomenon in deep reinforcement learning, where an agent's network suffers from an increasing number of inactive neurons, thereby affec…
Where to Pay Attention in Sparse Training for Feature Selection?
Ghada Sokar, Zahra Atashgahi, Mykola Pechenizkiy +1
A new line of research for feature selection based on neural networks has recently emerged. Despite its superiority to classical methods, it requires many training iterations to co…
Avoiding Forgetting and Allowing Forward Transfer in Continual Learning via Sparse Networks
Ghada Sokar, Decebal Constantin Mocanu, Mykola Pechenizkiy
Using task-specific components within a neural network in continual learning (CL) is a compelling strategy to address the stability-plasticity dilemma in fixed-capacity models with…
Deep Ensembling with No Overhead for either Training or Testing: The All-Round Blessings of Dynamic Sparsity
Shiwei Liu, Tianlong Chen, Zahra Atashgahi +6
The success of deep ensembles on improving predictive performance, uncertainty estimation, and out-of-distribution robustness has been extensively studied in the machine learning l…
Dynamic Sparse Training for Deep Reinforcement Learning
Ghada Sokar, Elena Mocanu, Decebal Constantin Mocanu +2
Deep reinforcement learning (DRL) agents are trained through trial-and-error interactions with the environment. This leads to a long training time for dense neural networks to achi…
Self-Attention Meta-Learner for Continual Learning
Ghada Sokar, Decebal Constantin Mocanu, Mykola Pechenizkiy
Continual learning aims to provide intelligent agents capable of learning multiple tasks sequentially with neural networks. One of its main challenging, catastrophic forgetting, is…
Learning Invariant Representation for Continual Learning
Ghada Sokar, Decebal Constantin Mocanu, Mykola Pechenizkiy
Continual learning aims to provide intelligent agents that are capable of learning continually a sequence of tasks, building on previously learned knowledge. A key challenge in thi…
Quick and Robust Feature Selection: the Strength of Energy-efficient Sparse Training for Autoencoders
Zahra Atashgahi, Ghada Sokar, Tim van der Lee +4
Major complications arise from the recent increase in the amount of high-dimensional data, including high computational costs and memory requirements. Feature selection, which iden…
Topological Insights into Sparse Neural Networks
Shiwei Liu, Tim Van der Lee, Anil Yaman +5
Sparse neural networks are effective approaches to reduce the resource requirements for the deployment of deep neural networks. Recently, the concept of adaptive sparse connectivit…
SpaceNet: Make Free Space For Continual Learning
Ghada Sokar, Decebal Constantin Mocanu, Mykola Pechenizkiy
The continual learning (CL) paradigm aims to enable neural networks to learn tasks continually in a sequential fashion. The fundamental challenge in this learning paradigm is catas…