46 citations · 297 across the 32 of their papers we have counts for
35 papers
Synaptic Weight Distributions Depend on the Geometry of Plasticity
Roman Pogodin, Jonathan Cornford, Arna Ghosh +3
A growing literature in computational neuroscience leverages gradient descent and learning algorithms that approximate it to study synaptic plasticity in the brain. However, the va…
Steerable Equivariant Representation Learning
Sangnie Bhardwaj, Willie McClinton, Tongzhou Wang +4
Pre-trained deep image representations are useful for post-training tasks such as classification through transfer learning, image retrieval, and object detection. Data augmentation…
Flexible Phase Dynamics for Bio-Plausible Contrastive Learning
Ezekiel Williams, Colin Bredenberg, Guillaume Lajoie
Many learning algorithms used as normative models in neuroscience or as candidate approaches for learning on neuromorphic chips learn by contrasting one set of network states with…
Sources of Richness and Ineffability for Phenomenally Conscious States
Xu Ji, Eric Elmoznino, George Deane +5
Conscious states (states that there is something it is like to be in) seem both rich or full of detail, and ineffable or hard to fully describe or recall. The problem of ineffabili…
Reliability of CKA as a Similarity Measure in Deep Learning
MohammadReza Davari, Stefan Horoi, Amine Natik +3
Comparing learned neural representations in neural networks is a challenging but important problem, which has been approached in different ways. The Centered Kernel Alignment (CKA)…
Transfer Entropy Bottleneck: Learning Sequence to Sequence Information Transfer
Damjan Kalajdzievski, Ximeng Mao, Pascal Fortier-Poisson +2
When presented with a data stream of two statistically dependent variables, predicting the future of one of the variables (the target stream) can benefit from information about bot…
Lazy vs hasty: linearization in deep networks impacts learning schedule based on example difficulty
Thomas George, Guillaume Lajoie, Aristide Baratin
Among attempts at giving a theoretical account of the success of deep neural networks, a recent line of work has identified a so-called lazy training regime in which the network ca…
On Neural Architecture Inductive Biases for Relational Tasks
Giancarlo Kerg, Sarthak Mittal, David Rolnick +3
Current deep learning approaches have shown good in-distribution generalization performance, but struggle with out-of-distribution generalization. This is especially true in the ca…
Is a Modular Architecture Enough?
Sarthak Mittal, Yoshua Bengio, Guillaume Lajoie
Inspired from human cognition, machine learning systems are gradually revealing advantages of sparser and more modular architectures. Recent work demonstrates that not only do some…
Beyond accuracy: generalization properties of bio-plausible temporal credit assignment rules
Yuhan Helena Liu, Arna Ghosh, Blake A. Richards +2
To unveil how the brain learns, ongoing work seeks biologically-plausible approximations of gradient descent algorithms for training recurrent neural networks (RNNs). Yet, beyond t…
Continuous-Time Meta-Learning with Forward Mode Differentiation
Tristan Deleu, David Kanaa, Leo Feng +4
Drawing inspiration from gradient-based meta-learning methods with infinitely small gradient steps, we introduce Continuous-Time Meta-Learning (COMLN), a meta-learning algorithm wh…
Clarifying MCMC-based training of modern EBMs : Contrastive Divergence versus Maximum Likelihood
Léo Gagnon, Guillaume Lajoie
The Energy-Based Model (EBM) framework is a very general approach to generative modeling that tries to learn and exploit probability distributions only defined though unnormalized…
Learning shared neural manifolds from multi-subject FMRI data
Jessie Huang, Erica L. Busch, Tom Wallenstein +6
Functional magnetic resonance imaging (fMRI) is a notoriously noisy measurement of brain activity because of the large variations between individuals, signals marred by environment…
Multi-scale Feature Learning Dynamics: Insights for Double Descent
Mohammad Pezeshki, Amartya Mitra, Yoshua Bengio +1
A key challenge in building theoretical foundations for deep learning is the complex optimization dynamics of neural networks, resulting from the high-dimensional interactions betw…
Compositional Attention: Disentangling Search and Retrieval
Sarthak Mittal, Sharath Chandra Raparthy, Irina Rish +2
Multi-head, key-value attention is the backbone of the widely successful Transformer model and its variants. This attention mechanism uses multiple parallel key-value attention blo…
Systematic Evaluation of Causal Discovery in Visual Model Based Reinforcement Learning
Nan Rosemary Ke, Aniket Didolkar, Sarthak Mittal +7
Inducing causal relationships from observations is a classic problem in machine learning. Most work in causality starts from the premise that the causal variables themselves are ob…
Embedding Signals on Knowledge Graphs with Unbalanced Diffusion Earth Mover's Distance
Alexander Tong, Guillaume Huguet, Dennis Shung +5
In modern relational machine learning it is common to encounter large graphs that arise via interactions or similarities between observations in many domains. Further, in many case…
Efficient and robust multi-task learning in the brain with modular latent primitives
Christian David Márton, Léo Gagnon, Guillaume Lajoie +1
Biological agents do not have infinite resources to learn new things. For this reason, a central aspect of human learning is the ability to recycle previously acquired knowledge in…
Exploring the Geometry and Topology of Neural Network Loss Landscapes
Stefan Horoi, Jessie Huang, Bastian Rieck +3
Recent work has established clear links between the generalization performance of trained neural networks and the geometry of their loss landscape near the local minima to which th…
Gradient Starvation: A Learning Proclivity in Neural Networks
Mohammad Pezeshki, Sékou-Oumar Kaba, Yoshua Bengio +3
We identify and formalize a fundamental gradient descent phenomenon resulting in a learning proclivity in over-parameterized neural networks. Gradient Starvation arises when cross-…
LEAD: Min-Max Optimization from a Physical Perspective
Reyhane Askari Hemmat, Amartya Mitra, Guillaume Lajoie +1
Adversarial formulations such as generative adversarial networks (GANs) have rekindled interest in two-player min-max games. A central obstacle in the optimization of such games is…
Implicit Regularization via Neural Feature Alignment
Aristide Baratin, Thomas George, César Laurent +4
We approach the problem of implicit regularization in deep learning from a geometrical viewpoint. We highlight a regularization effect induced by a dynamical alignment of the neura…
On Lyapunov Exponents for RNNs: Understanding Information Propagation Using Dynamical Systems Tools
Ryan Vogt, Maximilian Puelma Touzel, Eli Shlizerman +1
Recurrent neural networks (RNNs) have been successfully applied to a variety of problems involving sequential data, but their optimization is sensitive to parameter initialization,…
Advantages of biologically-inspired adaptive neural activation in RNNs during learning
Victor Geadah, Giancarlo Kerg, Stefan Horoi +2
Dynamic adaptation in single-neuron response plays a fundamental role in neural coding in biological neural networks. Yet, most neural activation functions used in artificial netwo…