105 citations · 187 across the 15 of their papers we have counts for
15 papers
Complex KDA: Understanding and Enhancing the Expressivity of Kimi Delta Attention
Julien Siems, Riccardo Grazzi, Korbinian Pöppel +8
Linear RNNs based on the delta-rule enable efficient sequence modeling, but their linear updates with a low-rank correction constrain their expressivity. Prior work has shown that…
Learning State-Tracking from Code Using Linear RNNs
Julien Siems, Riccardo Grazzi, Korbinian Pöppel +3
Over the last years, state-tracking tasks, particularly permutation composition, have become a testbed to understand the limits of sequence models architectures like Transformers a…
DeltaProduct: Improving State-Tracking in Linear RNNs via Householder Products
Julien Siems, Timur Carstensen, Arber Zela +3
Linear Recurrent Neural Networks (linear RNNs) have emerged as competitive alternatives to Transformers for sequence modeling, offering efficient training and linear-time inference…
Unlocking State-Tracking in Linear RNNs Through Negative Eigenvalues
Riccardo Grazzi, Julien Siems, Arber Zela +3
Linear Recurrent Neural Networks (LRNNs) such as Mamba, RWKV, GLA, mLSTM, and DeltaNet have emerged as efficient alternatives to Transformers for long sequences. However, both Tran…
Nonsmooth Implicit Differentiation: Deterministic and Stochastic Convergence Rates
Riccardo Grazzi, Massimiliano Pontil, Saverio Salzo
We study the problem of efficiently computing the derivative of the fixed-point of a parametric nondifferentiable contraction map. This problem has wide applications in machine lea…
Is Mamba Capable of In-Context Learning?
Riccardo Grazzi, Julien Siems, Simon Schrodi +2
State of the art foundation models such as GPT-4 perform surprisingly well at in-context learning (ICL), a variant of meta-learning concerning the learned ability to solve tasks du…
Learning invariant representations of time-homogeneous stochastic dynamical systems
Vladimir R. Kostic, Pietro Novelli, Riccardo Grazzi +2
We consider the general class of time-homogeneous stochastic dynamical systems, both discrete and continuous, and study the problem of learning a representation of the state that f…
Group Meritocratic Fairness in Linear Contextual Bandits
Riccardo Grazzi, Arya Akhavan, John Isak Texas Falk +2
We study the linear contextual bandit problem where an agent has to select one candidate from a pool and each candidate belongs to a sensitive group. In this setting, candidates' r…
Bilevel Optimization with a Lower-level Contraction: Optimal Sample Complexity without Warm-start
Riccardo Grazzi, Massimiliano Pontil, Saverio Salzo
We analyse a general class of bilevel problems, in which the upper-level problem consists in the minimization of a smooth objective function and the lower-level problem is to find…
Meta-Forecasting by combining Global Deep Representations with Local Adaptation
Riccardo Grazzi, Valentin Flunkert, David Salinas +3
While classical time series forecasting considers individual time series in isolation, recent advances based on deep learning showed that jointly learning from a large pool of rela…
Convergence Properties of Stochastic Hypergradients
Riccardo Grazzi, Massimiliano Pontil, Saverio Salzo
Bilevel optimization problems are receiving increasing attention in machine learning as they provide a natural framework for hyperparameter optimization and meta-learning. A key st…
On the Iteration Complexity of Hypergradient Computation
Riccardo Grazzi, Luca Franceschi, Massimiliano Pontil +1
We study a general class of bilevel problems, consisting in the minimization of an upper-level objective which depends on the solution to a parametric fixed-point equation. Importa…
Learning-to-Learn Stochastic Gradient Descent with Biased Regularization
Giulia Denevi, Carlo Ciliberto, Riccardo Grazzi +1
We study the problem of learning-to-learn: inferring a learning algorithm that works well on tasks sampled from an unknown distribution. As class of algorithms we consider Stochast…
Bilevel Programming for Hyperparameter Optimization and Meta-Learning
Luca Franceschi, Paolo Frasconi, Saverio Salzo +2
We introduce a framework based on bilevel programming that unifies gradient-based hyperparameter optimization and meta-learning. We show that an approximate version of the bilevel…
Far-HO: A Bilevel Programming Package for Hyperparameter Optimization and Meta-Learning
Luca Franceschi, Riccardo Grazzi, Massimiliano Pontil +2
In (Franceschi et al., 2018) we proposed a unified mathematical framework, grounded on bilevel programming, that encompasses gradient-based hyperparameter optimization and meta-lea…