7 papers
Tensor Methods for Language Models: From Token Representation to Training, Adaptation, Inference, Compression, and Interpretability
Matvei Tarasov, Salman Ahmadi-Asl, Andre L. F. de Almeida +1
Large language models (LLMs) are built from structured high-dimensional objects such as token representations, weights, adaptation updates, caches, and activations, whose multiline…
Automatically Differentiable Nonlinear Tensor Networks (ADNTNs) for Exponential Parameter Compression of Deep Neural Networks
Andrzej Cichocki, Michal Wietczak
Large deep neural networks are costly to store and deploy because inference must move and evaluate many parameters. This paper studies \emph{Automatically Differentiable Nonlinear…
Group Entropies and Mirror Duality: A Class of Flexible Mirror Descent Updates for Machine Learning
Andrzej Cichocki, Piergiulio Tempesta
We introduce a comprehensive theoretical and algorithmic framework that bridges formal group theory and group entropies with modern machine learning, paving the way for an infinite…
Mirror Descent Using the Tempesta Generalized Multi-parametric Logarithms
Andrzej Cichocki
In this paper, we develop a wide class Mirror Descent (MD) algorithms, which play a key role in machine learning. For this purpose we formulated the constrained optimization proble…
Mirror Descent and Novel Exponentiated Gradient Algorithms Using Trace-Form Entropies and Deformed Logarithms
Andrzej Cichocki, Toshihisa Tanaka, Frank Nielsen +1
This paper introduces a broad class of Mirror Descent (MD) and Generalized Exponentiated Gradient (GEG) algorithms derived from trace-form entropies defined via deformed logarithms…
Generalized Euler Logarithm and its Applications in Machine Learning: Natural Gradient, Backpropagation, Generalized EG, Mirror Descent and OLPS
Andrzej Cichocki
This paper investigates in depth the fundamental properties of the two-parameter generalized Euler logarithm and its inverse, the associated deformed -exponential function.…
Generalized Exponentiated Gradient Algorithms and Their Application to On-Line Portfolio Selection
Andrzej Cichocki, Sergio Cruces, Auxiliadora Sarmiento +1
This paper introduces a novel family of generalized exponentiated gradient (EG) updates derived from an Alpha-Beta divergence regularization function. Collectively referred to as E…