5 papers
Not All Drift Is a Mechanism: Tracing Prompt Adaptation in LLMs
Mikhail Aleksandrov, Nail Bashirov, Maksim Ivanov +6
Prompt-based adaptation offers several ways to specialize a frozen language model, ranging from natural-language prompt optimization to learned continuous prompts. These methods ar…
Monarch-Muon: Structure-Aware Muon for Fast and Memory-Efficient Pretraining
Alexander Miasnikov, Ruslan Kabirov, Gleb Molodtsov +7
Training large language models is constrained by both wall-clock time and memory footprint, and recent matrix-aware optimizers sharpen this tradeoff: Muon can improve optimization…
Rethinking the Role of Tensor Decompositions in Post-Training LLM Compression
Artur Zagitov, Alexander Miasnikov, Maxim Krutikov +5
Post-training compression is essential for deploying large language models (LLMs) under tight resource constraints. Tensor decompositions have emerged as a promising direction, off…
Adaptive Regularized Newton Method with Inexact Hessian
Aleksandr Shestakov, Nail Bashirov, Andrei Semenov +4
Newton's method is the most widespread high-order method, demanding the gradient and the Hessian of the objective function. However, one of the main disadvantages of Newtons method…
The Black-Box Optimization Problem: Zero-Order Accelerated Stochastic Method via Kernel Approximation
Aleksandr Lobanov, Nail Bashirov, Alexander Gasnikov
In this paper, we study the standard formulation of an optimization problem when the computation of gradient is not available. Such a problem can be classified as a "black box" opt…