3 citations · 7 across the 16 of their papers we have counts for
17 papers
Representation-Space MMD for Diffusion Language Models
Ilya Drobyshevskiy, Ilia Sudakov, Maksim Semenov +7
We introduce a post-training method for diffusion language models (DLMs) that minimizes Maximum Mean Discrepancy (MMD) between generated and reference distributions in the feature…
Unstable Features, Reproducible Subspaces: Understanding Seed Dependence in Sparse Autoencoders
Gleb Gerasimov, Timofei Rusalev, Nikita Balagansky +3
Sparse autoencoders (SAEs) are widely used to interpret neural network representations, but their utility depends on whether the learned features are reproducible across training r…
Interpreting and Steering a Text-to-Speech Language Model with Sparse Autoencoders
Nikita Koriagin, Georgii Aparin, Nikita Balagansky +1
Language models increasingly serve as the backbone of text-to-speech (TTS) systems, yet we understand little about the representations they build when text and generated speech tok…
Guided Star-Shaped Masked Diffusion
Viacheslav Meshchaninov, Egor Shibaev, Artem Makoian +5
The performance of pre-trained masked diffusion models is often constrained by their sampling procedure, which makes decisions irreversible and struggles in low-step generation reg…
Kronecker Factorization Improves Efficiency and Interpretability of Sparse Autoencoders
Vadim Kurochkin, Yaroslav Aksenov, Daniil Laptev +2
Sparse Autoencoders (SAEs) decompose language-model activations into sparse, interpretable features, but standard encoders usually treat the latent dictionary as a flat set of inde…
Steering LLM Reasoning Through Bias-Only Adaptation
Viacheslav Sinii, Alexey Gorbatovski, Artem Cherepanov +3
We show that training a single -dimensional steering vector per layer with reinforcement learning, while freezing all base weights, matches the accuracy of fully RL-tuned reason…
You Do Not Fully Utilize Transformer's Representation Capacity
Gleb Gerasimov, Yaroslav Aksenov, Nikita Balagansky +2
In contrast to RNNs, which compress their history into a single hidden state, Transformers can attend to all past tokens directly. However, standard Transformers rely solely on the…
Analyze Feature Flow to Enhance Interpretation and Steering in Language Models
Daniil Laptev, Nikita Balagansky, Yaroslav Aksenov +1
We introduce a new approach to systematically map features discovered by sparse autoencoder across consecutive layers of large language models, extending earlier work that examined…
Mechanistic Permutability: Match Features Across Layers
Nikita Balagansky, Ian Maksimov, Daniil Gavrilov
Understanding how features evolve across layers in deep neural networks is a fundamental challenge in mechanistic interpretability, particularly due to polysemanticity and feature…
Learn Your Reference Model for Real Good Alignment
Alexey Gorbatovski, Boris Shaposhnikov, Alexey Malakhov +5
Despite the fact that offline methods for Large Language Models (LLMs) alignment do not require a direct reward model, they remain susceptible to overoptimization. This issue arise…
Linear Transformers with Learnable Kernel Functions are Better In-Context Models
Yaroslav Aksenov, Nikita Balagansky, Sofia Maria Lo Cicero Vaina +3
Advancing the frontier of subquadratic architectures for Language Models (LMs) is crucial in the rapidly evolving field of natural language processing. Current innovations, includi…
Ahead-of-Time P-Tuning
Daniil Gavrilov, Nikita Balagansky
In this paper, we propose Ahead-of-Time (AoT) P-Tuning, a novel parameter-efficient fine-tuning method for pre-trained Language Models (LMs) that adds input-dependent bias before e…
Diffusion Language Models Generation Can Be Halted Early
Sofia Maria Lo Cicero Vaina, Nikita Balagansky, Daniil Gavrilov
Diffusion Language models (DLMs) are a promising avenue for text generation due to their practical properties on tractable controllable generation. They also have the advantage of…
Linear Interpolation In Parameter Space is Good Enough for Fine-Tuned Language Models
Mark Rofin, Nikita Balagansky, Daniil Gavrilov
The simplest way to obtain continuous interpolation between two points in high dimensional space is to draw a line between them. While previous works focused on the general connect…
Classifiers are Better Experts for Controllable Text Generation
Askhat Sitdikov, Nikita Balagansky, Daniil Gavrilov +1
This paper proposes a simple method for controllable text generation based on weighting logits with a free-form classifier, namely CAIF sampling. Using an arbitrary text classifier…
PALBERT: Teaching ALBERT to Ponder
Nikita Balagansky, Daniil Gavrilov
Currently, pre-trained models can be considered the default choice for a wide range of NLP tasks. Despite their SoTA results, there is practical evidence that these models may requ…
Weight Squeezing: Reparameterization for Knowledge Transfer and Model Compression
Artem Chumachenko, Daniil Gavrilov, Nikita Balagansky +1
In this work, we present a novel approach for simultaneous knowledge transfer and model compression called Weight Squeezing. With this method, we perform knowledge transfer from a…