4 citations · 28 across the 86 of their papers we have counts for
21 papers · 1 filter
From Stories to Features: Scaling Concept Vectors for AI Alignment & Interpretability
Dmitry Scherbakov, Oleg Chernikov, Victoria Khamash +5
We study feature vectors: directions in a language model's activation space that correspond to human-interpretable concepts, such as honesty or deception. Existing unsupervised met…
AI Consilium: a Benchmark of LLM Clinical Multidisciplinary Second Opinions
Alexey A. Bokov, Ivan I. Panin, Nikolay B. Nechaev +10
Large language models (LLMs) can generate differential diagnoses for complex cases, but a useful second opinion must also be factually grounded, clinically coherent, actionable, an…
E-MoE: Enhanced Mixture-of-Experts for Non-Factorized Diffusion Language Models
Arseny Ivanov, Alexander Kolesov, Alexander Korotin +2
Masked diffusion models (MDMs) generate sequences by progressively unmasking several tokens per denoising step, but their reverse process is typically factorized over positions, li…
LANTERN: Illuminating Hidden Mathematical Knowledge in Language Models
Pavel Tikhonov, Elena Tutubalina, Ivan Oseledets +2
Language models can now prove theorems, but people still decide which problems to pursue. We ask whether a model's internal representations can help identify promising mathematical…
Your Transformer Can Hold Two Thoughts at Once: Evidence of Linear Superposition in LLMs
Pavel Tikhonov, Anton Korznikov, Matvey Mikhalchuk +6
While Large Language Models (LLMs) rely on highly non-linear components, in this work we demonstrate that they exhibit fundamental linearity: when inputs from distinct text streams…
Evaluating Losslessness in Speculative Decoding Under Finite-Precision Inference
Ilya Koziev, Leonid Sinev, Ivan Oseledets
Lossless speculative decoding is typically defined at the algorithmic level: a speculative procedure proposes multiple tokens and a verification procedure is designed to preserve t…
Structured Transforms for Low-Overhead Quantization of Language Models
Daria Cherniuk, Alexander Rudikov, Boris Kashin +1
We revisit Kashin-decomposition-based weight quantization for large language models and propose an improved algorithm with stronger convergence properties and structured, efficient…
LLM-Guided Evolution for Medical Decision Pipelines
Ivan Sviridov, Artem Oskin, Ivan Panin +4
Adapting large language models (LLMs) to clinical workflows often requires costly fine-tuning or manual prompt and pipeline engineering. We study LLM-guided MAP-Elites evolution as…
OCC-RAG: Optimal Cognitive Core for Faithful Question Answering
Maksim Savkin, Mikhail Goncharov, Alexander Gambashidze +7
Recent progress in the development of language models has been defined by scale, with each generation absorbing more of the world's knowledge into its weights. However, many practi…
CLARITY: Clinical Assistant for Routing, Inference, and Triage
Vladimir Shaposhnikov, Aleksandr Nesterov, Ilia Kopanichuk +7
We present CLARITY (Clinical Assistant for Routing, Inference and Triage), an AI-driven platform designed to facilitate patient-to-specialist routing, clinical consultations, and s…
Geopolitical biases in LLMs: what are the "good" and the "bad" countries according to contemporary language models
Mikhail Salnikov, Dmitrii Korzh, Ivan Lazichny +7
This paper evaluates geopolitical biases in LLMs with respect to various countries though an analysis of their interpretation of historical events with conflicting national perspec…
Confidence Is All You Need: Few-Shot RL Fine-Tuning of Language Models
Pengyi Li, Matvey Skripkin, Alexander Zubrey +2
Large language models (LLMs) excel at reasoning, yet post-training remains critical for aligning their behavior with task goals. Existing reinforcement learning (RL) methods often…
One Task Vector is not Enough: A Large-Scale Study for In-Context Learning
Pavel Tikhonov, Ivan Oseledets, Elena Tutubalina
In-context learning (ICL) enables Large Language Models (LLMs) to adapt to new tasks using few examples, with task vectors - specific hidden state activations - hypothesized to enc…
Exploring the Hidden Capacity of LLMs for One-Step Text Generation
Gleb Mezentsev, Ivan Oseledets
A recent study showed that large language models (LLMs) can reconstruct surprisingly long texts - up to thousands of tokens - via autoregressive generation from just one trained in…
On the Spatial Structure of Mixture-of-Experts in Transformers
Daniel Bershatsky, Ivan Oseledets
A common assumption is that MoE routers primarily leverage semantic features for expert selection. However, our study challenges this notion by demonstrating that positional token…
I Have Covered All the Bases Here: Interpreting Reasoning Features in Large Language Models via Sparse Autoencoders
Andrey Galichin, Alexey Dontsov, Polina Druzhinina +4
Recent LLMs like DeepSeek-R1 have demonstrated state-of-the-art performance by integrating deep thinking and complex reasoning during generation. However, the internal mechanisms b…
LLM-Microscope: Uncovering the Hidden Role of Punctuation in Context Memory of Transformers
Anton Razzhigaev, Matvey Mikhalchuk, Temurbek Rahmatullaev +4
We introduce methods to quantify how Large Language Models (LLMs) encode and store contextual information, revealing that tokens often seen as minor (e.g., determiners, punctuation…
Logit-KL Flow Matching: Non-Autoregressive Text Generation via Sampling-Hybrid Inference
Egor Sevriugov, Nikita Dragunov, Anton Razzhigaev +2
Non-autoregressive (NAR) language models offer notable efficiency in text generation by circumventing the sequential bottleneck of autoregressive decoding. However, accurately mode…
SparseGrad: A Selective Method for Efficient Fine-tuning of MLP Layers
Viktoriia Chekalina, Anna Rudenko, Gleb Mezentsev +3
The performance of Transformer models has been enhanced by increasing the number of parameters and the length of the processed text. Consequently, fine-tuning the entire model beco…
LoTR: Low Tensor Rank Weight Adaptation
Daniel Bershatsky, Daria Cherniuk, Talgat Daulbaev +2
In this paper we generalize and extend an idea of low-rank adaptation (LoRA) of large language models (LLMs) based on Transformer architecture. Widely used LoRA-like methods of fin…
The Shape of Learning: Anisotropy and Intrinsic Dimensions in Transformer-Based Models
Anton Razzhigaev, Matvey Mikhalchuk, Elizaveta Goncharova +3
In this study, we present an investigation into the anisotropy dynamics and intrinsic dimension of embeddings in transformer architectures, focusing on the dichotomy between encode…