8 citations · 13 across the 28 of their papers we have counts for
31 papers
Diffuse AI Control on Fuzzy Tasks
Mikhail Terekhov, Caglar Gulcehre, Vivek Hebbar +1
AI models deployed in critical domains, such as AI safety research, may subtly sabotage our efforts due to misalignment. Diffuse AI Control is a subfield of AI safety concerned wit…
BlockGen: Flexible Blockwise Sequence Modeling with Hybrid Samplers
Justin Deschenaux, Caglar Gulcehre
Is the uniform-state diffusion framework a more powerful paradigm for discrete diffusion? Recent studies indicate that this may be the case. In combination with predictor-corrector…
Language Modeling with Hyperspherical Flows
Justin Deschenaux, Caglar Gulcehre
Discrete Diffusion Language Models progressed rapidly as an alternative to autoregressive (AR) models, motivated by their parallel generation abilities. However, for tractability,…
Augmenting Attention with Exponentially Decaying Memory Improves Query-Aware KV Sparsity
Xiuying Wei, Caglar Gulcehre
Efficient inference is critical for long-context language models, where attention computation and KV-cache access dominate the cost. Recent work RAT+, introduces a recurrence-augme…
The Future of Facts: Tracing the Factual Generation-Verification Gap
Tim R. Davidson, Anja Surina, Caglar Gulcehre
Language models are becoming the default interface to factual knowledge, yet they often verify outputs more reliably than they generate them. This generation-verification gap (GV-g…
The Diffusion Duality, Chapter II: -Samplers
Justin Deschenaux, Caglar Gulcehre, Subham Sekhar Sahoo
Uniform-state discrete diffusion models excel at few-step generation and guidance due to their ability to self-correct, making them preferred over autoregressive or Masked diffusio…
RAT+: Train Dense, Infer Sparse -- Recurrence Augmented Attention for Dilated Inference
Xiuying Wei, Caglar Gulcehre
Structured dilated attention has an appealing inference-time efficiency knob: it reduces the FLOPs of attention and the KV cache size by a factor of the dilation size D, while pres…
Loopholing Discrete Diffusion: Deterministic Bypass of the Sampling Wall
Mingyu Jo, Jaesik Yoon, Justin Deschenaux +2
Discrete diffusion models offer a promising alternative to autoregressive generation through parallel decoding, but they suffer from a sampling wall: once categorical sampling occu…
Adaptive Attacks on Trusted Monitors Subvert AI Control Protocols
Mikhail Terekhov, Alexander Panfilov, Daniil Dzenhaliou +4
AI control protocols serve as a defense mechanism to stop untrusted LLM agents from causing harm in autonomous settings. Prior work treats this as a security problem, stress testin…
Apertus: Democratizing Open and Compliant LLMs for Global Language Environments
Project Apertus, Alejandro Hernández-Cano, Alexander Hägele +100
We present Apertus, a fully open suite of large language models (LLMs) designed to address two systemic shortcomings in today's open model ecosystem: data compliance and multilingu…
Quantile Reward Policy Optimization: Alignment with Pointwise Regression and Exact Partition Functions
Simon Matrenok, Skander Moalla, Caglar Gulcehre
Aligning large language models with pointwise absolute rewards has so far required online, on-policy algorithms such as PPO and GRPO. In contrast, simpler methods that can leverage…
RAT: Bridging RNN Efficiency and Attention Accuracy via Chunk-based Sequence Modeling
Xiuying Wei, Anunay Yadav, Razvan Pascanu +1
Transformers have become the cornerstone of modern large-scale language models, but their reliance on softmax attention poses a computational bottleneck at both training and infere…
Control Tax: The Price of Keeping AI in Check
Mikhail Terekhov, Zhen Ning David Liu, Caglar Gulcehre +1
The rapid integration of agentic AI into high-stakes real-world applications requires robust oversight mechanisms. The emerging field of AI Control (AIC) aims to provide such an ov…
Partition Generative Modeling: Masked Modeling Without Masks
Justin Deschenaux, Lan Tran, Caglar Gulcehre
Masked generative models (MGMs) can generate tokens in parallel and in any order, unlike autoregressive models (ARMs), which decode one token at a time, left-to-right. However, MGM…
Algorithm Discovery With LLMs: Evolutionary Search Meets Reinforcement Learning
Anja Surina, Amin Mansouri, Lars Quaedvlieg +4
Discovering efficient algorithms for solving complex problems has been an outstanding challenge in mathematics and computer science, requiring substantial human expertise over the…
Regret-Optimized Portfolio Enhancement through Deep Reinforcement Learning and Future Looking Rewards
Daniil Karzanov, Rubén Garzón, Mikhail Terekhov +3
This paper introduces a novel agent-based approach for enhancing existing portfolio strategies using Proximal Policy Optimization (PPO). Rather than focusing solely on traditional…
From Markov to Laplace: How Mamba In-Context Learns Markov Chains
Marco Bondaschi, Nived Rajaraman, Xiuying Wei +5
While transformer-based language models have driven the AI revolution thus far, their computational complexity has spurred growing interest in viable alternatives, such as structur…
SIKeD: Self-guided Iterative Knowledge Distillation for mathematical reasoning
Shivam Adarsh, Kumar Shridhar, Caglar Gulcehre +2
Large Language Models (LLMs) can transfer their reasoning skills to smaller models by teaching them to generate the intermediate reasoning process required to solve multistep reaso…
One-Step is Enough: Sparse Autoencoders for Text-to-Image Diffusion Models
Viacheslav Surkov, Chris Wendler, Antonio Mari +5
For large language models (LLMs), sparse autoencoders (SAEs) have been shown to decompose intermediate representations that often are not interpretable directly into sparse sums of…
Beyond Autoregression: Fast LLMs via Self-Distillation Through Time
Justin Deschenaux, Caglar Gulcehre
Autoregressive (AR) Large Language Models (LLMs) have demonstrated significant success across numerous tasks. However, the AR modeling paradigm presents certain limitations; for in…
The Role of Deep Learning Regularizations on Actors in Offline RL
Denis Tarasov, Anja Surina, Caglar Gulcehre
Deep learning regularization techniques, such as dropout, layer normalization, or weight decay, are widely adopted in the construction of modern artificial neural networks, often r…
Investigating Low-Rank Training in Transformer Language Models: Efficiency and Scaling Analysis
Xiuying Wei, Skander Moalla, Razvan Pascanu +1
State-of-the-art LLMs often rely on scale with high computational costs, which has sparked a research agenda to reduce parameter counts and costs without significantly impacting pe…
Promises, Outlooks and Challenges of Diffusion Language Modeling
Justin Deschenaux, Caglar Gulcehre
The modern autoregressive Large Language Models (LLMs) have achieved outstanding performance on NLP benchmarks, and they are deployed in the real world. However, they still suffer…
HiPPO-Prophecy: State-Space Models can Provably Learn Dynamical Systems in Context
Federico Arangath Joseph, Kilian Konstantin Haefeli, Noah Liniger +1
This work explores the in-context learning capabilities of State Space Models (SSMs) and presents, to the best of our knowledge, the first theoretical explanation of a possible und…