13 citations · 44 across the 13 of their papers we have counts for
17 papers
TIPS: A Tiny Inference-time Policy for Scaling in Diffusion Language Models
Ya-Shi Zhang, Jian Tang, Pierre-André Noël +1
Masked diffusion language models expose an inference-time choice: which positions become visible after each denoising pass. Many existing decoding methods use hand-designed rules o…
Fixed State, Long Reach: What a Constant-Size Cache Buys Block Diffusion at Scale
Vaibhav Singh, Pierre-André Noël, Torsten Scholak +2
Diffusion language models decode tokens in parallel, but their bidirectional denoiser rules out the naive key--value (KV) cache behind fast autoregressive inference. Block diffusio…
Apriel-H1: Towards Efficient Enterprise Reasoning Models
Oleksiy Ostapenko, Luke Kumar, Raymond Li +10
Large Language Models (LLMs) achieve remarkable reasoning capabilities through transformer architectures with attention mechanisms. However, transformers suffer from quadratic time…
DiffuMamba: High-Throughput Diffusion LMs with Mamba Backbone
Vaibhav Singh, Oleksiy Ostapenko, Pierre-André Noël +2
Diffusion language models (DLMs) have emerged as a promising alternative to autoregressive (AR) generation, yet their reliance on Transformer backbones limits inference efficiency…
Apriel-Nemotron-15B-Thinker
Shruthan Radhakrishna, Soham Parikh, Gopal Sarda +32
While large language models (LLMs) have achieved remarkable reasoning capabilities across domains like code, math and other enterprise tasks, their significant memory and computati…
Using Scaling Laws for Data Source Utility Estimation in Domain-Specific Pre-Training
Oleksiy Ostapenko, Charles Guille-Escuret, Luke Kumar +7
We introduce a framework for optimizing domain-specific dataset construction in foundation model training. Specifically, we seek a cost-efficient way to estimate the quality of dat…
Exploring Sparse Adapters for Scalable Merging of Parameter Efficient Experts
Samin Yeasar Arnob, Zhan Su, Minseon Kim +6
Merging parameter-efficient task experts has recently gained growing attention as a way to build modular architectures that can be rapidly adapted on the fly for specific downstrea…
Towards Modular LLMs by Building and Reusing a Library of LoRAs
Oleksiy Ostapenko, Zhan Su, Edoardo Maria Ponti +5
The growing number of parameter-efficient adaptations of a base large language model (LLM) calls for studying whether we can reuse such trained adapters to improve performance for…
Guiding Language Model Reasoning with Planning Tokens
Xinyi Wang, Lucas Caccia, Oleksiy Ostapenko +3
Large language models (LLMs) have recently attracted considerable interest for their ability to perform complex reasoning tasks, such as chain-of-thought (CoT) reasoning. However,…
Continual Learning with Foundation Models: An Empirical Study of Latent Replay
Oleksiy Ostapenko, Timothee Lesort, Pau Rodríguez +4
Rapid development of large-scale pre-training has resulted in foundation models that can act as effective feature extractors on a variety of downstream tasks and domains. Motivated…
Challenging Common Assumptions about Catastrophic Forgetting
Timothée Lesort, Oleksiy Ostapenko, Diganta Misra +4
Building learning agents that can progressively learn and accumulate knowledge is the core goal of the continual learning (CL) research field. Unfortunately, training a model on ne…
Continual Learning via Local Module Composition
Oleksiy Ostapenko, Pau Rodriguez, Massimo Caccia +1
Modularity is a compelling solution to continual learning (CL), the problem of modeling sequences of related tasks. Learning and then composing modules to solve different tasks pro…
Sequoia: A Software Framework to Unify Continual Learning Research
Fabrice Normandin, Florian Golemo, Oleksiy Ostapenko +10
The field of Continual Learning (CL) seeks to develop algorithms that accumulate knowledge and skills over time through interaction with non-stationary environments. In practice, a…
Online Fast Adaptation and Knowledge Accumulation: a New Approach to Continual Learning
Massimo Caccia, Pau Rodriguez, Oleksiy Ostapenko +8
Continual learning studies agents that learn from streams of tasks without forgetting previous ones while adapting to new ones. Two recent continual-learning scenarios have opened…
Pruning at a Glance: Global Neural Pruning for Model Compression
Abdullah Salama, Oleksiy Ostapenko, Tassilo Klein +1
Deep Learning models have become the dominant approach in several areas due to their high performance. Unfortunately, the size and hence computational requirements of operating suc…
Learning to Remember: A Synaptic Plasticity Driven Framework for Continual Learning
Oleksiy Ostapenko, Mihai Puscas, Tassilo Klein +2
Models trained in the context of continual learning (CL) should be able to learn from a stream of data over an undefined period of time. The main challenges herein are: 1) maintain…
Self Paced Adversarial Training for Multimodal Few-shot Learning
Frederik Pahde, Oleksiy Ostapenko, Patrick Jähnichen +2
State-of-the-art deep learning algorithms yield remarkable results in many visual recognition tasks. However, they still fail to provide satisfactory results in scarce data regimes…