9 papers
How Depth Growth and Looping Shape In-Context Learning in Language Models
Ferdinand Kapl, Kaitlin Maile, Stefan Bauer +1
Looped and depth-grown language models reuse computations across depth and have attracted interest for their reasoning capabilities. Building on prior suggestive evidence of improv…
Growing to See: Efficient Pretraining of Self-Supervised Vision Transformers
Amir Mohammad Karimi Mamaghan, Ferdinand Kapl, Andrea Dittadi +4
Progressive depth growth builds large networks by starting with shallow models and inserting layers during training. Recent advances in large language models have renewed interest…
Amortised Inference through One-Step Implicit Sampling
Vincent Pauline, Kirill Tamogashev, Arran Carter +3
Efficient sampling from Bayesian posterior distributions in high dimensions remains a hard problem, despite advances in theory and algorithms. Multiple steps are usually required t…
Are Object-Centric Representations Better At Compositional Generalization?
Ferdinand Kapl, Amir Mohammad Karimi Mamaghan, Maximilian Seitzer +4
Compositional generalization, the ability to reason about novel combinations of familiar concepts, is fundamental to human cognition and a critical challenge for machine learning.…
From Growing to Looping: A Unified View of Iterative Computation in LLMs
Ferdinand Kapl, Emmanouil Angelis, Kaitlin Maile +2
Looping, reusing a block of layers across depth, and depth growing, training shallow-to-deep models by duplicating middle layers, have both been linked to stronger reasoning, but t…
Do Depth-Grown Models Overcome the Curse of Depth? An In-Depth Analysis
Ferdinand Kapl, Emmanouil Angelis, Tobias Höppe +4
Gradually growing the depth of Transformers during training can not only reduce training cost but also lead to improved reasoning performance, as shown by MIDAS (Saunshi et al., 20…
Foundations of Diffusion Models in General State Spaces: A Self-Contained Introduction
Vincent Pauline, Tobias Höppe, Kirill Neklyudov +3
Although diffusion models now occupy a central place in generative modeling, introductory treatments commonly assume Euclidean data and seldom clarify their connection to discrete-…
Breaking the Likelihood-Quality Trade-off in Diffusion Models by Merging Pretrained Experts
Yasin Esfandiari, Stefan Bauer, Sebastian U. Stich +1
Diffusion models for image generation often exhibit a trade-off between perceptual sample quality and data likelihood: training objectives emphasizing high-noise denoising steps yi…
Jasmine: A Simple, Performant and Scalable JAX-based World Modeling Codebase
Mihir Mahajan, Alfred Nguyen, Franz Srambical +1
While world models are increasingly positioned as a pathway to overcoming data scarcity in domains such as robotics, open training infrastructure for world modeling remains nascent…