8 citations · 9 across the 5 of their papers we have counts for
5 papers
Data Repetition Beats Data Scaling in Long-CoT Supervised Fine-Tuning
Dawid J. Kopiczko, Sagar Vaze, Tijmen Blankevoort +1
Supervised fine-tuning (SFT) on chain-of-thought data is an essential post-training step for reasoning language models. Standard machine learning intuition suggests that training w…
What Layers When: Learning to Skip Compute in LLMs with Residual Gates
Filipe Laitenberger, Dawid Kopiczko, Cees G. M. Snoek +1
We introduce GateSkip, a simple residual-stream gating mechanism that enables token-wise layer skipping in decoder-only LMs. Each Attention/MLP branch is equipped with a sigmoid-li…
KV Cache Steering for Controlling Frozen LLMs
Max Belitsky, Dawid J. Kopiczko, Michael Dorkenwald +4
We propose cache steering, a lightweight method for implicit steering of language models via a one-shot intervention applied directly to the key-value cache. To validate its effect…
Bitune: Leveraging Bidirectional Attention to Improve Decoder-Only LLMs
Dawid J. Kopiczko, Tijmen Blankevoort, Yuki M. Asano
Decoder-only large language models typically rely solely on masked causal attention, which limits their expressiveness by restricting information flow to one direction. We propose…
VeRA: Vector-based Random Matrix Adaptation
Dawid J. Kopiczko, Tijmen Blankevoort, Yuki M. Asano
Low-rank adapation (LoRA) is a popular method that reduces the number of trainable parameters when finetuning large language models, but still faces acute storage challenges when s…