5 papers
Consensus as Privileged Context for Label-Free Self-Distillation
John Gkountouras, Josip Jukić, Ivan Titov
Sampling multiple solutions and returning the majority answer is among the most reliable ways to improve the reasoning accuracy of large language models without labels, and a growi…
Geometric Self-Distillation for Reasoning Generalization
Josip Jukić, Ivan Titov
On-policy distillation provides dense teacher supervision on a language model's own trajectories. In self-distillation with privileged context, this supervision comes from the mode…
Context Parametrization with Compositional Adapters
Josip Jukić, Martin Tutek, Jan Šnajder
Large language models (LLMs) often seamlessly adapt to new tasks through in-context learning (ICL) or supervised fine-tuning (SFT). However, ICL is inefficient when handling many d…
Improving Data and Parameter Efficiency of Neural Language Models Using Representation Analysis
Josip Jukić
This thesis addresses challenges related to data and parameter efficiency in neural language models, with a focus on representation analysis and the introduction of new optimizatio…
Disentangling Latent Shifts of In-Context Learning with Weak Supervision
Josip Jukić, Jan Šnajder
In-context learning (ICL) enables large language models to perform few-shot learning by conditioning on labeled examples in the prompt. Despite its flexibility, ICL suffers from in…