From the 1 of 5 linked papers with an AI index.
5 papers
Consensus as Privileged Context for Label-Free Self-Distillation
John Gkountouras, Josip JukiÄ, Ivan Titov
The paper introduces CANON, a label‑free self‑distillation method that uses the majority answer from multiple sampled solutions as dense token‑level supervision to improve reasonin…
Geometric Self-Distillation for Reasoning Generalization
Josip JukiÄ, Ivan Titov
On-policy distillation is a practical post-training recipe for large language models, supplying dense teacher supervision on the student's own trajectories. In privileged-context s…
Context Parametrization with Compositional Adapters
Josip JukiÄ, Martin Tutek, Jan Å najder
Large language models (LLMs) often seamlessly adapt to new tasks through in-context learning (ICL) or supervised fine-tuning (SFT). However, ICL is inefficient when handling many d…
Disentangling Latent Shifts of In-Context Learning with Weak Supervision
Josip JukiÄ, Jan Å najder
In-context learning (ICL) enables large language models to perform few-shot learning by conditioning on labeled examples in the prompt. Despite its flexibility, ICL suffers from in…
Improving Data and Parameter Efficiency of Neural Language Models Using Representation Analysis
Josip JukiÄ
This thesis addresses challenges related to data and parameter efficiency in neural language models, with a focus on representation analysis and the introduction of new optimizatio…