4 papers
LeRoPE: Learnable RoPE Frequencies Improve Language Modeling
Petros Karypis, Sean O'Brien, Shreyas Kadekodi +2
Rotary Positional Encodings (RoPE) are currently the most popular positional encodings used in modern language models. RoPE rotates two-dimensional chunks of query and key vectors,…
Fine-Tuning Language Models on Multiple Datasets for Citation Intention Classification
Zeren Shui, Petros Karypis, Daniel S. Karls +4
Citation intention Classification (CIC) tools classify citations by their intention (e.g., background, motivation) and assist readers in evaluating the contribution of scientific l…
Extending Input Contexts of Language Models through Training on Segmented Sequences
Petros Karypis, Julian McAuley, George Karypis
Effectively training language models on long inputs poses many technical challenges. As a cost consideration, languages models are pretrained on a fixed sequence length before bein…
Pack of LLMs: Model Fusion at Test-Time via Perplexity Optimization
Costas Mavromatis, Petros Karypis, George Karypis
Fusing knowledge from multiple Large Language Models (LLMs) can combine their diverse strengths to achieve improved performance on a given task. However, current fusion approaches…