6 papers
AION: Next-Generation Tasks and Practical Harness for Time Series
Tianxiang Zhan, Xiaobao Song, Tong Guan +2
Time series research is moving beyond fixed forecasting benchmarks toward realistic tasks that combine prediction, contextual reasoning, tool use, and structured decision support.…
On The Adaptation of Unlimiformer for Decoder-Only Transformers
Kian Ahrabian, Alon Benhaim, Barun Patra +3
One of the prominent issues stifling the current generation of large language models is their limited context length. Recent proprietary models such as GPT-4 and Claude 2 have intr…
POROver: Improving Safety and Reducing Overrefusal in Large Language Models with Overgeneration and Preference Optimization
Batuhan K. Karaman, Ishmam Zabir, Alon Benhaim +3
Achieving both high safety and high usefulness simultaneously in large language models has become a critical challenge in recent years.Models often exhibit unsafe behavior or adopt…
Scaling Laws for Multilingual Language Models
Yifei He, Alon Benhaim, Barun Patra +6
We propose a novel scaling law for general-purpose decoder-only language models (LMs) trained on multilingual data, tackling the problem of balancing languages during multilingual…
Scaling Optimal LR Across Token Horizons
Johan Bjorck, Alon Benhaim, Vishrav Chaudhary +2
State-of-the-art LLMs are powered by scaling -- scaling model size, dataset size and cluster size. It is economically infeasible to extensively tune hyperparameter for the largest…
ZO-AdaMU Optimizer: Adapting Perturbation by the Momentum and Uncertainty in Zeroth-order Optimization
Shuoran Jiang, Qingcai Chen, Youchen Pan +5
Lowering the memory requirement in full-parameter training on large models has become a hot research area. MeZO fine-tunes the large language models (LLMs) by just forward passes i…