5 papers
PARTREP: Learning What to Repeat for Decoder-only LLMs
Andikawati P Widjaja, Yongjun Kim, Hyounghun Kim +1
While decoder-only LLMs excel at a vast array of natural language tasks, it suffers from an asymmetric information flow induced by causal attention: later tokens are richer in cont…
ForecastBench-Sim: A Simulated-World Forecasting Benchmark
Jaeho Lee, Nick Merrill, Ezra Karger
Forecasting benchmarks for general-purpose AI systems usually inherit the constraints of the real world: outcomes resolve slowly, tail events are rare, and counterfactual questions…
Beware of the Batch Size: Hyperparameter Bias in Evaluating LoRA
Sangyoon Lee, Jaeho Lee
Low-rank adaptation (LoRA) is a standard approach for fine-tuning large language models, yet its many variants report conflicting empirical gains, often on the same benchmarks. We…
Is Capability a Liability? More Capable Language Models Make Worse Forecasts When It Matters Most
Nick Merrill, Jaeho Lee, Ezra Karger
We document inverse scaling in LLMs on forecasting problems whose underlying time series exhibit superlinear growth and tail risk of regime change, a structure common in finance an…
eMamba: Efficient Acceleration Framework for Mamba Models in Edge Computing
Jiyong Kim, Jaeho Lee, Jiahao Lin +4
State Space Model (SSM)-based machine learning architectures have recently gained significant attention for processing sequential data. Mamba, a recent sequence-to-sequence SSM, of…