Regress, Don't Guess -- A Regression-like Loss on Number Tokens for Language Models

Jonas Zausinger, Lars Pennig, Anamarija Kozina +13

While language models have exceptional capabilities at text generation, they lack a natural inductive bias for emitting numbers and thus struggle in tasks involving quantitative re…