7 papers
Language Models Learn Universal Representations of Numbers and Here's Why You Should Care
Michal Å tefánik, Timothee Mickus, Marek KadlÄÃk +7
Prior work has shown that large language models (LLMs) often converge to accurate input embedding for numbers, based on sinusoidal representations. In this work, we quantify that t…
Pre-trained Language Models Learn Remarkably Accurate Representations of Numbers
Marek KadlÄÃk, Michal Å tefánik, Timothee Mickus +2
Pretrained language models (LMs) are prone to arithmetic errors. Existing work showed limited success in probing numeric values from models' representations, indicating that these…
Can Out-of-Distribution Evaluations Uncover Reliance on Shortcuts? A Case Study in Question Answering
Michal Å tefánik, Timothee Mickus, Marek KadlÄÃk +2
A majority of recent work in AI assesses models' generalization capabilities through the lens of performance on out-of-distribution (OOD) datasets. Despite their practicality, such…
Attend or Perish: Benchmarking Attention in Algorithmic Reasoning
Michal Spiegel, Michal Å tefánik, Marek KadlÄÃk +1
Can transformers learn to perform algorithmic tasks reliably across previously unseen input/output domains? While pre-trained language models show solid accuracy on benchmarks inco…
VectorEdits: A Dataset and Benchmark for Instruction-Based Editing of Vector Graphics
Josef KuchaÅ, Marek KadlÄÃk, Michal Spiegel +1
We introduce a large-scale dataset for instruction-guided vector image editing, consisting of over 270,000 pairs of SVG images paired with natural language edit instructions. Our d…
Negation: A Pink Elephant in the Large Language Models' Room?
Tereza Vrabcová, Marek KadlÄÃk, Petr Sojka +2
Negations are key to determining sentence meaning, making them essential for logical reasoning. Despite their importance, negations pose a substantial challenge for large language…