8 citations · 18 across the 14 of their papers we have counts for
13 papers · 1 filter
Synthetic Pre-pretraining Survives Scale, but Not as a Grammatical Prior
Atsuki Yamaguchi, Tatsuro Inaba, Joel Niklaus +3
Pre-pretraining (PPT) on synthetic non-natural language data improves token efficiency during language model pre-training (PT). Prior work attributes this gain to a grammatical pri…
Edustories: A Collection of Real-world Case Studies from Classroom Practices
Michal Štefánik, Jan Nehyba, Jirina Karasova +4
Despite the widely recognized potential of AI in education, most prior work has focused on individualized student assistance. In contrast, the majority of educational practice worl…
Language Models Learn Universal Representations of Numbers and Here's Why You Should Care
Michal Štefánik, Timothee Mickus, Marek Kadlčík +7
Prior work has shown that large language models (LLMs) often converge to accurate input embedding for numbers, based on sinusoidal representations. In this work, we quantify that t…
Can Out-of-Distribution Evaluations Uncover Reliance on Shortcuts? A Case Study in Question Answering
Michal Štefánik, Timothee Mickus, Marek Kadlčík +2
A majority of recent work in AI assesses models' generalization capabilities through the lens of performance on out-of-distribution (OOD) datasets. Despite their practicality, such…
Pre-trained Language Models Learn Remarkably Accurate Representations of Numbers
Marek Kadlčík, Michal Štefánik, Timothee Mickus +2
Pretrained language models (LMs) are prone to arithmetic errors. Existing work showed limited success in probing numeric values from models' representations, indicating that these…
Negation: A Pink Elephant in the Large Language Models' Room?
Tereza Vrabcová, Marek Kadlčík, Petr Sojka +2
Negations are key to determining sentence meaning, making them essential for logical reasoning. Despite their importance, negations pose a substantial challenge for large language…
Self-training Language Models for Arithmetic Reasoning
Marek Kadlčík, Michal Štefánik
Recent language models achieve impressive results in tasks involving complex multistep reasoning, but scaling these capabilities further traditionally requires expensive collection…
Concept-aware Data Construction Improves In-context Learning of Language Models
Michal Štefánik, Marek Kadlčík, Petr Sojka
Many recent language models (LMs) are capable of in-context learning (ICL), manifested in the LMs' ability to perform a new task solely from natural-language instruction. Previous…
People and Places of Historical Europe: Bootstrapping Annotation Pipeline and a New Corpus of Named Entities in Late Medieval Texts
Vít Novotný, Kristýna Luger, Michal Štefánik +2
Although pre-trained named entity recognition (NER) models are highly accurate on modern corpora, they underperform on historical texts due to differences in language OCR errors. I…
Concept-aware Training Improves In-context Learning Ability of Language Models
Michal Štefánik, Marek Kadlčík
Many recent language models (LMs) of Transformers family exhibit so-called in-context learning (ICL) ability, manifested in the LMs' ability to modulate their function by a task de…
Methods for Estimating and Improving Robustness of Language Models
Michal Štefánik
Despite their outstanding performance, large language models (LLMs) suffer notorious flaws related to their preference for simple, surface-level textual relations over full semanti…
Regressive Ensemble for Machine Translation Quality Evaluation
Michal Štefánik, Vít Novotný, Petr Sojka
This work introduces a simple regressive ensemble for evaluating machine translation quality based on a set of novel and established metrics. We evaluate the ensemble using a corre…
When FastText Pays Attention: Efficient Estimation of Word Representations using Constrained Positional Weighting
Vít Novotný, Michal Štefánik, Eniafe Festus Ayetiran +2
In 2018, Mikolov et al. introduced the positional language model, which has characteristics of attention-based neural machine translation models and which achieved state-of-the-art…