4 papers
How much pretraining data do language models need to learn syntax?
Laura Pérez-Mayos, Miguel Ballesteros, Leo Wanner
Transformers-based pretrained language models achieve outstanding results in many well-known NLU benchmarks. However, while pretraining methods are very convenient, they are expens…
Assessing the Syntactic Capabilities of Transformer-based Multilingual Language Models
Laura Pérez-Mayos, Alba Táboas García, Simon Mille +1
Multilingual Transformer-based language models, usually pretrained on more than 100 languages, have been shown to achieve outstanding results in a wide range of cross-lingual trans…
On the Evolution of Syntactic Information Encoded by BERT's Contextualized Representations
Laura Pérez-Mayos, Roberto Carlini, Miguel Ballesteros +1
The adaptation of pretrained language models to solve supervised tasks has become a baseline in NLP, and many recent works have focused on studying how linguistic information is en…
Concept Extraction Using Pointer-Generator Networks
Alexander Shvets, Leo Wanner
Concept extraction is crucial for a number of downstream applications. However, surprisingly enough, straightforward single token/nominal chunk-concept alignment or dictionary look…