9 papers
Large-Scale User Behavior Analysis in Multimodal AI-Assisted Manual Task Execution
Rafael Ferreira, Diogo Tavares, Diogo Glória-Silva +2
Conversational Task Assistants (CTAs) are multimodal dialogue systems that support users in complex real-world tasks such as cooking and DIY through voice, text, image, and video i…
AMALIA-VL: A Native European Portuguese Open-Source Vision and Language Model
Diogo Glória-Silva, João Cardeira, Manuel Letras da Luz +8
Large Vision and Language Models (LVLMs) have advanced rapidly, yet European Portuguese (pt-PT) remains systematically underserved by existing open-source multimodal models, which…
P3B3: A Multi-Turn Conversational Benchmark for Measuring European and Brazilian Portuguese Variety Bias in LLMs
Rafael Ferreira, Inês Vieira, Inês Calvo +6
As Large Language Models (LLMs) become embedded in everyday communication, capturing regional linguistic variation is essential for reliable and equitable language use. In Portugue…
PorTEXTO: A European Portuguese Benchmark for Visual Text Extraction
João Cardeira, Diogo Glória-Silva, Manuel Letras da Luz +4
European Portuguese (pt-PT) is largely absent from Optical Character Recognition (OCR) benchmarks, which skew toward high-resource languages. The few benchmarks that cover pt-PT fo…
VIGiA: Instructional Video Guidance via Dialogue Reasoning and Retrieval
Diogo Glória-Silva, David Semedo, João Maglhães
We introduce VIGiA, a novel multimodal dialogue model designed to understand and reason over complex, multi-step instructional video action plans. Unlike prior work which focuses m…
ALBA: A European Portuguese Benchmark for Evaluating Language and Linguistic Dimensions in Generative LLMs
Inês Vieira, Inês Calvo, Iago Paulo +6
As Large Language Models (LLMs) expand across multilingual domains, evaluating their performance in under-represented languages becomes increasingly important. European Portuguese…
AMALIA Technical Report: A Fully Open Source Large Language Model for European Portuguese
Afonso Simplício, Gonçalo Vinagre, Miguel Moura Ramos +19
Despite rapid progress in open large language models (LLMs), European Portuguese (pt-PT) remains underrepresented in both training data and native evaluation, with machine-translat…
RedTWIZ: Diverse LLM Red Teaming via Adaptive Attack Planning
Artur Horal, Daniel Pina, Henrique Paz +7
This paper presents the vision, scientific contributions, and technical details of RedTWIZ: an adaptive and diverse multi-turn red teaming framework, to audit the robustness of Lar…
Global PIQA: Evaluating Commonsense Reasoning Across 100+ Languages and Cultures
Tyler A. Chang, Catherine Arnett, Abdelrahman Sadallah +377
To date, there exist almost no culturally-specific evaluation benchmarks for large language models (LLMs) that cover a large number of languages and cultures. In this paper, we pre…