204 citations · 437 across the 22 of their papers we have counts for
22 papers
MEDAL: A Framework for Benchmarking LLMs as Multilingual Open-Domain Dialogue Evaluators
John Mendonça, Alon Lavie, Isabel Trancoso
Evaluating the quality of open-domain chatbots has become increasingly reliant on LLMs acting as automatic judges. However, existing meta-evaluation benchmarks are static, outdated…
Soda-Eval: Open-Domain Dialogue Evaluation in the age of LLMs
John Mendonça, Isabel Trancoso, Alon Lavie
Although human evaluation remains the gold standard for open-domain dialogue evaluation, the growing popularity of automated evaluation using Large Language Models (LLMs) has also…
ECoh: Turn-level Coherence Evaluation for Multilingual Dialogues
John Mendonça, Isabel Trancoso, Alon Lavie
Despite being heralded as the new standard for dialogue evaluation, the closed-source nature of GPT-4 poses challenges for the community. Motivated by the need for lightweight, ope…
On the Benchmarking of LLMs for Open-Domain Dialogue Evaluation
John Mendonça, Alon Lavie, Isabel Trancoso
Large Language Models (LLMs) have showcased remarkable capabilities in various Natural Language Processing tasks. For automatic open-domain dialogue evaluation in particular, LLMs…
Improving Membership Inference in ASR Model Auditing with Perturbed Loss Features
Francisco Teixeira, Karla Pizzi, Raphael Olivier +3
Membership Inference (MI) poses a substantial privacy threat to the training data of Automatic Speech Recognition (ASR) systems, while also offering an opportunity to audit these m…
Dialogue Quality and Emotion Annotations for Customer Support Conversations
John Mendonça, Patrícia Pereira, Miguel Menezes +6
Task-oriented conversational datasets often lack topic variability and linguistic diversity. However, with the advent of Large Language Models (LLMs) pretrained on extensive, multi…
Privacy-oriented manipulation of speaker representations
Francisco Teixeira, Alberto Abad, Bhiksha Raj +1
Speaker embeddings are ubiquitous, with applications ranging from speaker recognition and diarization to speech synthesis and voice anonymisation. The amount of information held by…
Simple LLM Prompting is State-of-the-Art for Robust and Multilingual Dialogue Evaluation
John Mendonça, Patrícia Pereira, Helena Moniz +3
Despite significant research effort in the development of automatic dialogue evaluation metrics, little thought is given to evaluating dialogues other than in English. At the same…
Towards Multilingual Automatic Dialogue Evaluation
John Mendonça, Alon Lavie, Isabel Trancoso
The main limiting factor in the development of robust multilingual dialogue evaluation metrics is the lack of multilingual data and the limited availability of open sourced multili…
Privacy-preserving Automatic Speaker Diarization
Francisco Teixeira, Alberto Abad, Bhiksha Raj +1
Automatic Speaker Diarization (ASD) is an enabling technology with numerous applications, which deals with recordings of multiple speakers, raising special concerns in terms of pri…
Towards End-to-End Private Automatic Speaker Recognition
Francisco Teixeira, Alberto Abad, Bhiksha Raj +1
The development of privacy-preserving automatic speaker verification systems has been the focus of a number of studies with the intent of allowing users to authenticate themselves…
Using Self-Supervised Feature Extractors with Attention for Automatic COVID-19 Detection from Speech
John Mendonça, Rubén Solera-Ureña, Alberto Abad +1
The ComParE 2021 COVID-19 Speech Sub-challenge provides a test-bed for the evaluation of automatic detectors of COVID-19 from speech. Such models can be of value by providing test…
FoolHD: Fooling speaker identification by Highly imperceptible adversarial Disturbances
Ali Shahin Shamsabadi, Francisco Sepúlveda Teixeira, Alberto Abad +3
Speaker identification models are vulnerable to carefully designed adversarial perturbations of their input signals that induce misclassification. In this work, we propose a white-…
Pathological speech detection using x-vector embeddings
Catarina Botelho, Francisco Teixeira, Thomas Rolland +2
The potential of speech as a non-invasive biomarker to assess a speaker's health has been repeatedly supported by the results of multiple works, for both physical and psychological…
The GDPR & Speech Data: Reflections of Legal and Technology Communities, First Steps towards a Common Understanding
Andreas Nautsch, Catherine Jasserand, Els Kindt +3
Privacy preservation and the protection of speech data is in high demand, not least as a result of recent regulation, e.g. the General Data Protection Regulation (GDPR) in the EU.…
Assessing User Expertise in Spoken Dialog System Interactions
Eugénio Ribeiro, Fernando Batista, Isabel Trancoso +3
Identifying the level of expertise of its users is important for a system since it can lead to a better interaction through adaptation techniques. Furthermore, this information can…
Character-based Neural Machine Translation
Wang Ling, Isabel Trancoso, Chris Dyer +1
We introduce a neural machine translation model that views the input and output sentences as sequences of characters rather than words. Since word-level information provides a cruc…
Privacy-Preserving Multi-Document Summarization
Luís Marujo, José Portêlo, Wang Ling +6
State-of-the-art extractive multi-document summarization systems are usually designed without any concern about privacy issues, meaning that all documents are open to third parties…
Finding Function in Form: Compositional Character Models for Open Vocabulary Word Representation
Wang Ling, Tiago Luís, Luís Marujo +5
We introduce a model for constructing vector representations of words by composing characters using bidirectional LSTMs. Relative to traditional word representation models that hav…
Towards Using Machine Translation Techniques to Induce Multilingual Lexica of Discourse Markers
António Lopes, David Martins de Matos, Vera Cabarrão +4
Discourse markers are universal linguistic events subject to language variation. Although an extensive literature has already reported language specific traits of these events, lit…
Privacy-Preserving Important Passage Retrieval
Luis Marujo, José Portêlo, David Martins de Matos +5
State-of-the-art important passage retrieval methods obtain very good results, but do not take into account privacy issues. In this paper, we present a privacy preserving method th…
Automatic Fado Music Classification
Pedro Girão Antunes, David Martins de Matos, Ricardo Ribeiro +1
In late 2011, Fado was elevated to the oral and intangible heritage of humanity by UNESCO. This study aims to develop a tool for automatic detection of Fado music based on the audi…