6 papers
Translation or Recitation? Calibrating Evaluation Scores for Machine Translation of Extremely Low-Resource Languages
Danlu Chen, Ka Sing He, Jiahe Tian +4
The landscape of extremely low-resource machine translation (MT) is characterized by perplexing variability in reported performance, often making results across different language…
MAEB: Massive Audio Embedding Benchmark
Adnan El Assadi, Isaac Chung, Chenghao Xiao +15
We introduce the Massive Audio Embedding Benchmark (MAEB), a large-scale benchmark covering 30 tasks across speech, music, environmental sounds, and cross-modal audio-text reasonin…
MMTEB: Massive Multilingual Text Embedding Benchmark
Kenneth Enevoldsen, Isaac Chung, Imene Kerboua +83
Text embeddings are typically evaluated on a limited set of tasks, which are constrained by language, domain, and task diversity. To address these limitations and provide a more co…
Quantifying Semantic Shift in Financial NLP: Robust Metrics for Market Prediction Stability
Zhongtian Sun, Chenghao Xiao, Anoushka Harit +1
Financial news is essential for accurate market prediction, but evolving narratives across macroeconomic regimes introduce semantic and causal drift that weaken model reliability.…
Analyzing LLMs' Knowledge Boundary Cognition Across Languages Through the Lens of Internal Representations
Chenghao Xiao, Hou Pong Chan, Hao Zhang +4
While understanding the knowledge boundaries of LLMs is crucial to prevent hallucination, research on the knowledge boundaries of LLMs has predominantly focused on English. In this…
MIEB: Massive Image Embedding Benchmark
Chenghao Xiao, Isaac Chung, Imene Kerboua +7
Image representations are often evaluated through disjointed, task-specific protocols, leading to a fragmented understanding of model capabilities. For instance, it is unclear whet…