5 papers
DRESS: Disentangled Representation-based Self-Supervised Meta-Learning for Diverse Tasks
Wei Cui, Tongzi Wu, Jesse C. Cresswell +2
Meta-learning represents a strong class of approaches for solving few-shot learning tasks. Nonetheless, recent research suggests that simply pre-training a generic encoder can pote…
CanLegalRAGBench: Evaluating Retrieval-Augmented Generation on Canadian Case Law
Ethan Zhao, Maksym Taranukhin, Wei Cui +2
RAG-based legal assistants have been growing in popularity, but LLM hallucinations remain a key issue and potentially undermines justice. While benchmarks have been developed to ev…
Conf-Gen: Conformal Uncertainty Quantification for Generative Models
Gabriel Loaiza-Ganem, Kevin Zhang, Wei Cui +2
Conformal prediction (CP) and its extension, conformal risk control (CRC), are established frameworks for quantifying uncertainty in supervised machine learning through formal guar…
Flare: Anomaly Diagnostics for Divergent LLM Training in GPU Clusters of Thousand-Plus Scale
Weihao Cui, Ji Zhang, Han Zhao +5
The rapid proliferation of large language models has driven the need for efficient GPU training clusters. However, it is challenging due to the frequent occurrence of training anom…
FlagEval Findings Report: A Preliminary Evaluation of Large Reasoning Models on Automatically Verifiable Textual and Visual Questions
Bowen Qin, Chen Yue, Fang Yin +26
We conduct a moderate-scale contamination-free (to some extent) evaluation of current large reasoning models (LRMs) with some preliminary findings. We also release ROME, our evalua…