7 papers · 1 filter
What Really Matters for Table LLMs? A Meta-Evaluation of Model and Data Effects
Naihao Deng, Sheng Zhang, Henghui Zhu +7
Table modeling has progressed for decades. In this work, we revisit this trajectory and highlight emerging challenges in the LLM era, particularly the paradox of choice: the diffic…
PRACTIQ: A Practical Conversational Text-to-SQL dataset with Ambiguous and Unanswerable Queries
Mingwen Dong, Nischal Ashok Kumar, Yiqun Hu +9
Previous text-to-SQL datasets and systems have primarily focused on user questions with clear intentions that can be answered. However, real user questions can often be ambiguous w…
Towards a Holistic Evaluation of LLMs on Factual Knowledge Recall
Jiaqing Yuan, Lin Pan, Chung-Wei Hang +5
Large language models (LLMs) have shown remarkable performance on a variety of NLP tasks, and are being rapidly adopted in a wide range of use cases. It is therefore of vital impor…
Benchmarking Diverse-Modal Entity Linking with Generative Models
Sijia Wang, Alexander Hanbo Li, Henry Zhu +9
Entities can be expressed in diverse formats, such as texts, images, or column names and cell values in tables. While existing entity linking (EL) models work well on per modality…
UNITE: A Unified Benchmark for Text-to-SQL Evaluation
Wuwei Lan, Zhiguo Wang, Anuj Chauhan +15
A practical text-to-SQL system should generalize well on a wide variety of natural language questions, unseen database schemas, and novel SQL query structures. To comprehensively e…
Multilingual BERT Post-Pretraining Alignment
Lin Pan, Chung-Wei Hang, Haode Qi +3
We propose a simple method to align multilingual contextual embeddings as a post-pretraining step for improved zero-shot cross-lingual transferability of the pretrained models. Usi…