8 papers
RegExtract: A Benchmark for PFAS Regulatory Information Extraction with LLMs
Defne Circi, Krystelle Lionti, Sarathkrishna Swaminathan +8
Extracting structured information from regulatory documents is a critical but challenging task. Relevant facts are scattered across long, legally dense texts, and traceability of e…
LLMON: An LLM-native Markup Language to Leverage Structure and Semantics at the LLM Interface
Michael Hind, Basel Shbita, Bo Wu +5
Textual Large Language Models (LLMs) provide a simple and familiar interface: a string of text is used for both input and output. However, the information conveyed to an LLM often…
Adaptive Decoding via Test-Time Policy Learning for Self-Improving Generation
Asmita Bhardwaj, Yuya Jeremy Ong, Eelaaf Zahid +1
Decoding strategies largely determine the quality of Large Language Model (LLM) outputs, yet widely used heuristics such as greedy or fixed temperature/top-p decoding are static an…
Evaluating Ill-Defined Tasks in Large Language Models
Yi Zhou, Basel Shbita
Many evaluations of Large Language Models (LLMs) target tasks that are inherently ill-defined, with unclear input and output spaces and ambiguous success criteria. We analyze why e…
MermaidSeqBench: An Evaluation Benchmark for NL-to-Mermaid Sequence Diagram Generation
Basel Shbita, Farhan Ahmed, Chad DeLuca
Large language models (LLMs) have shown great promise in generating structured diagrams from natural language descriptions, particularly Mermaid sequence diagrams for software engi…
OneShield -- the Next Generation of LLM Guardrails
Chad DeLuca, Anna Lisa Gentile, Shubhi Asthana +7
The rise of Large Language Models has created a general excitement about the great potential for a myriad of applications. While LLMs offer many possibilities, questions about safe…
An Automatic Approach for Generating Rich, Linked Geo-Metadata from Historical Map Images
Zekun Li, Yao-Yi Chiang, Sasan Tavakkol +4
Historical maps contain detailed geographic information difficult to find elsewhere covering long-periods of time (e.g., 125 years for the historical topographic maps in the US). H…
Viola: A Topic Agnostic Generate-and-Rank Dialogue System
Hyundong Cho, Basel Shbita, Kartik Shenoy +5
We present Viola, an open-domain dialogue system for spoken conversation that uses a topic-agnostic dialogue manager based on a simple generate-and-rank approach. Leveraging recent…