5 papers
MermaidSeqBench: An Evaluation Benchmark for NL-to-Mermaid Sequence Diagram Generation
Basel Shbita, Farhan Ahmed, Chad DeLuca
Large language models (LLMs) have shown great promise in generating structured diagrams from natural language descriptions, particularly Mermaid sequence diagrams for software engi…
LLMON: An LLM-native Markup Language to Leverage Structure and Semantics at the LLM Interface
Michael Hind, Basel Shbita, Bo Wu +5
Textual Large Language Models (LLMs) provide a simple and familiar interface: a string of text is used for both input and output. However, the information conveyed to an LLM often…
Adaptive Decoding via Test-Time Policy Learning for Self-Improving Generation
Asmita Bhardwaj, Yuya Jeremy Ong, Eelaaf Zahid +1
Decoding strategies largely determine the quality of Large Language Model (LLM) outputs, yet widely used heuristics such as greedy or fixed temperature/top-p decoding are static an…
Evaluating Ill-Defined Tasks in Large Language Models
Yi Zhou, Basel Shbita
Many evaluations of Large Language Models (LLMs) target tasks that are inherently ill-defined, with unclear input and output spaces and ambiguous success criteria. We analyze why e…
OneShield -- the Next Generation of LLM Guardrails
Chad DeLuca, Anna Lisa Gentile, Shubhi Asthana +7
The rise of Large Language Models has created a general excitement about the great potential for a myriad of applications. While LLMs offer many possibilities, questions about safe…