6 papers
The Effect of Document Selection on Query-focused Text Analysis
Sandesh S Rangreji, Mian Zhong, Anjalie Field
Analyses of document collections often require selecting what data to analyze, as not all documents are relevant to a particular research question and computational constraints pre…
What Makes a Good Response? An Empirical Analysis of Quality in Qualitative Interviews
Jonathan Ivey, Anjalie Field, Ziang Xiao
Qualitative interviews provide essential insights into human experiences when they elicit high-quality responses. While qualitative and NLP researchers have proposed various measur…
HICode: Hierarchical Inductive Coding with LLMs
Mian Zhong, Pristina Wang, Anjalie Field
Despite numerous applications for fine-grained corpus analysis, researchers continue to rely on manual labeling, which does not scale, or statistical tools like topic modeling, whi…
From Queries to Criteria: Understanding How Astronomers Evaluate LLMs
Alina Hyk, Kiera McCormick, Mian Zhong +7
There is growing interest in leveraging LLMs to aid in astronomy and other scientific research, but benchmarks for LLM evaluation in general have not kept pace with the increasingl…
pathfinder: A Semantic Framework for Literature Review and Knowledge Discovery in Astronomy
Kartheik G. Iyer, Mikaeel Yunus, Charles O'Neill +27
The exponential growth of astronomical literature poses significant challenges for researchers navigating and synthesizing general insights or even domain-specific knowledge. We pr…
Designing an Evaluation Framework for Large Language Models in Astronomy Research
John F. Wu, Alina Hyk, Kiera McCormick +15
Large Language Models (LLMs) are shifting how scientific research is done. It is imperative to understand how researchers interact with these models and how scientific sub-communit…