565 citations · 653 across the 28 of their papers we have counts for
30 papers
Evaluating LLM Creativity through Long-Tail Performance
Yichen Wang, Tenghao Huang, Vishakh Padmakumar +1
Creativity benchmarks for large language models (LLMs) vary widely in task and metric design. This raises the question of whether they capture a common notion of creativity. We pro…
LLMs as Oracles: Reliance on LLMs for Subjective Personal Questions
Myra Cheng, Lujain Ibrahim, Grace Liu +5
We characterize how people are turning to LLMs as oracles: all-knowing authorities on subjective personal questions. Motivated by risks to users' autonomy and well-being, we develo…
Offloading Score: Measuring AI Reliance Through Counterfactual Workflows
Vishakh Padmakumar, Lujain Ibrahim, Zora Zhiruo Wang +3
AI tools are increasingly integrated into real-world workflows. However, existing measures of reliance on these tools focus on AI output adoption or on self-reported indicators, ra…
SWE-chat: Coding Agent Interactions From Real Users in the Wild
Joachim Baumann, Vishakh Padmakumar, Xiang Li +3
AI coding agents are being adopted at scale, yet we lack empirical evidence on how people actually use them and how much of their output is useful in practice. We present SWE-chat,…
No Single Best Model for Diversity: Learning a Router for Sample Diversity
Yuhan Liu, Fangyuan Xu, Vishakh Padmakumar +2
When posed with prompts that permit a large number of valid answers, comprehensively generating them is the first step towards satisfying a wide range of users. In this paper, we s…
SparkMe: Adaptive Semi-Structured Interviewing for Qualitative Insight Discovery
David Anugraha, Vishakh Padmakumar, Diyi Yang
Qualitative insights from user experiences are critical for informing product and policy decisions, but collecting such data at scale is constrained by the time and availability of…
LiteraryTaste: A Preference Dataset for Creative Writing Personalization
John Joon Young Chung, Vishakh Padmakumar, Melissa Roemmele +7
People have different creative writing preferences, and large language models (LLMs) for these tasks can benefit from adapting to each user's preferences. However, these models are…
Intent-Aware Schema Generation And Refinement For Literature Review Tables
Vishakh Padmakumar, Joseph Chee Chang, Kyle Lo +2
The increasing volume of academic literature makes it essential for researchers to organize, compare, and contrast collections of documents. Large language models (LLMs) can suppor…
Principled Content Selection to Generate Diverse and Personalized Multi-Document Summaries
Vishakh Padmakumar, Zichao Wang, David Arbour +1
While large language models (LLMs) are increasingly capable of handling longer contexts, recent work has demonstrated that they exhibit the "lost in the middle" phenomenon (Liu et…
Evaluating the Diversity and Quality of LLM Generated Content
Alexander Shypula, Shuo Li, Botong Zhang +3
Recent work suggests that preference-tuning techniques -- such as Reinforcement Learning from Human Feedback (RLHF) methods like PPO and GRPO, as well as alternatives like DPO -- r…
Measuring LLM Novelty As The Frontier Of Original And High-Quality Output
Vishakh Padmakumar, Chen Yueh-Han, Jane Pan +2
As large language models (LLMs) are increasingly used for ideation and scientific discovery, it is important to evaluate their ability to generate novel output. Prior work evaluate…
Modifying Large Language Model Post-Training for Diverse Creative Writing
John Joon Young Chung, Vishakh Padmakumar, Melissa Roemmele +2
As creative writing tasks do not have singular correct answers, large language models (LLMs) trained to perform these tasks should be able to generate diverse valid outputs. Howeve…
Whose Boat Does it Float? Improving Personalization in Preference Tuning via Inferred User Personas
Nishant Balepur, Vishakh Padmakumar, Fumeng Yang +3
LLMs are aligned to follow input instructions by learning which of two responses users prefer for a prompt. However, such preference data do not convey why users prefer responses t…
Beyond the Binary: Capturing Diverse Preferences With Reward Regularization
Vishakh Padmakumar, Chuanyang Jin, Hannah Rose Kirk +1
Large language models (LLMs) are increasingly deployed via public-facing interfaces to interact with millions of users, each with diverse preferences. Despite this, preference tuni…
Transformers Struggle to Learn to Search
Abulhair Saparov, Srushti Pawar, Shreyas Pimpalgaonkar +6
Search is an ability foundational in many important tasks, and recent studies have shown that large language models (LLMs) struggle to perform search robustly. It is unknown whethe…
Debate Helps Supervise Unreliable Experts
Julian Michael, Salsabila Mahdi, David Rein +4
As AI systems are used to answer more difficult questions and potentially help create new knowledge, judging the truthfulness of their outputs becomes more difficult and more impor…
Creativity Support in the Age of Large Language Models: An Empirical Study Involving Emerging Writers
Tuhin Chakrabarty, Vishakh Padmakumar, Faeze Brahman +1
The development of large language models (LLMs) capable of following instructions and engaging in conversational interactions sparked increased interest in their utilization across…
Does Writing with Language Models Reduce Content Diversity?
Vishakh Padmakumar, He He
Large language models (LLMs) have led to a surge in collaborative writing with model assistance. As different users incorporate suggestions from the same model, there is a risk of…
Testing the General Deductive Reasoning Capacity of Large Language Models Using OOD Examples
Abulhair Saparov, Richard Yuanzhe Pang, Vishakh Padmakumar +4
Given the intractably large size of the space of proofs, any model that is capable of general deductive reasoning must generalize to proofs of greater complexity. Recent studies ha…
Two Failures of Self-Consistency in the Multi-Step Reasoning of LLMs
Angelica Chen, Jason Phang, Alicia Parrish +4
Large language models (LLMs) have achieved widespread success on a variety of in-context few-shot tasks, but this success is typically evaluated via correctness rather than consist…
Extrapolative Controlled Sequence Generation via Iterative Refinement
Vishakh Padmakumar, Richard Yuanzhe Pang, He He +1
We study the problem of extrapolative controlled generation, i.e., generating sequences with attribute values beyond the range seen in training. This task is of significant importa…
Reward Gaming in Conditional Text Generation
Richard Yuanzhe Pang, Vishakh Padmakumar, Thibault Sellam +2
To align conditional text generation model outputs with desired behaviors, there has been an increasing focus on training the model using reinforcement learning (RL) with reward fu…
Help me write a poem: Instruction Tuning as a Vehicle for Collaborative Poetry Writing
Tuhin Chakrabarty, Vishakh Padmakumar, He He
Recent work in training large language models (LLMs) to follow natural language instructions has opened up exciting opportunities for natural language interface design. Building on…
Two-Turn Debate Doesn't Help Humans Answer Hard Reading Comprehension Questions
Alicia Parrish, Harsh Trivedi, Nikita Nangia +4
The use of language-model-based question-answering systems to aid humans in completing difficult tasks is limited, in part, by the unreliability of the text these systems generate.…