53 citations · 187 across the 29 of their papers we have counts for
15 papers · 1 filter
GEO: Generative Engine Optimization
Pranjal Aggarwal, Vishvak Murahari, Tanmay Rajpurohit +3
The advent of large language models (LLMs) has ushered in a new paradigm of search engines that use generative models to gather and summarize information to answer user queries. Th…
QualEval: Qualitative Evaluation for Model Improvement
Vishvak Murahari, Ameet Deshpande, Peter Clark +4
Quantitative evaluation metrics have traditionally been pivotal in gauging the advancements of artificial intelligence systems, including large language models (LLMs). However, the…
FireAct: Toward Language Agent Fine-tuning
Baian Chen, Chang Shu, Ehsan Shareghi +3
Recent efforts have augmented language models (LMs) with external tools or environments, leading to the development of language agents that can reason and act. However, most of the…
SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
Carlos E. Jimenez, John Yang, Alexander Wettig +4
Language models have outpaced our ability to evaluate them effectively, but for their future development it is essential to study the frontier of their capabilities. We find real-w…
COLLIE: Systematic Construction of Constrained Text Generation Tasks
Shunyu Yao, Howard Chen, Austin W. Hanjie +2
Text generation under constraints have seen increasing interests in natural language processing, especially with the rapidly improving capabilities of large language models. Howeve…
InstructEval: Systematic Evaluation of Instruction Selection Methods
Anirudh Ajith, Chris Pan, Mengzhou Xia +2
In-context learning (ICL) performs tasks by prompting a large language model (LLM) using an instruction and a small set of annotated examples called demonstrations. Recent work has…
InterCode: Standardizing and Benchmarking Interactive Coding with Execution Feedback
John Yang, Akshara Prabhakar, Karthik Narasimhan +1
Humans write code in a fundamentally interactive manner and rely on constant execution feedback to correct errors, resolve ambiguities, and decompose tasks. While LLMs have recentl…
Referral Augmentation for Zero-Shot Information Retrieval
Michael Tang, Shunyu Yao, John Yang +1
We propose Referral-Augmented Retrieval (RAR), a simple technique that concatenates document indices with referrals, i.e. text from other documents that cite or link to the given d…
Anthropomorphization of AI: Opportunities and Risks
Ameet Deshpande, Tanmay Rajpurohit, Karthik Narasimhan +1
Anthropomorphization is the tendency to attribute human-like traits to non-human entities. It is prevalent in many social contexts -- children anthropomorphize toys, adults do so w…
C-STS: Conditional Semantic Textual Similarity
Ameet Deshpande, Carlos E. Jimenez, Howard Chen +6
Semantic textual similarity (STS), a cornerstone task in NLP, measures the degree of similarity between a pair of sentences, and has broad application in fields such as information…
PruMUX: Augmenting Data Multiplexing with Model Compression
Yushan Su, Vishvak Murahari, Karthik Narasimhan +1
As language models increase in size by the day, methods for efficient inference are critical to leveraging their capabilities for various applications. Prior work has investigated…
Toxicity in ChatGPT: Analyzing Persona-assigned Language Models
Ameet Deshpande, Vishvak Murahari, Tanmay Rajpurohit +2
Large language models (LLMs) have shown incredible capabilities and transcended the natural language processing (NLP) community, with adoption throughout many services like healthc…
MUX-PLMs: Data Multiplexing for High-throughput Language Models
Vishvak Murahari, Ameet Deshpande, Carlos E. Jimenez +4
The widespread adoption of large language models such as ChatGPT and Bard has led to unprecedented demand for these technologies. The burgeoning cost of inference for ever-increasi…
SemSup-XC: Semantic Supervision for Zero and Few-shot Extreme Classification
Pranjal Aggarwal, Ameet Deshpande, Karthik Narasimhan
Extreme classification (XC) involves predicting over large numbers of classes (thousands to millions), with real-world applications like news article classification and e-commerce…
Building Scalable Video Understanding Benchmarks through Sports
Aniket Agarwal, Alex Zhang, Karthik Narasimhan +3
Existing benchmarks for evaluating long video understanding falls short on two critical aspects, either lacking in scale or quality of annotations. These limitations arise from the…