53 citations · 187 across the 29 of their papers we have counts for
9 papers · 2 filters
FireAct: Toward Language Agent Fine-tuning
Baian Chen, Chang Shu, Ehsan Shareghi +3
Recent efforts have augmented language models (LMs) with external tools or environments, leading to the development of language agents that can reason and act. However, most of the…
SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
Carlos E. Jimenez, John Yang, Alexander Wettig +4
Language models have outpaced our ability to evaluate them effectively, but for their future development it is essential to study the frontier of their capabilities. We find real-w…
COLLIE: Systematic Construction of Constrained Text Generation Tasks
Shunyu Yao, Howard Chen, Austin W. Hanjie +2
Text generation under constraints have seen increasing interests in natural language processing, especially with the rapidly improving capabilities of large language models. Howeve…
InstructEval: Systematic Evaluation of Instruction Selection Methods
Anirudh Ajith, Chris Pan, Mengzhou Xia +2
In-context learning (ICL) performs tasks by prompting a large language model (LLM) using an instruction and a small set of annotated examples called demonstrations. Recent work has…
InterCode: Standardizing and Benchmarking Interactive Coding with Execution Feedback
John Yang, Akshara Prabhakar, Karthik Narasimhan +1
Humans write code in a fundamentally interactive manner and rely on constant execution feedback to correct errors, resolve ambiguities, and decompose tasks. While LLMs have recentl…
Referral Augmentation for Zero-Shot Information Retrieval
Michael Tang, Shunyu Yao, John Yang +1
We propose Referral-Augmented Retrieval (RAR), a simple technique that concatenates document indices with referrals, i.e. text from other documents that cite or link to the given d…
C-STS: Conditional Semantic Textual Similarity
Ameet Deshpande, Carlos E. Jimenez, Howard Chen +6
Semantic textual similarity (STS), a cornerstone task in NLP, measures the degree of similarity between a pair of sentences, and has broad application in fields such as information…
Toxicity in ChatGPT: Analyzing Persona-assigned Language Models
Ameet Deshpande, Vishvak Murahari, Tanmay Rajpurohit +2
Large language models (LLMs) have shown incredible capabilities and transcended the natural language processing (NLP) community, with adoption throughout many services like healthc…
SemSup-XC: Semantic Supervision for Zero and Few-shot Extreme Classification
Pranjal Aggarwal, Ameet Deshpande, Karthik Narasimhan
Extreme classification (XC) involves predicting over large numbers of classes (thousands to millions), with real-world applications like news article classification and e-commerce…