221 citations · 1.1k across the 24 of their papers we have counts for
28 papers
In-Context Learning with Long-Context Models: An In-Depth Exploration
Amanda Bertsch, Maor Ivgi, Emily Xiao +4
As model context lengths continue to increase, the number of demonstrations that can be provided in-context approaches the size of entire training datasets. We study the behavior o…
In-Context Principle Learning from Mistakes
Tianjun Zhang, Aman Madaan, Luyu Gao +5
In-context learning (ICL, also known as few-shot prompting) has been the standard method of adapting LLMs to downstream tasks, by learning from a few input-output examples. Nonethe…
WebArena: A Realistic Web Environment for Building Autonomous Agents
Shuyan Zhou, Frank F. Xu, Hao Zhu +9
With advances in generative AI, there is now potential for autonomous agents to manage daily tasks via natural language commands. However, current agents are primarily created and…
GPT-Calls: Enhancing Call Segmentation and Tagging by Generating Synthetic Conversations via Large Language Models
Itzik Malkiel, Uri Alon, Yakir Yehuda +4
Transcriptions of phone calls are of significant value across diverse fields, such as sales, customer service, healthcare, and law enforcement. Nevertheless, the analysis of these…
On the Expressivity Role of LayerNorm in Transformers' Attention
Shaked Brody, Uri Alon, Eran Yahav
Layer Normalization (LayerNorm) is an inherent component in all Transformer-based models. In this paper, we show that LayerNorm is crucial to the expressivity of the multi-head att…
Unlimiformer: Long-Range Transformers with Unlimited Length Input
Amanda Bertsch, Uri Alon, Graham Neubig +1
Since the proposal of transformers, these models have been limited to bounded input lengths, because of their need to attend to every token in the input. In this work, we propose U…
Self-Refine: Iterative Refinement with Self-Feedback
Aman Madaan, Niket Tandon, Prakhar Gupta +13
Like humans, large language models (LLMs) do not always generate the best output on their first try. Motivated by how humans refine their written text, we introduce Self-Refine, an…
Learning Performance-Improving Code Edits
Alexander Shypula, Aman Madaan, Yimeng Zeng +7
With the decline of Moore's law, optimizing program performance has become a major focus of software research. However, high-level optimizations such as API and algorithm changes r…
CodeBERTScore: Evaluating Code Generation with Pretrained Models of Code
Shuyan Zhou, Uri Alon, Sumit Agarwal +1
Since the rise of neural natural-language-to-code models (NL->Code) that can generate long expressions and statements rather than a single next-token, one of the major problems has…
Why do Nearest Neighbor Language Models Work?
Frank F. Xu, Uri Alon, Graham Neubig
Language models (LMs) compute the probability of a text by sequentially computing a representation of an already-seen context and using this representation to predict the next word…
PAL: Program-aided Language Models
Luyu Gao, Aman Madaan, Shuyan Zhou +5
Large language models (LLMs) have recently demonstrated an impressive ability to perform arithmetic and symbolic reasoning tasks, when provided with a few examples at test time ("f…
Language Models of Code are Few-Shot Commonsense Learners
Aman Madaan, Shuyan Zhou, Uri Alon +2
We address the general task of structured commonsense reasoning: given a natural language input, the goal is to generate a graph such as an event -- or a reasoning-graph. To employ…
Oversquashing in GNNs through the lens of information contraction and graph expansion
Pradeep Kr. Banerjee, Kedar Karhadkar, Yu Guang Wang +2
The quality of signal propagation in message-passing graph neural networks (GNNs) strongly influences their expressivity as has been observed in recent works. In particular, for pr…
DocPrompting: Generating Code by Retrieving the Docs
Shuyan Zhou, Uri Alon, Frank F. Xu +3
Publicly available source-code libraries are continuously growing and changing. This makes it impossible for models of code to keep current with all available APIs by simply traini…
A Systematic Evaluation of Large Language Models of Code
Frank F. Xu, Uri Alon, Graham Neubig +1
Large language models (LMs) of code have recently shown tremendous promise in completing code and synthesizing code from natural language descriptions. However, the current state-o…
Neuro-Symbolic Language Modeling with Automaton-augmented Retrieval
Uri Alon, Frank F. Xu, Junxian He +3
Retrieval-based language models (R-LM) model the probability of natural language text by combining a standard language model (LM) with examples retrieved from an external datastore…
How Attentive are Graph Attention Networks?
Shaked Brody, Uri Alon, Eran Yahav
Graph Attention Networks (GATs) are one of the most popular GNN architectures and are considered as the state-of-the-art architecture for representation learning with graphs. In GA…
Single-Node Attacks for Fooling Graph Neural Networks
Ben Finkelshtein, Chaim Baskin, Evgenii Zheltonozhskii +1
Graph neural networks (GNNs) have shown broad applicability in a variety of domains. These domains, e.g., social networks and product recommendations, are fertile ground for malici…
On the Bottleneck of Graph Neural Networks and its Practical Implications
Uri Alon, Eran Yahav
Since the proposal of the graph neural network (GNN) by Gori et al. (2005) and Scarselli et al. (2008), one of the major problems in training GNNs was their struggle to propagate i…
A Structural Model for Contextual Code Changes
Shaked Brody, Uri Alon, Eran Yahav
We address the problem of predicting edit completions based on a learned model that was trained on past edits. Given a code snippet that is partially edited, our goal is to predict…
Adversarial Examples for Models of Code
Noam Yefet, Uri Alon, Eran Yahav
Neural models of code have shown impressive results when performing tasks such as predicting method names and identifying certain kinds of bugs. We show that these models are vulne…
Structural Language Models of Code
Uri Alon, Roy Sadaka, Omer Levy +1
We address the problem of any-code completion - generating a missing piece of source code in a given program without any restriction on the vocabulary or structure. We introduce a…
Lingvo: a Modular and Scalable Framework for Sequence-to-Sequence Modeling
Jonathan Shen, Patrick Nguyen, Yonghui Wu +88
Lingvo is a Tensorflow framework offering a complete solution for collaborative deep learning research, with a particular focus towards sequence-to-sequence models. Lingvo models a…
Neural Reverse Engineering of Stripped Binaries using Augmented Control Flow Graphs
Yaniv David, Uri Alon, Eran Yahav
We address the problem of reverse engineering of stripped executables, which contain no debug information. This is a challenging problem because of the low amount of syntactic info…