36 citations · 69 across the 19 of their papers we have counts for
21 papers
Thoughtbubbles: an Unsupervised Method for Parallel Thinking in Latent Space
Houjun Liu, Shikhar Murty, Christopher D. Manning +1
Current approaches for scaling inference-time compute in transformers train them to emit explicit chain-of-thought tokens before producing an answer. While these methods are powerf…
WebDS: An End-to-End Benchmark for Web-based Data Science
Ethan Hsu, Hong Meng Yam, Ines Bouissou +9
Many real-world data science tasks involve complex web-based interactions: finding appropriate data available on the internet, synthesizing multimodal data from different locations…
Mechanisms vs. Outcomes: Probing for Syntax Fails to Explain Performance on Targeted Syntactic Evaluations
Ananth Agarwal, Jasper Jian, Christopher D. Manning +1
Large Language Models (LLMs) exhibit a robust mastery of syntax when processing and generating text. While this suggests internalized understanding of hierarchical syntax and depen…
Sneaking Syntax into Transformer Language Models with Tree Regularization
Ananjan Nandi, Christopher D. Manning, Shikhar Murty
While compositional accounts of human language understanding are based on a hierarchical tree-like process, neural models like transformers lack a direct inductive bias for such tr…
MrT5: Dynamic Token Merging for Efficient Byte-level Language Models
Julie Kallini, Shikhar Murty, Christopher D. Manning +2
Models that rely on subword tokenization have significant drawbacks, such as sensitivity to character-level noise like spelling errors and inconsistent compression rates across dif…
NNetNav: Unsupervised Learning of Browser Agents Through Environment Interaction in the Wild
Shikhar Murty, Hao Zhu, Dzmitry Bahdanau +1
We introduce NNetNav, a method for unsupervised interaction with websites that generates synthetic demonstrations for training browser agents. Given any website, NNetNav produces t…
BAGEL: Bootstrapping Agents by Guiding Exploration with Language
Shikhar Murty, Christopher Manning, Peter Shaw +2
Following natural language instructions by executing actions in digital environments (e.g. web-browsers and REST APIs) is a challenging task for language model (LM) agents. Unfortu…
Pushdown Layers: Encoding Recursive Structure in Transformer Language Models
Shikhar Murty, Pratyusha Sharma, Jacob Andreas +1
Recursion is a prominent feature of human language, and fundamentally challenging for self-attention due to the lack of an explicit recursive-state tracking mechanism. Consequently…
Pseudointelligence: A Unifying Framework for Language Model Evaluation
Shikhar Murty, Orr Paradise, Pratyusha Sharma
With large language models surpassing human performance on an increasing number of benchmarks, we must take a principled approach for targeted evaluation of model capabilities. Ins…
Grokking of Hierarchical Structure in Vanilla Transformers
Shikhar Murty, Pratyusha Sharma, Jacob Andreas +1
For humans, language production and comprehension is sensitive to the hierarchical structure of sentences. In natural language processing, past work has questioned how effectively…
Fixing Model Bugs with Natural Language Patches
Shikhar Murty, Christopher D. Manning, Scott Lundberg +1
Current approaches for fixing systematic problems in NLP models (e.g. regex patches, finetuning on more data) are either brittle, or labor-intensive and liable to shortcuts. In con…
On Measuring the Intrinsic Few-Shot Hardness of Datasets
Xinran Zhao, Shikhar Murty, Christopher D. Manning
While advances in pre-training have led to dramatic improvements in few-shot learning of NLP tasks, there is limited understanding of what drives successful few-shot adaptation in…
Characterizing Intrinsic Compositionality in Transformers with Tree Projections
Shikhar Murty, Pratyusha Sharma, Jacob Andreas +1
When trained on language data, do transformers learn some arbitrary computation that utilizes the full capacity of the architecture or do they learn a simpler, tree-like computatio…
ExpBERT: Representation Engineering with Natural Language Explanations
Shikhar Murty, Pang Wei Koh, Percy Liang
Suppose we want to specify the inductive bias that married couples typically go on honeymoons for the task of extracting pairs of spouses from text. In this paper, we allow model d…
CLOSURE: Assessing Systematic Generalization of CLEVR Models
Dzmitry Bahdanau, Harm de Vries, Timothy J. O'Donnell +4
The CLEVR dataset of natural-looking questions about 3D-rendered scenes has recently received much attention from the research community. A number of models have been proposed for…
Embedded-State Latent Conditional Random Fields for Sequence Labeling
Dung Thai, Sree Harsha Ramesh, Shikhar Murty +2
Complex textual information extraction tasks are often posed as sequence labeling or \emph{shallow parsing}, where fields are extracted using local labels made consistent through p…
Hierarchical Losses and New Resources for Fine-grained Entity Typing and Linking
Shikhar Murty*, Patrick Verga*, Luke Vilnis +2
Extraction from raw text to a knowledge base of entities and fine-grained types is often cast as prediction into a flat set of entity and type labels, neglecting the rich hierarchi…
Probabilistic Embedding of Knowledge Graphs with Box Lattice Measures
Luke Vilnis, Xiang Li, Shikhar Murty +1
Embedding methods which enforce a partial order or lattice structure over the concept space, such as Order Embeddings (OE) (Vendrov et al., 2016), are a natural way to model transi…
Finer Grained Entity Typing with TypeNet
Shikhar Murty, Patrick Verga, Luke Vilnis +1
We consider the challenging problem of entity typing over an extremely fine grained set of types, wherein a single mention or entity can have many simultaneous and often hierarchic…
Low-Rank Hidden State Embeddings for Viterbi Sequence Labeling
Dung Thai, Shikhar Murty, Trapit Bansal +3
In textual information extraction and other sequence labeling tasks it is now common to use recurrent neural networks (such as LSTM) to form rich embedded representations of long-t…
Joint Matrix-Tensor Factorization for Knowledge Base Inference
Prachi Jain, Shikhar Murty, Mausam +1
While several matrix factorization (MF) and tensor factorization (TF) models have been proposed for knowledge base (KB) inference, they have rarely been compared across various dat…