Publications (76)
Natural Language to Code Generation in Interactive Data Science Notebooks
Pengcheng Yin, Wen-Ding Li, Kefan Xiao +9
Computational notebooks, such as Jupyter notebooks, are interactive computing environments that are ubiquitous among data scientists to perform data wrangling and analytic tasks. T…
Learning Discrete Energy-based Models via Auxiliary-variable Local Exploration
Hanjun Dai, Rishabh Singh, Bo Dai +2
Discrete structures play an important role in applications like program language modeling and software engineering. Current approaches to predicting complex structures typically co…
Incremental Sampling Without Replacement for Sequence Models
Kensen Shi, David Bieber, Charles Sutton
Sampling is a fundamental technique, and sampling without replacement is often desirable when duplicate samples are not beneficial. Within machine learning, sampling is useful for…
Learning to Execute Programs with Instruction Pointer Attention Graph Neural Networks
David Bieber, Charles Sutton, Hugo Larochelle +1
Graph neural networks (GNNs) have emerged as a powerful tool for learning software engineering tasks including code completion, bug finding, and program repair. They benefit from l…
A Probabilistic Framework for Modular Continual Learning
Lazar Valkov, Akash Srivastava, Swarat Chaudhuri +1
Modular approaches that use a different composition of modules for each problem are a promising direction in continual learning (CL). However, searching through the large, discrete…
Bayesian inference for queueing networks and modeling of internet services
Charles Sutton, Michael I. Jordan
Modern Internet services, such as those at Google, Yahoo!, and Amazon, handle billions of requests per day on clusters of thousands of computers. Because these services operate und…
Mining Idioms from Source Code
Miltiadis Allamanis, Charles Sutton
We present the first method for automatically mining code idioms from a corpus of previously written, idiomatic software projects. We take the view that a code idiom is a syntactic…
CrossBeam: Learning to Search in Bottom-Up Program Synthesis
Kensen Shi, Hanjun Dai, Kevin Ellis +1
Many approaches to program synthesis perform a search within an enormous space of programs to find one that satisfies a given specification. Prior works have used neural models to…
Piecewise Training for Undirected Models
Charles Sutton, Andrew McCallum
For many large undirected models that arise in real-world applications, exact maximumlikelihood training is intractable, because it requires computing marginal distributions of the…
ExeDec: Execution Decomposition for Compositional Generalization in Neural Program Synthesis
Kensen Shi, Joey Hong, Yinlin Deng +3
When writing programs, people have the ability to tackle a new complex task by decomposing it into smaller and more familiar subtasks. While it is difficult to measure whether neur…
How Often Do Single-Statement Bugs Occur? The ManySStuBs4J Dataset
Rafael-Michael Karampatsis, Charles Sutton
Program repair is an important but difficult software engineering problem. One way to achieve acceptable performance is to focus on classes of simple bugs, such as bugs with single…
Tailored Mutants Fit Bugs Better
Miltiadis Allamanis, Earl T. Barr, René Just +1
Mutation analysis measures test suite adequacy, the degree to which a test suite detects seeded faults: one test suite is better than another if it detects more mutants. Mutation a…
Interpreting Deep Classifier by Visual Distillation of Dark Knowledge
Kai Xu, Dae Hoon Park, Chang Yi +1
Interpreting black box classifiers, such as deep networks, allows an analyst to validate a classifier before it is deployed in a high-stakes setting. A natural idea is to visualize…
Generative Ratio Matching Networks
Akash Srivastava, Kai Xu, Michael U. Gutmann +1
Deep generative models can learn to generate realistic-looking images, but many of the most effective methods are adversarial and involve a saddlepoint optimization, which requires…
Modified Family-Vicsek Scaling and Probability Distributions for Brownian Castle Interfaces
Noah Sublett, Charles Sutton, Benjamin Long +1
The paper introduces the Brownian Castle interface growth model and uses numerical simulations to show that its interface width follows a modified Family‑Vicsek scaling with specif…
Learning to Represent Programs with Property Signatures
Augustus Odena, Charles Sutton
We introduce the notion of property signatures, a representation for programs and program specifications meant for consumption by machine learning algorithms. Given a function with…
Learning Semantic Annotations for Tabular Data
Jiaoyan Chen, Ernesto Jimenez-Ruiz, Ian Horrocks +1
The usefulness of tabular data such as web tables critically depends on understanding their semantics. This study focuses on column type prediction for tables without any meta data…
A Survey of Machine Learning for Big Code and Naturalness
Miltiadis Allamanis, Earl T. Barr, Premkumar Devanbu +1
Research at the intersection of machine learning, programming languages, and software engineering has recently taken important steps in proposing learnable probabilistic models of…
Interleaved Factorial Non-Homogeneous Hidden Markov Models for Energy Disaggregation
Mingjun Zhong, Nigel Goddard, Charles Sutton
To reduce energy demand in households it is useful to know which electrical appliances are in use at what times. Monitoring individual appliances is costly and intrusive, whereas d…
UQE: A Query Engine for Unstructured Databases
Hanjun Dai, Bethany Yixin Wang, Xingchen Wan +7
Analytics on structured data is a mature field with many successful methods. However, most real world data exists in unstructured form, such as images and conversations. We investi…
Towards Modular Algorithm Induction
Daniel A. Abolafia, Rishabh Singh, Manzil Zaheer +1
We present a modular neural network architecture Main that learns algorithms given a set of input-output examples. Main consists of a neural controller that interacts with a variab…
Scheduled denoising autoencoders
Krzysztof J. Geras, Charles Sutton
We present a representation learning method that learns features at multiple different levels of scale. Working within the unsupervised framework of denoising autoencoders, we obse…
Autoencoding Variational Inference For Topic Models
Akash Srivastava, Charles Sutton
Topic models are one of the most popular methods for learning representations of text, but a major challenge is that any change to the topic model requires mathematically deriving…
Learning to Fix Build Errors with Graph2Diff Neural Networks
Daniel Tarlow, Subhodeep Moitra, Andrew Rice +4
Professional software developers spend a significant amount of time fixing builds, but this has received little attention as a problem in automatic program repair. We present a new…
A Subsequence Interleaving Model for Sequential Pattern Mining
Jaroslav Fowkes, Charles Sutton
Recent sequential pattern mining methods have used the minimum description length (MDL) principle to define an encoding scheme which describes an algorithm for mining the most comp…
Conditional independence by typing
Maria I. Gorinova, Andrew D. Gordon, Charles Sutton +1
A central goal of probabilistic programming languages (PPLs) is to separate modelling from inference. However, this goal is hard to achieve in practice. Users are often forced to r…
Learning Continuous Semantic Representations of Symbolic Expressions
Miltiadis Allamanis, Pankajan Chanthirasegaran, Pushmeet Kohli +1
Combining abstract, symbolic reasoning with continuous neural reasoning is a grand challenge of representation learning. As a step in this direction, we propose a new architecture,…
Improved Dynamic Schedules for Belief Propagation
Charles Sutton, Andrew McCallum
Belief propagation and its variants are popular methods for approximate inference, but their running time and even their convergence depend greatly on the schedule used to send the…
Variational Inference In Pachinko Allocation Machines
Akash Srivastava, Charles Sutton
The Pachinko Allocation Machine (PAM) is a deep topic model that allows representing rich correlation structures among topics by a directed acyclic graph over topics. Because of th…
Popularity of arXiv.org within Computer Science
Charles Sutton, Linan Gong
It may seem surprising that, out of all areas of science, computer scientists have been slow to post electronic versions of papers on sites like arXiv.org. Instead, computer scient…
A Bayesian Network Model for Interesting Itemsets
Jaroslav Fowkes, Charles Sutton
Mining itemsets that are the most interesting under a statistical model of the underlying data is a commonly used and well-studied technique for exploratory data analysis, with the…
Universal Self-Consistency for Large Language Model Generation
Xinyun Chen, Renat Aksitov, Uri Alon +7
Self-consistency with chain-of-thought prompting (CoT) has demonstrated remarkable performance gains on various challenging tasks, by utilizing multiple reasoning paths sampled fro…
Blending LSTMs into CNNs
Krzysztof J. Geras, Abdel-rahman Mohamed, Rich Caruana +6
We consider whether deep convolutional networks (CNNs) can represent decision functions with similar accuracy as recurrent networks such as LSTMs. First, we show that a deep CNN wi…
Sequence-to-point learning with neural networks for nonintrusive load monitoring
Chaoyun Zhang, Mingjun Zhong, Zongzuo Wang +2
Energy disaggregation (a.k.a nonintrusive load monitoring, NILM), a single-channel blind source separation problem, aims to decompose the mains which records the whole house electr…
Word Storms: Multiples of Word Clouds for Visual Comparison of Documents
Quim Castella, Charles Sutton
Word clouds are a popular tool for visualizing documents, but they are not a good tool for comparing documents, because identical words are not presented consistently across differ…
Program Synthesis with Large Language Models
Jacob Austin, Augustus Odena, Maxwell Nye +8
This paper explores the limits of the current generation of large language models for program synthesis in general purpose programming languages. We evaluate a collection of such m…
Probabilistic Programming with Densities in SlicStan: Efficient, Flexible and Deterministic
Maria I. Gorinova, Andrew D. Gordon, Charles Sutton
Stan is a probabilistic programming language that has been increasingly used for real-world scalable projects. However, to make practical inference possible, the language sacrifice…
A Convolutional Attention Network for Extreme Summarization of Source Code
Miltiadis Allamanis, Hao Peng, Charles Sutton
Attention mechanisms in neural networks have proved useful for problems in which the input and output do not have fixed dimension. Often there exist features that are locally trans…
SpreadsheetCoder: Formula Prediction from Semi-structured Context
Xinyun Chen, Petros Maniatis, Rishabh Singh +4
Spreadsheet formula prediction has been an important program synthesis problem with many real-world applications. Previous works typically utilize input-output examples as the spec…
Training Chain-of-Thought via Latent-Variable Inference
Du Phan, Matthew D. Hoffman, David Dohan +7
Large language models (LLMs) solve problems more accurately and interpretably when instructed to work out the answer step by step using a ``chain-of-thought'' (CoT) prompt. One can…
Parameter-Free Probabilistic API Mining across GitHub
Jaroslav Fowkes, Charles Sutton
Existing API mining algorithms can be difficult to use as they require expensive parameter tuning and the returned set of API calls can be large, highly redundant and difficult to…
Compositional Generalization and Decomposition in Neural Program Synthesis
Kensen Shi, Joey Hong, Manzil Zaheer +2
When writing programs, people have the ability to tackle a new complex task by decomposing it into smaller and more familiar subtasks. While it is difficult to measure whether neur…
GEMSEC: Graph Embedding with Self Clustering
Benedek Rozemberczki, Ryan Davies, Rik Sarkar +1
Modern graph embedding procedures can efficiently process graphs with millions of nodes. In this paper, we propose GEMSEC -- a graph embedding algorithm which learns a clustering o…
Gemini: A Family of Highly Capable Multimodal Models
Gemini Team, Rohan Anil, Sebastian Borgeaud +1340
This report introduces a new family of multimodal models, Gemini, that exhibit remarkable capabilities across image, audio, video, and text understanding. The Gemini family consist…
Latent Bayesian melding for integrating individual and population models
Mingjun Zhong, Nigel Goddard, Charles Sutton
In many statistical problems, a more coarse-grained model may be suitable for population-level behaviour, whereas a more detailed model is appropriate for accurate modelling of ind…
Neural Program Synthesis with a Differentiable Fixer
Matej Balog, Rishabh Singh, Petros Maniatis +1
We present a new program synthesis approach that combines an encoder-decoder based synthesis architecture with a differentiable program fixer. Our approach is inspired from the fac…
Semi-Separable Hamiltonian Monte Carlo for Inference in Bayesian Hierarchical Models
Yichuan Zhang, Charles Sutton
Sampling from hierarchical Bayesian models is often difficult for MCMC methods, because of the strong correlations between the model parameters and the hyperparameters. Recent Riem…
Capturing Data Uncertainty in High-Volume Stream Processing
Yanlei Diao, Boduo Li, Anna Liu +4
We present the design and development of a data stream system that captures data uncertainty from data collection to query processing to final result generation. Our system focuses…
Robust Variational Autoencoders for Outlier Detection and Repair of Mixed-Type Data
Simão Eduardo, Alfredo Nazábal, Christopher K. I. Williams +1
We focus on the problem of unsupervised cell outlier detection and repair in mixed-type tabular data. Traditional methods are concerned only with detecting which rows in the datase…
Deep Learning to Detect Redundant Method Comments
Annie Louis, Santanu Kumar Dash, Earl T. Barr +1
Comments in software are critical for maintenance and reuse. But apart from prescriptive advice, there is little practical support or quantitative understanding of what makes a com…
Autofolding for Source Code Summarization
Jaroslav Fowkes, Pankajan Chanthirasegaran, Razvan Ranca +3
Developers spend much of their time reading and browsing source code, raising new opportunities for summarization methods. Indeed, modern code editors provide code folding, which a…
HOUDINI: Lifelong Learning as Program Synthesis
Lazar Valkov, Dipak Chaudhari, Akash Srivastava +2
We present a neurosymbolic framework for the lifelong learning of algorithmic tasks that mix perception and procedural reasoning. Reusing high-level concepts across domains and lea…
Clustering with a Reject Option: Interactive Clustering as Bayesian Prior Elicitation
Akash Srivastava, James Zou, Ryan P. Adams +1
A good clustering can help a data analyst to explore and understand a data set, but what constitutes a good clustering may depend on domain-specific and application-specific criter…
Distributed Inference and Query Processing for RFID Tracking and Monitoring
Zhao Cao, Charles Sutton, Yanlei Diao +1
In this paper, we present the design of a scalable, distributed stream processing system for RFID tracking and monitoring. Since RFID data lacks containment and location informatio…
Maybe Deep Neural Networks are the Best Choice for Modeling Source Code
Rafael-Michael Karampatsis, Charles Sutton
Statistical language modeling techniques have successfully been applied to source code, yielding a variety of new software development tools, such as tools for code suggestion and…
NExT: Teaching Large Language Models to Reason about Code Execution
Ansong Ni, Miltiadis Allamanis, Arman Cohan +4
A fundamental skill among human developers is the ability to understand and reason about program execution. As an example, a programmer can mentally simulate code execution in natu…
A Library for Representing Python Programs as Graphs for Machine Learning
David Bieber, Kensen Shi, Petros Maniatis +4
Graph representations of programs are commonly a central element of machine learning for code research. We introduce an open source Python library python_graphs that applies static…
Wrangling Messy CSV Files by Detecting Row and Type Patterns
Gerrit J. J. van den Burg, Alfredo Nazabal, Charles Sutton
It is well known that data scientists spend the majority of their time on preparing data for analysis. One of the first steps in this preparation phase is to load the data from the…
VEEGAN: Reducing Mode Collapse in GANs using Implicit Variational Learning
Akash Srivastava, Lazar Valkov, Chris Russell +2
Deep generative models provide powerful tools for distributions over complicated manifolds, such as those of natural images. But many of these methods, including generative adversa…
LambdaBeam: Neural Program Search with Higher-Order Functions and Lambdas
Kensen Shi, Hanjun Dai, Wen-Ding Li +2
Search is an important technique in program synthesis that allows for adaptive strategies such as focusing on particular search directions based on execution results. Several prior…
Show Your Work: Scratchpads for Intermediate Computation with Language Models
Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari +9
Large pre-trained language models perform remarkably well on tasks that can be done "in one pass", such as generating realistic text or synthesizing computer programs. However, the…
Repairing Systematic Outliers by Learning Clean Subspaces in VAEs
Simao Eduardo, Kai Xu, Alfredo Nazabal +1
Data cleaning often comprises outlier detection and data repair. Systematic errors result from nearly deterministic transformations that occur repeatedly in the data, e.g. specific…
PaLM: Scaling Language Modeling with Pathways
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin +64
Large language models have been shown to achieve remarkable performance across a variety of natural language tasks using few-shot learning, which drastically reduces the number of…
Learning Natural Coding Conventions
Miltiadis Allamanis, Earl T. Barr, Christian Bird +1
Every programmer has a characteristic style, ranging from preferences about identifier naming to preferences about object relationships and design patterns. Coding conventions defi…
Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities
Gheorghe Comanici, Eric Bieber, Mike Schaekermann +3431
In this report, we introduce the Gemini 2.X model family: Gemini 2.5 Pro and Gemini 2.5 Flash, as well as our earlier Gemini 2.0 Flash and Flash-Lite models. Gemini 2.5 Pro is our…
Latent Programmer: Discrete Latent Codes for Program Synthesis
Joey Hong, David Dohan, Rishabh Singh +2
In many sequence learning tasks, such as program synthesis and document summarization, a key problem is searching over a large space of possible output sequences. We propose to lea…
Big Code != Big Vocabulary: Open-Vocabulary Models for Source Code
Rafael-Michael Karampatsis, Hlib Babii, Romain Robbes +2
Statistical language modeling techniques have successfully been applied to large source code corpora, yielding a variety of new software development tools, such as tools for code s…
Language Model Cascades
David Dohan, Winnie Xu, Aitor Lewkowycz +9
Prompted models have demonstrated impressive few-shot learning abilities. Repeated interactions at test-time with a single model, or the composition of multiple models together, fu…
Clustering with a Reject Option: Interactive Clustering as Bayesian Prior Elicitation
Akash Srivastava, James Zou, Charles Sutton
A good clustering can help a data analyst to explore and understand a data set, but what constitutes a good clustering may depend on domain-specific and application-specific criter…
BUSTLE: Bottom-Up Program Synthesis Through Learning-Guided Exploration
Augustus Odena, Kensen Shi, David Bieber +3
Program synthesis is challenging largely because of the difficulty of search in a large space of programs. Human programmers routinely tackle the task of writing complex programs b…
SCELMo: Source Code Embeddings from Language Models
Rafael - Michael Karampatsis, Charles Sutton
Continuous embeddings of tokens in computer programs have been used to support a variety of software development tools, including readability, code search, and program repair. Cont…
ColNet: Embedding the Semantics of Web Tables for Column Type Prediction
Jiaoyan Chen, Ernesto Jimenez-Ruiz, Ian Horrocks +1
Automatically annotating column types with knowledge base (KB) concepts is a critical task to gain a basic understanding of web tables. Current methods rely on either table metadat…
An Introduction to Conditional Random Fields
Charles Sutton, Andrew McCallum
Often we wish to predict a large number of variables that depend on each other as well as on other observed variables. Structured prediction methods are essentially a combination o…
Natural Language Outlines for Code: Literate Programming in the LLM Era
Kensen Shi, Deniz Altınbüken, Saswat Anand +17
We propose using natural language outlines as a novel modality and interaction surface for providing AI assistance to developers throughout the software development process. An NL…
Couplings for Multinomial Hamiltonian Monte Carlo
Kai Xu, Tor Erlend Fjelde, Charles Sutton +1
Hamiltonian Monte Carlo (HMC) is a popular sampling method in Bayesian inference. Recently, Heng & Jacob (2019) studied Metropolis HMC with couplings for unbiased Monte Carlo estim…
OptTyper: Probabilistic Type Inference by Optimising Logical and Natural Constraints
Irene Vlassi Pandi, Earl T. Barr, Andrew D. Gordon +1
We present a new approach to the type inference problem for dynamic languages. Our goal is to combine \emph{logical} constraints, that is, deterministic information from a type sys…