1.4k citations · 3.6k across the 31 of their papers we have counts for
18 papers · 1 filter
Large Language Models as Optimizers
Chengrun Yang, Xuezhi Wang, Yifeng Lu +4
Optimization is ubiquitous. While derivative-based algorithms have been powerful tools for various problems, the absence of gradient imposes challenges on many real-world applicati…
TabNAS: Rejection Sampling for Neural Architecture Search on Tabular Datasets
Chengrun Yang, Gabriel Bender, Hanxiao Liu +5
The best neural architecture for a given machine learning problem depends on many factors: not only the complexity and structure of the dataset, but also on resource constraints in…
Transformer Quality in Linear Time
Weizhe Hua, Zihang Dai, Hanxiao Liu +1
We revisit the design choices in Transformers, and propose methods to address their weaknesses in handling long sequences. First, we propose a simple layer named gated attention un…
Combined Scaling for Zero-shot Transfer Learning
Hieu Pham, Zihang Dai, Golnaz Ghiasi +9
We present a combined scaling method - named BASIC - that achieves 85.7% top-1 accuracy on the ImageNet ILSVRC-2012 validation set without learning from any labeled ImageNet exampl…
Primer: Searching for Efficient Transformers for Language Modeling
David R. So, Wojciech Mańke, Hanxiao Liu +3
Large Transformer models have been central to recent advances in natural language processing. The training and inference costs of these models, however, have grown rapidly and beco…
Pay Attention to MLPs
Hanxiao Liu, Zihang Dai, David R. So +1
Transformers have become one of the most important architectural innovations in deep learning and have enabled many breakthroughs over the past few years. Here we propose a simple…
PyGlove: Symbolic Programming for Automated Machine Learning
Daiyi Peng, Xuanyi Dong, Esteban Real +7
Neural networks are sensitive to hyper-parameter and architecture choices. Automated Machine Learning (AutoML) is a promising paradigm for automating these choices. Current ML soft…
Transferable Graph Optimizers for ML Compilers
Yanqi Zhou, Sudip Roy, Amirali Abdolrashidi +9
Most compilers for machine learning (ML) frameworks need to solve many correlated optimization problems to generate efficient machine code. Current ML compilers rely on heuristics…
Can weight sharing outperform random architecture search? An investigation with TuNAS
Gabriel Bender, Hanxiao Liu, Bo Chen +4
Efficient Neural Architecture Search methods based on weight sharing have shown good promise in democratizing Neural Architecture Search for computer vision models. There is, howev…
Evolving Normalization-Activation Layers
Hanxiao Liu, Andrew Brock, Karen Simonyan +1
Normalization layers and activation functions are fundamental components in deep networks and typically co-locate with each other. Here we propose to design them using an automated…
Neural Predictor for Neural Architecture Search
Wei Wen, Hanxiao Liu, Hai Li +3
Neural Architecture Search methods are effective but often use complex algorithms to come up with the best architecture. We propose an approach with three basic steps that is conce…
GDP: Generalized Device Placement for Dataflow Graphs
Yanqi Zhou, Sudip Roy, Amirali Abdolrashidi +8
Runtime and scalability of large neural networks can be significantly affected by the placement of operations in their dataflow graphs on suitable devices. With increasingly comple…
DARTS: Differentiable Architecture Search
Hanxiao Liu, Karen Simonyan, Yiming Yang
This paper addresses the scalability challenge of architecture search by formulating the task in a differentiable manner. Unlike conventional approaches of applying evolution or re…
Hierarchical Representations for Efficient Architecture Search
Hanxiao Liu, Karen Simonyan, Oriol Vinyals +2
We explore efficient neural architecture search methods and show that a simple yet powerful evolutionary algorithm can discover new architectures with excellent performance. Our ap…
Learning Depthwise Separable Graph Convolution from Data Manifold
Guokun Lai, Hanxiao Liu, Yiming Yang
Convolution Neural Network (CNN) has gained tremendous success in computer vision tasks with its outstanding ability to capture the local latent features. Recently, there has been…
Analogical Inference for Multi-Relational Embeddings
Hanxiao Liu, Yuexin Wu, Yiming Yang
Large-scale multi-relational embedding refers to the task of learning the latent representations for entities and relations in large knowledge graphs. An effective and scalable sol…
Modeling Long- and Short-Term Temporal Patterns with Deep Neural Networks
Guokun Lai, Wei-Cheng Chang, Yiming Yang +1
Multivariate time series forecasting is an important machine learning problem across many domains, including predictions of solar plant energy output, electricity consumption, and…
Cross-Graph Learning of Multi-Relational Associations
Hanxiao Liu, Yiming Yang
Cross-graph Relational Learning (CGRL) refers to the problem of predicting the strengths or labels of multi-relational tuples of heterogeneous object types, through the joint infer…