118 citations · 809 across the 58 of their papers we have counts for
16 papers · 1 filter
Learning from learning machines: a new generation of AI technology to meet the needs of science
Luca Pion-Tonachini, Kristofer Bouchard, Hector Garcia Martin +33
We outline emerging opportunities and challenges to enhance the utility of AI for scientific discovery. The distinct goals of AI for industry versus the goals of AI for science cre…
Noisy Feature Mixup
Soon Hoe Lim, N. Benjamin Erichson, Francisco Utrera +2
We introduce Noisy Feature Mixup (NFM), an inexpensive yet effective method for data augmentation that combines the best of interpolation based training and noise injection schemes…
Long Expressive Memory for Sequence Modeling
T. Konstantin Rusch, Siddhartha Mishra, N. Benjamin Erichson +1
We propose a novel method called Long Expressive Memory (LEM) for learning long-term sequential dependencies. LEM is gradient-based, it can efficiently process sequential tasks wit…
What's Hidden in a One-layer Randomly Weighted Transformer?
Sheng Shen, Zhewei Yao, Douwe Kiela +2
We demonstrate that, hidden within one-layer randomly weighted neural networks, there exist subnetworks that can achieve impressive performance, without ever modifying the weight i…
Characterizing possible failure modes in physics-informed neural networks
Aditi S. Krishnapriyan, Amir Gholami, Shandian Zhe +2
Recent work in scientific machine learning has developed so-called physics-informed neural network (PINN) models. The typical approach is to incorporate physical domain knowledge a…
Taxonomizing local versus global structure in neural network loss landscapes
Yaoqing Yang, Liam Hodgkinson, Ryan Theisen +4
Viewing neural network models in terms of their loss landscapes has a long history in the statistical mechanics approach to learning, and in recent years it has received attention…
Stateful ODE-Nets using Basis Function Expansions
Alejandro Queiruga, N. Benjamin Erichson, Liam Hodgkinson +1
The recently-introduced class of ordinary differential equation networks (ODE-Nets) establishes a fruitful connection between deep learning and dynamical systems. In this work, we…
Post-mortem on a deep learning contest: a Simpson's paradox and the complementary roles of scale metrics versus shape metrics
Charles H. Martin, Michael W. Mahoney
To understand better good generalization performance in state-of-the-art neural network (NN) models, and in particular the success of the ALPHAHAT metric based on Heavy-Tailed Self…
LocalNewton: Reducing Communication Bottleneck for Distributed Learning
Vipul Gupta, Avishek Ghosh, Michal Derezinski +3
To address the communication bottleneck problem in distributed optimization within a master-worker framework, we propose LocalNewton, a distributed second-order algorithm with loca…
LEAP: Learnable Pruning for Transformer-based Models
Zhewei Yao, Xiaoxia Wu, Linjian Ma +4
Pruning is an effective method to reduce the memory footprint and computational cost associated with large natural language processing models. However, current pruning algorithms e…
ActNN: Reducing Training Memory Footprint via 2-Bit Activation Compressed Training
Jianfei Chen, Lianmin Zheng, Zhewei Yao +4
The increasing size of neural network models has been critical for improvements in their accuracy, but device memory is not growing at the same rate. This creates fundamental chall…
Integer-only Zero-shot Quantization for Efficient Speech Recognition
Sehoon Kim, Amir Gholami, Zhewei Yao +7
End-to-end neural network models achieve improved performance on various automatic speech recognition (ASR) tasks. However, these models perform poorly on edge hardware due to larg…
A Survey of Quantization Methods for Efficient Neural Network Inference
Amir Gholami, Sehoon Kim, Zhen Dong +3
As soon as abstract mathematical computations were adapted to computation on digital computers, the problem of efficient representation, manipulation, and communication of the nume…
Noisy Recurrent Neural Networks
Soon Hoe Lim, N. Benjamin Erichson, Liam Hodgkinson +1
We provide a general framework for studying recurrent neural networks (RNNs) trained by injecting noise into hidden states. Specifically, we consider RNNs that can be viewed as dis…
Hessian-Aware Pruning and Optimal Neural Implant
Shixing Yu, Zhewei Yao, Amir Gholami +4
Pruning is an effective method to reduce the memory footprint and FLOPs associated with neural network models. However, existing structured-pruning methods often result in signific…
I-BERT: Integer-only BERT Quantization
Sehoon Kim, Amir Gholami, Zhewei Yao +2
Transformer based models, like BERT and RoBERTa, have achieved state-of-the-art results in many Natural Language Processing tasks. However, their memory footprint, inference latenc…