118 citations · 809 across the 58 of their papers we have counts for
60 papers
GEANN: Scalable Graph Augmentations for Multi-Horizon Time Series Forecasting
Sitan Yang, Malcolm Wolff, Shankar Ramasubramanian +3
Encoder-decoder deep neural networks have been increasingly studied for multi-horizon time series forecasting, especially in real-world applications. However, to forecast accuratel…
CLOVER : Probabilistic Forecasting with Coherent Learning Objective Reparameterization
Kin G. Olivares, Geoffrey Négiar, Ruijun Ma +3
Obtaining accurate probabilistic forecasts is an operational challenge in many applications, such as energy management, climate forecasting, supply chain planning, and resource all…
SuperBench: A Super-Resolution Benchmark Dataset for Scientific Machine Learning
Pu Ren, N. Benjamin Erichson, Junyi Guo +4
Super-resolution (SR) techniques aim to enhance data resolution, enabling the retrieval of finer details, and improving the overall quality and fidelity of the data representation.…
SqueezeLLM: Dense-and-Sparse Quantization
Sehoon Kim, Coleman Hooper, Amir Gholami +5
Generative Large Language Models (LLMs) have demonstrated remarkable results for a wide range of tasks. However, deploying these models for inference has been a significant challen…
A Three-regime Model of Network Pruning
Yefan Zhou, Yaoqing Yang, Arin Chang +1
Recent work has highlighted the complex influence training hyperparameters, e.g., the number of training epochs, can have on the prunability of machine learning models. Perhaps sur…
Constrained Optimization via Exact Augmented Lagrangian and Randomized Iterative Sketching
Ilgee Hong, Sen Na, Michael W. Mahoney +1
We consider solving equality-constrained nonlinear, nonconvex optimization problems. This class of problems appears widely in a variety of applications in machine learning and engi…
When are ensembles really effective?
Ryan Theisen, Hyunsuk Kim, Yaoqing Yang +2
Ensembling has a long history in statistical data analysis, with many impactful applications. However, in many modern machine learning settings, the benefits of ensembling are less…
End-to-end codesign of Hessian-aware quantized neural networks for FPGAs and ASICs
Javier Campos, Zhen Dong, Javier Duarte +4
We develop an end-to-end workflow for the training and implementation of co-designed neural networks (NNs) for efficient field-programmable gate array (FPGA) and application-specif…
Outlier Suppression+: Accurate quantization of large language models by equivalent and optimal shifting and scaling
Xiuying Wei, Yunchen Zhang, Yuhang Li +4
Post-training quantization~(PTQ) of transformer language models faces significant challenges due to the existence of detrimental outliers in activations. We observe that these outl…
Full Stack Optimization of Transformer Inference: a Survey
Sehoon Kim, Coleman Hooper, Thanakul Wattanawong +9
Recent advances in state-of-the-art DNN architecture design have been moving toward Transformer models. These models achieve superior accuracy across a wide range of applications.…
Learning Physical Models that Can Respect Conservation Laws
Derek Hansen, Danielle C. Maddix, Shima Alizadeh +2
Recent work in scientific machine learning (SciML) has focused on incorporating partial differential equation (PDE) information into the learning process. Much of this work has foc…
Speculative Decoding with Big Little Decoder
Sehoon Kim, Karttikeya Mangalam, Suhong Moon +4
The recent emergence of Large Language Models based on the Transformer architecture has enabled dramatic advancements in the field of Natural Language Processing. However, these mo…
Gated Recurrent Neural Networks with Weighted Time-Delay Feedback
N. Benjamin Erichson, Soon Hoe Lim, Michael W. Mahoney
In this paper, we present a novel approach to modeling long-term dependencies in sequential data by introducing a gated recurrent unit (GRU) with a weighted time-delay feedback mec…
Monotonicity and Double Descent in Uncertainty Estimation with Gaussian Processes
Liam Hodgkinson, Chris van der Heide, Fred Roosta +1
Despite their importance for assessing reliability of predictions, uncertainty quantification (UQ) measures for machine learning models have only recently begun to be rigorously ch…
Gradient Gating for Deep Multi-Rate Learning on Graphs
T. Konstantin Rusch, Benjamin P. Chamberlain, Michael W. Mahoney +2
We present Gradient Gating (G), a novel framework for improving the performance of Graph Neural Networks (GNNs). Our framework is based on gating the output of GNN layers with…
Adaptive Self-supervision Algorithms for Physics-informed Neural Networks
Shashank Subramanian, Robert M. Kirby, Michael W. Mahoney +1
Physics-informed neural networks (PINNs) incorporate physical knowledge from the problem domain as a soft constraint on the loss function, but recent work has shown that this can l…
Learning differentiable solvers for systems with hard constraints
Geoffrey Négiar, Michael W. Mahoney, Aditi S. Krishnapriyan
We introduce a practical method to enforce partial differential equation (PDE) constraints for functions defined by neural networks (NNs), with a high degree of accuracy and up to…
Neurotoxin: Durable Backdoors in Federated Learning
Zhengming Zhang, Ashwinee Panda, Linyue Song +5
Due to their decentralized nature, federated learning (FL) systems have an inherent vulnerability during their training to adversarial backdoor attacks. In this type of attack, the…
Squeezeformer: An Efficient Transformer for Automatic Speech Recognition
Sehoon Kim, Amir Gholami, Albert Shaw +5
The recently proposed Conformer model has become the de facto backbone model for various downstream speech tasks based on its hybrid attention-convolution architecture that capture…
Fat-Tailed Variational Inference with Anisotropic Tail Adaptive Flows
Feynman Liang, Liam Hodgkinson, Michael W. Mahoney
While fat-tailed densities commonly arise as posterior and marginal distributions in robust models and scale mixtures, they present challenges when Gaussian-based variational infer…
A Fast Post-Training Pruning Framework for Transformers
Woosuk Kwon, Sehoon Kim, Michael W. Mahoney +3
Pruning is an effective way to reduce the huge inference cost of Transformer models. However, prior work on pruning Transformers requires retraining the models. This can add high t…
Learning continuous models for continuous physics
Aditi S. Krishnapriyan, Alejandro F. Queiruga, N. Benjamin Erichson +1
Dynamical systems that evolve continuously over time are ubiquitous throughout science and engineering. Machine learning (ML) provides data-driven approaches to model and predict t…
NoisyMix: Boosting Model Robustness to Common Corruptions
N. Benjamin Erichson, Soon Hoe Lim, Winnie Xu +3
For many real-world applications, obtaining stable and robust statistical performance is more important than simply achieving state-of-the-art predictive test accuracy, and thus ro…
Learning from learning machines: a new generation of AI technology to meet the needs of science
Luca Pion-Tonachini, Kristofer Bouchard, Hector Garcia Martin +33
We outline emerging opportunities and challenges to enhance the utility of AI for scientific discovery. The distinct goals of AI for industry versus the goals of AI for science cre…