13 papers
Neural Scaling Laws of Transformer Operator Network
Haoran Yan, Zhongjie Shi, Yuanzhe Xi +2
Transformers have emerged as powerful architectures for learning solution operators of physical systems. Empirically the prediction error has been observed to decrease when the dat…
Hierarchical Muon: Tiled Newton-Schulz Updates for Efficient Muon Optimization
Ziyuan Tang, Tianshi Xu, Yousef Saad +1
Muon-type optimizers construct update directions for dense neural-network weights by applying a finite Newton-Schulz map to momentum-gradient matrices. For an matrix,…
Factored Sparse Approximate Inverse Preconditioning via Spectral Optimization
Francesco Brarda, Tianshi Xu, Vassilis Kalantzis +2
In this paper, we study value selection for fixed-pattern factorized sparse approximate inverse preconditioners. Given a prescribed sparsity pattern for a factor we choose its…
Hybrid Digital-Analog Approximate Inverse Preconditioning for Krylov Methods
Shikhar Shah, Rui Peng Li, Tayfun Gokmen +3
Analog in-memory computing enables highly parallel matrix-vector multiplications with reduced data movement, but the resulting operations are noisy, quantized, and affected by devi…
Design Criteria for SGD Preconditioners: Local Conditioning, Noise Floors, and Basin Stability
Mitchell Scott, Tianshi Xu, Ziyuan Tang +4
Stochastic Gradient Descent (SGD) often slows in the late stage of training due to anisotropic curvature and gradient noise. We analyze preconditioned SGD in the geometry induced b…
A Joint Variational Framework for Multimodal X-ray Ptychography and Fluorescence Reconstruction
Chengru Eric Zou, Elle Buser, Zichao Wendy Di +1
Recovering high-resolution structural and compositional information from coherent X-ray measurements involves solving coupled, nonlinear, and ill-posed inverse problems. Ptychograp…
Neural Approximate Inverse Preconditioners
Tianshi Xu, Rui Peng Li, Yuanzhe Xi
In this paper, we propose a learning-based framework for constructing approximate inverse preconditioners for partial differential equations (PDEs) by approximating the Green's fun…
Preconditioned Truncated Single-Sample Estimators for Scalable Stochastic Optimization
Tianshi Xu, Difeng Cai, Hua Huang +2
Many large-scale stochastic optimization algorithms involve repeated solutions of linear systems or evaluations of log-determinants. In these regimes, computing exact solutions is…
Rethinking Neural-based Matrix Inversion: Why can't, and Where can
Yuliang Ji, Jian Wu, Yuanzhe Xi
Deep neural networks have achieved substantial success across various scientific computing tasks. A pivotal challenge within this domain is the rapid and parallel approximation of…
Mixed Precision Orthogonalization-Free Projection Methods for Eigenvalue and Singular Value Problems
Tianshi Xu, Zechen Zhang, Jie Chen +2
Mixed-precision arithmetic offers significant computational advantages for large-scale matrix computation tasks, yet preserving accuracy and stability in eigenvalue problems and th…
Preconditioned Additive Gaussian Processes with Fourier Acceleration
Theresa Wagner, Tianshi Xu, Franziska Nestler +2
Gaussian processes (GPs) are crucial in machine learning for quantifying uncertainty in predictions. However, their associated covariance matrices, defined by kernel functions, are…
HiGP: A high-performance Python package for Gaussian Process
Hua Huang, Tianshi Xu, Yuanzhe Xi +1
Gaussian Processes (GPs) are flexible, nonparametric Bayesian models widely used for regression and classification because of their ability to capture complex data patterns and qua…
Multiscale Neural Networks for Approximating Green's Functions
Wenrui Hao, Rui Peng Li, Yuanzhe Xi +2
Neural networks (NNs) have been widely used to solve partial differential equations (PDEs) in the applications of physics, biology, and engineering. One effective approach for solv…