7 citations · 17 across the 13 of their papers we have counts for
7 papers · 1 filter
How Transformers Get Rich: Approximation and Dynamics Analysis
Mingze Wang, Ruoxi Yu, Weinan E +1
Transformers have demonstrated exceptional in-context learning capabilities, yet the theoretical understanding of the underlying mechanisms remains limited. A recent work (Elhage e…
: Language Modeling with Explicit Memory
Hongkang Yang, Zehao Lin, Wenjin Wang +13
The training and inference of large language models (LLMs) are together a costly process that transports knowledge from raw data to meaningful computation. Inspired by the memory h…
Coarse-graining conformational dynamics with multi-dimensional generalized Langevin equation: how, when, and why
Pinchen Xie, Yunrui Qiu, Weinan E
A data-driven ab initio generalized Langevin equation (AIGLE) approach is developed to learn and simulate high-dimensional, heterogeneous, coarse-grained conformational dynamics. C…
Improving Generalization and Convergence by Enhancing Implicit Regularization
Mingze Wang, Jinbo Wang, Haotian He +6
In this work, we propose an Implicit Regularization Enhancement (IRE) framework to accelerate the discovery of flat solutions in deep learning, thereby improving generalization and…
Understanding the Expressive Power and Mechanisms of Transformer for Sequence Modeling
Mingze Wang, Weinan E
We conduct a systematic study of the approximation properties of Transformer for sequence modeling with long, sparse and complicated memory. We investigate the mechanisms through w…
Anchor function: a type of benchmark functions for studying language models
Zhongwang Zhang, Zhiwei Wang, Junjie Yao +4
Understanding transformer-based language models is becoming increasingly crucial, particularly as they play pivotal roles in advancing towards artificial general intelligence. Howe…
Solving multiscale dynamical systems by deep learning
Junjie Yao, Yuxiao Yi, Liangkai Hang +5
Multiscale dynamical systems, modeled by high-dimensional stiff ordinary differential equations (ODEs) with wide-ranging characteristic timescales, arise across diverse fields of s…