5 papers
Efficient Propose-Test-Release for Optimal Differentially Private Estimation
Tao Shen, Xin T. Tong, Wanjie Wang
Differential privacy (DP) is a rigorous framework that protects the participation of individuals in a dataset by controlling information leakage through released estimators. It bri…
Dynamical Properties of Tokens in Self-Attention and Effects of Positional Encoding
Duy-Tung Pham, An The Nguyen, Viet-Hoang Tran +4
This paper investigates the dynamical properties of tokens in pre-trained Transformer models and explores their application to improving Transformers. To this end, we analyze the d…
Resolving Memorization in Empirical Diffusion Model for Manifold Data in High-Dimensional Spaces
Yang Lyu, Tan Minh Nguyen, Yuchun Qian +1
Diffusion models are popular tools for generating new data samples, using a forward process that adds noise to data and a reverse process to denoise and produce samples. However, w…
Demystifying the Token Dynamics of Deep Selective State Space Models
Thieu N Vo, Tung D. Pham, Xin T. Tong +1
Selective state space models (SSM), such as Mamba, have gained prominence for their effectiveness in modeling sequential data. Despite their outstanding empirical performance, a co…
Stochastic Gradient Descent with Adaptive Data
Ethan Che, Jing Dong, Xin T. Tong
Stochastic gradient descent (SGD) is a powerful optimization technique that is particularly useful in online learning scenarios. Its convergence analysis is relatively well underst…