6 papers · 1 filter
Functional Equivalence in Attention: A Comprehensive Study with Applications to Linear Mode Connectivity
Viet-Hoang Tran, Vinh Khanh Bui, Van-Hoan Trinh +2
Neural network parameter spaces are inherently non-injective, as distinct parameter configurations can realize identical functions through functional equivalence. While this symmet…
Dynamical Properties of Tokens in Self-Attention and Effects of Positional Encoding
Duy-Tung Pham, An The Nguyen, Viet-Hoang Tran +4
This paper investigates the dynamical properties of tokens in pre-trained Transformer models and explores their application to improving Transformers. To this end, we analyze the d…
On Linear Mode Connectivity of Mixture-of-Experts Architectures
Viet-Hoang Tran, Van Hoan Trinh, Khanh Vinh Bui +1
Linear Mode Connectivity (LMC) is a notable phenomenon in the loss landscapes of neural networks, wherein independently trained models have been observed to be connected--up to per…
Tree-Sliced Wasserstein Distance with Nonlinear Projection
Thanh Tran, Viet-Hoang Tran, Thanh Chu +4
Tree-Sliced methods have recently emerged as an alternative to the traditional Sliced Wasserstein (SW) distance, replacing one-dimensional lines with tree-based metric spaces and i…
Equivariant Polynomial Functional Networks
Thieu N. Vo, Viet-Hoang Tran, Tho Tran Huu +5
Neural Functional Networks (NFNs) have gained increasing interest due to their wide range of applications, including extracting information from implicit representations of data, e…
Equivariant Neural Functional Networks for Transformers
Viet-Hoang Tran, Thieu N. Vo, An Nguyen The +5
This paper systematically explores neural functional networks (NFN) for transformer architectures. NFN are specialized neural networks that treat the weights, gradients, or sparsit…